Data processing method, electronic device and program product for block storage

Abstract

The present disclosure provides a data processing method, electronic device and program product for block storage. The data processing method for block storage disclosed in the present disclosure includes: constructing a mapping index tree, the mapping index tree is used to uniformly manage the mapping relationship between logical block addresses and physical block addresses, and the mapping index tree is a multi-level cache architecture including cache layers, segmented memory tables and multi-layer ordered tables arranged in sequence; determining the storage priority of the data to be written according to the received data write request, and the storage priority is used to characterize the importance of the data to be written; writing the data to be written to the cache layer or the segmented memory table for storage according to the storage priority; and when the storage data in the segmented memory table or the multi-layer ordered table meets the merge migration rule, merging and migrating the storage data to the storage space of the next level. The technical solution of the embodiment of the present disclosure can realize unified scheduling of cross-layer storage mapping, reduce multi-layer redundant management and write amplification effect.

Description

Technical Field

[0001] The present disclosure relates to a data processing method, electronic device, and program product for block storage. Background Art

[0002] Block storage manages data in logical blocks and is widely used in high-performance computing, virtualization, and containerized environments. In multi-layer virtualization scenarios, the host, virtual machines / containers, and SSD firmware layers each maintain independent mapping mechanisms from logical block addresses to physical block addresses. This can lead to redundant logging, multiple garbage collections, and cross-layer scheduling failures, which in turn can cause write amplification, storage fragmentation, and I / O performance fluctuations. Summary of the Invention

[0003] The present disclosure provides a data processing method, electronic device, and program product for block storage.

[0004] According to one aspect of the present disclosure, a data processing method for block storage is provided, comprising:

[0005] Constructing a mapping index tree, wherein the mapping index tree is used to uniformly manage the mapping relationship between logical block addresses and physical block addresses, and the mapping index tree is a multi-level cache architecture including sequentially arranged cache layers, segmented memory tables, and multi-layer ordered tables;

[0006] Determining a storage priority of the data to be written according to the received data write request, wherein the storage priority is used to represent the importance of the data to be written;

[0007] Writing the data to be written into the cache layer or the segmented memory table for storage according to the storage priority; and

[0008] When the storage data in the segmented memory table or the multi-layer ordered table meets the merge migration rule, the storage data is merged and migrated to the storage space of the next level.

[0009] According to one aspect of the technical solution, a mapping index tree is constructed to uniformly manage the mapping relationship between logical block addresses and physical block addresses. The mapping index tree is a multi-level cache architecture comprising sequentially arranged cache layers, segmented memory tables, and multi-layer ordered tables. Next, based on a received data write request, a storage priority for the data to be written is determined. The storage priority is used to represent the importance of the data to be written. Based on the storage priority, the data to be written is written to the cache layer or the segmented memory table for storage. Furthermore, if the data stored in the segmented memory table or the multi-layer ordered table meets the merge migration rules, the data is merged and migrated to the next level of storage space.

[0010] Therefore, based on the mapping index tree, unified scheduling of cross-layer storage mapping can be achieved, reducing multi-layer redundant management and thus improving I / O access efficiency.

[0011] In addition, according to the storage priority of the data to be written, the data to be written is written to the cache layer or segmented memory table for storage, allowing non-urgent writes to be accumulated in the cache layer first, avoiding frequent access to the memory table, and thus reducing the write amplification effect.

[0012] In at least one embodiment of the present disclosure, writing the to-be-written data to the cache layer or the segmented memory table for storage according to the storage priority includes:

[0013] When the storage priority is the first level, writing the data to be written into the cache layer for storage;

[0014] When the storage priority is the second level, the data to be written is written into the segmented memory table for storage, and the importance of the second level is higher than that of the first level.

[0015] According to the technical solution of this embodiment, frequent access to the memory table can be avoided, reducing the write amplification effect.

[0016] In at least one embodiment of the present disclosure, when the stored data in the segmented memory table or the multi-layer ordered table meets the merge migration rule, the stored data is merged and migrated to the storage space of the next level, the method includes:

[0017] Determining a first ratio of the storage data in the segmented memory table to its own total storage capacity;

[0018] The stored data in the segmented memory table whose first ratio is greater than or equal to the first capacity threshold is merged and migrated to the multi-layer ordered table.

[0019] According to the technical solution of this embodiment, when a single segment memory table is full, a merge migration operation can be independently triggered, and other segment memory tables can continue to receive new write requests to avoid blocking.

[0020] In at least one embodiment of the present disclosure, the method further comprises:

[0021] Get system load information;

[0022] The first capacity threshold is determined according to the system load information.

[0023] According to the technical solution of this embodiment, the first capacity threshold can be dynamically adjusted based on the real-time load of the system, thereby reducing write amplification, lowering I / O oscillation, and improving system stability.

[0024] In at least one embodiment of the present disclosure, when the stored data in the segmented memory table or the multi-layer ordered table meets the merge migration rule, when the stored data is merged and migrated to the storage space of the next level, the method further includes:

[0025] For multi-layer ordered tables, in L n The number of ordered lists of the level reaches the threshold, and L n When the second ratio of the storage data in the ordered table of the level to the total storage capacity of the level is greater than or equal to the second capacity threshold, n At least some ordered lists in the hierarchy are selected as ordered lists to be merged, where n is an integer ≥ 0;

[0026] L n The storage data of the hierarchical ordered table to be merged is the same as L n+1 The storage data of the hierarchical ordered table to be merged are merged and sorted in the order of the logical block address to obtain a temporary ordered table, where L n+1 The hierarchical ordered table to be merged is the key value range and L n The key value ranges of the hierarchical ordered tables to be merged overlap with each other;

[0027] Write the temporary ordered table into L n+1 level, and discard L n Level and L n+1 The ordered list to be merged in the hierarchy.

[0028] According to the technical solution of this embodiment, by merging ordered tables with overlapping key value ranges, duplicate data storage can be reduced, thereby improving space utilization. In addition, the merged ordered tables are continuously distributed in the order of logical block addresses, which can improve query efficiency.

[0029] In at least one embodiment of the present disclosure, the method further comprises:

[0030] Determine the data failure ratio corresponding to the ordered table based on the number of invalid key-value pairs and the total number of key-value pairs in the ordered table;

[0031] When the data failure ratio reaches a failure ratio threshold, the stored data in the ordered table is merged and migrated to an ordered table of a next level.

[0032] According to the technical solution of this embodiment, resources can be effectively recovered and the overhead of reading invalid records can be reduced. Partial merging ensures that the multi-layer ordered table is not occupied by a large amount of invalid data or outdated versions for a long time.

[0033] In at least one embodiment of the present disclosure, the mapping index tree is integrated into the kernel block layer to eliminate duplicate logging and mapping management.

[0034] According to the technical solution of this embodiment, the repeated log records and mapping management between the original layers can be effectively eliminated, avoiding the waste of storage resources and write amplification caused by redundant indexes and frequent updates.

[0035] In at least one embodiment of the present disclosure, the method further comprises:

[0036] Determine the logical block address to be queried according to the received data query request;

[0037] According to the logical block address, query is performed in each storage space in the order of the cache layer, the segmented memory table and the multi-layer ordered table;

[0038] According to the query result, the physical block address corresponding to the logical block address is fed back.

[0039] According to the technical solution of this embodiment, data query is performed based on the mapping index tree, which can improve query efficiency.

[0040] According to another aspect of the present disclosure, there is provided a data processing device for block storage, comprising:

[0041] A construction module is used to construct a mapping index tree, wherein the mapping index tree is used to uniformly manage a unified mapping relationship between a logical block address and a physical block address, and the mapping index tree is a multi-level cache architecture including sequentially arranged cache layers, segmented memory tables, and multi-layer ordered tables;

[0042] a determination module, configured to determine a storage priority of the data to be written according to the received data write request, wherein the storage priority is used to represent the importance of the data to be written;

[0043] a writing module, configured to write the data to be written into the cache layer or the segmented memory table for storage according to the storage priority; and

[0044] The processing module is configured to merge and migrate the stored data to a storage space of a next level when the stored data in the segmented memory table or the multi-layer ordered table meets a merge and migration rule.

[0045] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the data processing method for block storage according to any embodiment of the present disclosure.

[0046] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, the execution instructions are used to implement the data processing method for block storage according to any embodiment of the present disclosure.

[0047] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the data processing method for block storage according to any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0049] Figure 1 A flow chart of a data processing method for block storage according to an embodiment of the present disclosure is shown.

[0050] Figure 2 A schematic structural diagram of a mapping index tree according to an embodiment of the present disclosure is shown.

[0051] Figure 3 A schematic diagram of a system structure applicable to an embodiment of the present disclosure is shown.

[0052] Figure 4 FIG. 1 is a flow chart of step S130 in a data processing method for block storage according to an embodiment of the present disclosure.

[0053] Figure 5 FIG. 1 is a flow chart of step S140 in a data processing method for block storage according to an embodiment of the present disclosure.

[0054] Figure 6 A schematic diagram of a flow chart of determining a first capacity threshold value is shown in a data processing method for block storage according to an embodiment of the present disclosure.

[0055] Figure 7 A schematic diagram of a process for triggering a merge migration operation of a segmented memory table in a data processing method for block storage according to an embodiment of the present disclosure is shown.

[0056] Figure 8 A schematic diagram of a process of triggering a merge migration operation of a multi-layer ordered table, which is also included in step S140 in the data processing method for block storage according to an embodiment of the present disclosure, is shown.

[0057] Figure 9A schematic diagram of a process of triggering a partial merge migration operation of a multi-layer ordered table, which is also included in step S140 in the data processing method for block storage according to an embodiment of the present disclosure, is shown.

[0058] Figure 10 A data processing method for block storage according to an embodiment of the present disclosure further includes a flow chart of data query.

[0059] Figure 11 A technical architecture diagram applicable to the embodiments of the present disclosure is shown.

[0060] Figure 12 A flow chart of a data processing method for block storage according to another embodiment of the present disclosure is shown.

[0061] Figure 13 A block diagram of a data processing device for block storage according to an embodiment of the present disclosure is shown.

[0062] Figure 14 The figure is a schematic structural block diagram of an electronic device equipped with a data processing device for block storage according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0063] The present disclosure is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are intended only to illustrate the relevant content and are not intended to limit the present disclosure. It should also be noted that, for ease of description, only the portions relevant to the present disclosure are shown in the accompanying drawings.

[0064] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0065] Block storage is a storage technology that manages and reads and writes data using fixed-size blocks as the basic unit. Its core mechanism is to achieve direct access to storage media through a mapping relationship between logical block addresses (LBAs) and physical block addresses (PBAs). Block storage, with its efficient data management capabilities, excellent compatibility, and flexible deployment methods, has been widely used in high-performance computing (HPC), large-scale model training, virtualization platforms, and other fields.

[0066] By managing and reading and writing data in blocks, block storage provides a unified storage interface across different systems and applications, reducing the adaptation overhead of file systems or distributed storage layers. In high-performance computing (HPC) scenarios, block storage often serves as the underlying support for parallel file systems (such as Lustre and GPFS), providing high-throughput data access for large-scale scientific computing. In large-scale deep learning model training, the massive parameter updates and high-concurrency I / O requests place extreme demands on storage systems for low latency and high throughput. In virtualized and containerized environments, efficient management and isolation of block devices become particularly critical due to multi-tenancy requirements and the elastic expansion of computing resources.

[0067] At the same time, multi-layer virtualization technologies (such as KVM and Docker) are widely used in cloud computing and large-scale data centers, providing resource isolation, elastic scalability, and efficient management. However, multi-layer virtualization introduces multiple functionally similar but independent data management layers, including the host operating system, the block device mapping mechanism within the virtual machine or container, and the Flash Translation Layer (FTL) of the underlying storage device (such as SSD). These layers each maintain independent logical block number (LBA) to physical block number (PBA) mappings, increasing data management redundancy and leading to problems such as write amplification, storage fragmentation, and reduced I / O performance, thus impacting the overall storage efficiency and lifespan of the system.

[0068] To this end, the present disclosure proposes the following technical solution, which, based on a mapping index tree, enables unified scheduling of cross-layer storage mappings, reduces multi-layer redundant management, and thus improves I / O access efficiency. Furthermore, based on the storage priority of the data to be written, the data to be written is written to the cache layer or segmented memory table for storage, allowing non-urgent writes to be accumulated in the cache layer first, avoiding frequent access to the memory table and thus reducing the write amplification effect.

[0069] Figure 1 FIG. 1 is a flow chart showing a data processing method for block storage according to an embodiment of the present disclosure. Figure 1 The method shown includes at least steps S110 to S140, which are described in detail as follows:

[0070] In step S110, a mapping index tree is constructed, which is used to uniformly manage the unified mapping relationship between logical block addresses and physical block addresses. The mapping index tree is a multi-level cache architecture including cache layers, segmented memory tables and multi-layer ordered tables arranged in sequence.

[0071] In this embodiment, the Mapping Tree (M-Tree) can be a multi-level cache architecture integrated into the operating system, which is used to uniformly manage the mapping relationship between logical block addresses and physical block addresses. Its core structure includes a cache layer, a segmented memory layer, and a multi-layer ordered table arranged in sequence.

[0072] Specifically, if Figure 2 As shown, the cache layer can be implemented as a lock-free ring buffer, supporting multi-threaded concurrent writes and temporarily storing newly generated LBA→PBA mappings or pending write requests. In other words, when a write request is received, the LBA→PBA mapping can be temporarily stored in the cache layer, avoiding the frequent I / O operations caused by writing directly to the persistent structure. This allows the cache layer to aggregate small, random write requests, reducing the direct impact on segmented memory tables, significantly reducing write latency, and improving throughput in high-concurrency scenarios.

[0073] The segmented memory table uses hash sharding to map LBAs to multiple independent segments. A skip list structure can be used within each segment to maintain an orderly LBA-to-PBA mapping. In one example, an LBA can be hashed and modulo-calculated to assign it to a specific segment. Segments in the segmented memory table do not interfere with each other, improving concurrency. Hash sharding also enables load balancing, enabling concurrent writes and queries, and avoiding global lock contention.

[0074] A multi-layer ordered table consists of multiple layers of persistent structures (such as SSTables) sorted by LBA. Each layer can be equipped with a Bloom filter (BF) to accelerate queries. Data is flushed from the segmented memory table to the highest layer (i.e., L0) of the multi-layer ordered table to form persistent storage. Furthermore, when the number of ordered tables or the proportion of invalid data in a layer exceeds a threshold (see below), a compaction operation can be triggered. This consolidates and sorts data from adjacent layers, moving it down to the next layer and clearing invalid records. This reduces redundant data writes (lowering write amplification) and optimizes storage space utilization through hierarchical storage and dynamic merging. Bloom filters can quickly filter invalid query paths, improving retrieval efficiency. Furthermore, some merging strategies reduce I / O resource usage and alleviate performance jitter.

[0075] In some embodiments of the present disclosure, the mapping index tree can be integrated into the kernel block layer of the operating system, which is an important intermediate layer in the operating system that processes block device I / O requests. Figure 3 As shown in the figure, read and write requests submitted by upper layers (such as file systems and user-mode applications) are uniformly connected to the kernel block layer. After queuing, scheduling, and forwarding, data access operations are ultimately completed. By providing a unified abstract interface and adaptation functions, the kernel block layer ensures stable and efficient access to storage devices for upper-layer applications.

[0076] Compared to placing the mapping index tree at a higher layer (file system or user mode) or a lower layer (such as SSD firmware FTL), the kernel block layer, due to its location at a key node in the data flow path, naturally offers the advantage of unified cross-layer mapping scheduling, covering the multi-layer data management needs of the host, virtual machines / containers, and storage media. Therefore, placing the mapping index tree at the kernel block layer effectively eliminates duplicate logging and mapping management between layers, avoiding the waste of storage resources and write amplification caused by redundant indexes and frequent updates.

[0077] At the same time, the unified mapping tree can more efficiently identify hot data, stratify cold data, and optimize local data merging, thereby achieving adaptive load adjustment in high-concurrency scenarios and better balancing performance stability and equipment life extension.

[0078] Please continue to refer to Figure 1 In step S120, the storage priority of the data to be written is determined according to the received data writing request, and the storage priority is used to represent the importance of the data to be written.

[0079] Storage priority can be used to characterize the importance or urgency of the data to be written. In one example, those skilled in the art can pre-assign corresponding storage priorities to different data based on the importance of the data. For example, large-block sequential writes, metadata updates, explicit synchronization requests, and crash-consistency guarantee operations (such as transaction log commits) can be assigned high priority. Small-scale random writes (such as temporary container logs and intermediate edge computing results) can be assigned low priority to allow delayed writes and thus aggregate batch operations.

[0080] In this embodiment, after receiving a data write request, the data write request can be parsed to determine the storage priority corresponding to the data to be written. In one example, a determination can be made based on metadata in the data write request (such as I / O type, data size, source application, etc.) to determine the corresponding storage priority for the data to be written. For example, based on the received data write request, if the bi_opf field included in the struct bio in the data write request is displayed as REQ_META or REQ_SYNC, it indicates that the write is metadata, log write, or explicit synchronization request, and is a write of high urgency, so it can be marked as high priority.

[0081] In step S130, the data to be written is written into the cache layer or the segmented memory table for storage according to the storage priority.

[0082] In this embodiment, based on the determined storage priority, low-priority data to be written can be temporarily stored in the cache layer, while high-priority data to be written can be directly written to the segmented memory table. In this way, the write path is dynamically selected based on the storage priority of the data to be written, which can achieve efficient allocation of I / O resources and optimize system performance.

[0083] In step S140 , when the storage data in the segmented memory table or the multi-layer ordered table meets the merge migration rule, the storage data is merged and migrated to the storage space of the next level.

[0084] The merge migration rule may be a rule predetermined by those skilled in the art based on prior experience and used to trigger data merging. In one example, the merge migration rule may be based on system load and storage data volume as a basis for determining whether to enable data merging.

[0085] In this embodiment, data merge migration may occur in two locations: one is the merge migration of the segment inner table to the highest level (i.e., L0 level) of the multi-layer ordered table; the other is the merge migration of the segment inner table to the highest level (i.e., L0 level) of the multi-layer ordered table. n Layer-Ordered Surface L n+1 Merge and migrate layer-ordered tables.

[0086] In one example, a segmented memory table is written sequentially to the L0 layer of a multi-layer ordered table to form an ordered table. Each layer has a limit on the number of ordered tables. If the limit is exceeded, a compaction is triggered. After the compaction, a new file is generated and placed on the next layer, and so on, until the data is completely cooled or no longer frequently updated. As a result, newer or hot data tends to stay in higher layers (such as L0 and L1), while cold data sinks to deeper layers.

[0087] It's important to note that the purpose of merging segmented memory tables into the L0 layer of a multi-layered ordered table is to quickly free up cache space so that new data can quickly enter persistent storage. Therefore, when the capacity of a segmented memory table reaches a certain threshold, the merge migration of that segmented memory table is triggered, while other segments can continue to accept new requests, preventing the merge migration process from blocking overall write throughput. In other words, segmented refreshes can prevent a single large memory table from blocking all writes when it reaches its limit, improving scalability and throughput in high-concurrency environments.

[0088] For the multi-layer ordered list L n Layer L n+1 The merging of layers is mainly to reclaim space and remove invalid data, which does not require much adjustment space. Based on the merging migration rules, hot data with high frequency updates can be kept in high layers (such as L0 and L1 layers), and cold data with low frequency access can be sunk to deep layers (such as L2 and L3 layers). n layer), thus adapting to mixed load scenarios.

[0089] So, based on Figure 1 The illustrated embodiment constructs a mapping index tree to uniformly manage the mapping relationship between logical block addresses and physical block addresses. The mapping index tree is a multi-level cache architecture that includes a cache layer, a segmented memory table, and a multi-layer ordered table arranged in sequence. Next, based on the received data write request, the storage priority of the data to be written is determined. The storage priority is used to represent the importance of the data to be written. Based on the storage priority, the data to be written is written to the cache layer or the segmented memory table for storage. Moreover, when the stored data in the segmented memory table or the multi-layer ordered table meets the merge migration rule, the stored data is merged and migrated to the storage space of the next level. Thus, based on the mapping index tree, unified scheduling of cross-layer storage mapping can be achieved, reducing multi-layer redundant management, thereby improving I / O access efficiency. Furthermore, based on the storage priority of the data to be written, the data to be written is written to the cache layer or the segmented memory table for storage, allowing non-urgent writes to be accumulated in the cache layer first, avoiding frequent access to the memory table, thereby reducing the write amplification effect.

[0090] Figure 4 FIG. 1 shows a flow chart of step S130 in a data processing method for block storage according to an embodiment of the present disclosure. Figure 4 As shown, step S130 at least includes steps S131 to S132, which are described in detail as follows.

[0091] In step S131 , when the storage priority is the first level, the data to be written is written into the cache layer for storage.

[0092] In this embodiment, the first level can be a low priority level. The first level is suitable for write requests that allow short delays but require high throughput, such as temporary container logs and intermediate edge computing results. After determining that the storage priority of the data to be written is the first level, the data to be written can be directly allocated to the cache layer for storage. In one example, the cache layer can locally aggregate data according to the logical block address range and merge multiple small random writes into batch operations, thereby reducing the number of single I / O operations.

[0093] In step S132, when the storage priority is the second level, the data to be written is written into the segmented memory table for storage, and the importance of the second level is higher than the importance of the first level.

[0094] In this embodiment, the second level of importance is higher than the first level of importance, meaning the second level can be a high priority level. It should be understood that data to be written with a storage priority of the second level indicates that it is highly important or has strong real-time requirements, and that rapid write-to-disk storage is prioritized. Therefore, data to be written with a storage priority of the second level can be written directly to the segmented memory table for storage, thereby ensuring the real-time and reliability of highly important data.

[0095] In one example, when writing data to a segmented memory table, a hash operation can be performed based on the logical block address to determine the corresponding segmented memory table number (e.g., segmented memory table 3). The LBA→PBA mapping to be written is then inserted in ascending order into the skip list structure within the segmented memory table corresponding to the segmented memory table number, thereby ensuring query efficiency.

[0096] As a result, first-level data is written to the cache layer, and second-level data is written directly to the segmented memory table, bypassing the cache layer. This allows non-urgent writes to accumulate in the cache layer first, avoiding frequent access to the memory table and reducing the write amplification effect.

[0097] Figure 5 FIG. 1 shows a flow chart of step S140 in a data processing method for block storage according to an embodiment of the present disclosure. Figure 5 As shown, step S140 at least includes steps S141 to S142, which are described in detail as follows.

[0098] In step S141, a first ratio of the storage data in the segment memory table to its own total storage capacity is determined.

[0099] In this embodiment, for each segmented memory table, the amount of data stored in the segmented memory table and the corresponding total storage capacity of the segmented memory table can be obtained. Then, the corresponding first ratio can be obtained by dividing the amount of data stored in the segmented memory table by the total storage capacity of the segmented memory table. For example, if the total storage capacity of the segmented memory table is 1000 mapping records and 800 are currently stored, the corresponding first ratio is 80%.

[0100] In one example, the total storage capacity of each segmented memory table may be fixedly allocated, or it may be allocated in proportion to the total system memory (ie, dynamic capacity), which is not specifically limited in the present disclosure.

[0101] In step S142, the stored data in the segmented memory table whose first ratio is greater than or equal to the first capacity threshold is merged and migrated to the multi-layer ordered table.

[0102] In this embodiment, the first capacity threshold may be a preset ratio threshold for triggering the merging and migration of the stored data in the segmented memory table. For example, the first capacity threshold may be 70% or 80%.

[0103] After determining the first ratio corresponding to the segmented memory table, the first ratio may be compared with a first capacity threshold. If the first ratio threshold is greater than or equal to the first capacity threshold, a merge migration operation of the segmented memory table may be triggered, that is, the stored data in the segmented memory table is merged and migrated to the highest layer (i.e., L0 layer) in the multi-layer ordered table.

[0104] Therefore, when the first ratio corresponding to a single segmented memory table reaches the first capacity threshold, the merge and migration operation of the data stored in the segmented memory table can be triggered, and other segmented memory tables can continue to receive new write requests, thereby avoiding blocking all writes after a single large memory table reaches the upper limit, and improving scalability and throughput in a high-concurrency environment.

[0105] Based on the above embodiments, Figure 6 FIG. 1 shows a flow chart of determining a first capacity threshold value in a data processing method for block storage according to an embodiment of the present disclosure. Figure 6 As shown, determining the first capacity threshold at least includes steps S143 to S144, which are described in detail below.

[0106] In step S143, system load information is obtained.

[0107] Among them, the system load information can be parameter information used to characterize the system resource usage status, which may include but is not limited to at least one of the system's CPU utilization (for example, the percentage of CPU non-idle state per unit time), I / O queue depth (the number of I / O requests to be processed by the storage device, measuring the degree of congestion in the storage layer), and memory usage (the ratio of occupied memory to total memory capacity, indicating the tightness of memory resources).

[0108] In this embodiment, the system load information may be collected periodically to be used for subsequent determination of the first capacity threshold.

[0109] In step S144, the first capacity threshold is determined according to the system load information.

[0110] In this embodiment, the first capacity threshold corresponding to the segmented memory table can be dynamically adjusted based on the collected system load information. It should be noted that the greater the system load pressure, the larger the first capacity threshold, thereby delaying the merge migration process and ensuring the read and write priority of the foreground. Conversely, the smaller the system load pressure, the smaller the first capacity threshold, thereby quickly clearing the memory and accelerating the speed at which new data enters persistent storage.

[0111] In one embodiment, the overall system load level can be determined based on the collected system load information. For example, a weighted calculation can be performed based on the collected system load information to obtain a corresponding load score. This load score is then compared with a pre-set load threshold, and a corresponding first capacity threshold is determined based on the comparison result.

[0112] For example, when the system load score is less than 30%, the first capacity threshold that triggers merge migration can be lowered to 60%, allowing for faster memory cleanup and faster entry of new data into persistent storage. When the system load score is greater than 80%, the first capacity threshold that triggers merge migration can be raised to 90%, delaying the merge migration process to ensure foreground I / O priority.

[0113] Based on the above embodiments, Figure 7 The following is a flow chart showing a method for processing block storage data according to an embodiment of the present disclosure for triggering a merge migration operation of a segmented memory table. Figure 7 As shown in the figure, triggering the merge migration operation of the segmented memory table includes at least the following steps:

[0114] In step S710, system load information is periodically collected.

[0115] That is, according to the set period, the system load information is collected to provide a basis for subsequent threshold adjustment and merging trigger decisions.

[0116] In step S720, it is determined whether the system load is less than 30%, that is, the system load level is quantified based on the collected system load information, and it is determined whether the current system load is less than 30%.

[0117] In step S730 , if the system load is lower than 30%, the first capacity threshold is adjusted to 60%.

[0118] In step S740 , if the system load is greater than or equal to 30%, it is further determined whether the system load is greater than 80%.

[0119] In step S750 , if the system load is less than or equal to 80%, the first capacity threshold is adjusted to 70%.

[0120] In step S760, if the system load is greater than 80%, the first capacity threshold is adjusted to 90%.

[0121] In step S770, based on the determined first capacity threshold, it is checked whether the first ratio corresponding to a certain segment memory table reaches the first capacity threshold.

[0122] In step S780 , if the first ratio corresponding to the segmented memory table reaches a first capacity threshold (ie, ≥ the first capacity threshold), merging and migration to the L0 layer of the multi-layer ordered table is triggered.

[0123] In step S790, if the first ratio corresponding to the segmented memory table does not reach the first capacity threshold, then wait for the next cycle, restart the periodic collection of system load information, and then perform subsequent judgments and operations.

[0124] It should be noted that the above figures are only illustrative examples. Those skilled in the art can set corresponding first capacity thresholds for different load conditions according to specific implementation needs, and this disclosure does not make any special limitations on this.

[0125] Figure 8 FIG. 1 shows a flow chart of triggering a merge migration operation of a multi-layer ordered table in step S140 of a data processing method for block storage according to an embodiment of the present disclosure. Figure 8 As shown, triggering the merge migration operation of the multi-layer ordered table includes at least steps S145 to S147, which are described in detail below.

[0126] In step S145, for the multi-layer ordered table, n The number of ordered lists of the level reaches the threshold, and L n When the second ratio of the storage data in the ordered table of the level to the total storage capacity of the level is greater than or equal to the second capacity threshold, n At least some of the ordered lists in the hierarchy are selected as the ordered lists to be merged, where n is an integer ≥ 0.

[0127] The quantity threshold may be a pre-set threshold used to determine whether to trigger the merge migration. n Upper limit of the number of ordered lists of a layer. It should be understood that the number thresholds corresponding to different layers may be different, for example, the number threshold corresponding to the ordered list of the L0 layer is 1, the number threshold corresponding to the ordered list of the L1 layer is 2, and so on.

[0128] The second capacity threshold is a second lower limit of the ratio for triggering the merge migration operation of the ordered table.

[0129] In this embodiment, the system can periodically count L n If the number of ordered tables at the level reaches the corresponding threshold (i.e. greater than or equal to the threshold), the system can enter the capacity ratio verification phase. n For all ordered tables in the hierarchy, a second ratio corresponding to each ordered table is calculated. The second ratio is the ratio of the stored data in the ordered table to the total storage capacity of the ordered table.

[0130] If L nIf the second ratio of all ordered tables in the hierarchy is greater than the second capacity threshold (e.g. 70%), the merge migration operation of the ordered tables is triggered. n Select a part of the ordered list of the level as the ordered list to be merged. For example, you can randomly select n Select one or two ordered tables in the hierarchy as the ordered tables to be merged for merge migration.

[0131] In step S146, L n The storage data of the hierarchical ordered table to be merged is the same as L n+1 The storage data of the hierarchical ordered table to be merged are merged and sorted in the order of the logical block address to obtain a temporary ordered table, where L n+1 The hierarchical ordered table to be merged is the key value range and L n The key value ranges of the hierarchical ordered tables to be merged overlap.

[0132] In this embodiment, the n+1 Select the key value range and L in the hierarchy n The key value ranges of the ordered lists to be merged at the level have overlapping ordered lists as L n+1 The ordered list to be merged at the level, for example, L n The key value range of the hierarchical ordered table to be merged covers LBA 0x1000-0x2000, and L n+1 The key value range of the hierarchical ordered table to be merged covers LBA 0x1800-0x2800.

[0133] In determining L n Level and L n+1 After the ordered lists to be merged at different levels are formed, the stored data in the ordered lists to be merged at two levels can be merged and sorted in the order of the logical block addresses to obtain a temporary ordered list.

[0134] In step S147, the temporary ordered table is written into L n+1 level, and discard L n Level and L n+1 The ordered list to be merged in the hierarchy.

[0135] In this embodiment, after obtaining the temporary ordered table, the temporary ordered table can be written into L n+1 level, and discard L n Level and L n+1 Several ordered lists designed in the previous merge migration operation in the hierarchy (i.e., the ordered lists to be merged determined in the two hierarchies mentioned above).

[0136] In this way, by merging ordered tables with overlapping key value ranges, duplicate data storage can be reduced, thereby improving space utilization. In addition, the merged ordered tables are continuously distributed in the order of logical block addresses, which can improve query efficiency.

[0137] Based on the above embodiments, Figure 9 FIG. 1 shows a flow chart of a partial merge migration operation of a multi-layer ordered table, which is also included in step S140 of a data processing method for block storage according to an embodiment of the present disclosure. Figure 9 As shown, triggering a partial merge migration operation of a multi-layer ordered table includes at least steps S148 to S149, which are described in detail below.

[0138] In step S148, the data failure ratio corresponding to the ordered table is determined according to the number of failed key-value pairs and the total number of key-value pairs in the ordered table.

[0139] In this embodiment, each ordered table can maintain a data failure ratio (failure number / total number of key-value pairs). The failure number of the key-value pairs is obtained in the previous merging process. n The data written down by the layer (LBA, PBA) is the latest data. n+1 The key-value pairs corresponding to the original LBAs are recorded as invalid, thereby obtaining the number of failures in each ordered table, and then combining it with the total number of key-value pairs to calculate the data failure ratio.

[0140] In step S149 , when the data failure ratio reaches a failure ratio threshold, the stored data in the ordered table is merged and migrated to an ordered table at a next level.

[0141] In this embodiment, those skilled in the art can pre-set a failure ratio threshold (e.g., 50%) based on prior experience as the upper limit for the data failure ratio that triggers a partial merge migration operation. The system can periodically determine the data failure ratio for each ordered table and compare it with the failure ratio threshold. If the data failure ratio of a particular ordered table reaches the failure ratio threshold (i.e., greater than or equal to the failure ratio threshold), the stored data of that ordered table can be merged to a lower layer. The merge operation steps can be referred to in the previous embodiment and will not be repeated here.

[0142] In this way, by detecting and promptly sorting out ordered tables with high failure rates, resources can be effectively recovered and the overhead of reading invalid records can be reduced. Partial merging ensures that multi-layer ordered tables are not occupied by large amounts of invalid data or outdated versions for a long time. It should be understood that in a multi-layer ordered table, the upper the layer, the smaller the data capacity, the "fresher" the data stored, and the smaller the corresponding data failure rate. By using the above-mentioned partial merging and migration method, space can be freed up in the middle layer of the multi-layer ordered table, avoiding multi-layer chain merging caused by merging and migrating from the upper layer to the middle and lower layers. This maintains the overall performance and storage efficiency of the system in high concurrency and frequent update scenarios.

[0143] It can be clearly seen that for high-concurrency small random write and mixed load scenarios, the embodiments of the present disclosure propose an adaptive merging mechanism that can dynamically adjust the merging strategy and combine with partial merging mechanisms to reduce write amplification and I / O oscillation, thereby improving the stability of the storage system.

[0144] Figure 10 FIG. 1 shows a data processing method for block storage according to an embodiment of the present disclosure, which also includes a flow chart of data query. Figure 10 As shown, the data query includes at least steps S150 to S170, which are described in detail below.

[0145] In step S150, the logical block address to be queried is determined according to the received data query request.

[0146] In this embodiment, the data query request may be information for querying a physical block address corresponding to a certain logical block address. After receiving the data query request, the data query request may be parsed to obtain the logical block address to be queried.

[0147] In step S160, according to the logical block address, a query is performed in each storage space in the order of the cache layer, the segmented memory table and the multi-layer ordered table.

[0148] In this embodiment, queries can be performed in each storage space in the order of the cache layer, the segmented memory table, and the multi-layer ordered table. Specifically, the cache layer is queried first, and if a hit is found, a result is fed back. If a hit is not found, the segmented memory table is queried. Before querying the segmented memory table, the segment corresponding to the physical block address of the logical block address to be queried can be determined, and then the segmented memory table corresponding to the segment can be determined for querying, thereby reducing unnecessary queries.

[0149] If the result is hit in the segmented memory table, the result is fed back. If it is not hit, the multi-layer ordered table is queried layer by layer. In addition, when querying the multi-layer ordered table, Bloom filters can be used to skip non-target tables, thereby speeding up the query.

[0150] In step S170, the physical block address corresponding to the logical block address is fed back according to the query result.

[0151] In this embodiment, after performing a query in the mapping index tree, the physical block address corresponding to the queried logical block address can be fed back to the user or system based on the query result, thereby improving query efficiency and reducing query latency.

[0152] Based on the technical solutions of the above embodiments, a specific application scenario of the embodiments of the present application is introduced below:

[0153] This paper proposes an efficient block storage data organization method for multi-layer virtualization environments, called AdaptiDisk. Its core goal is to achieve efficient mapping index management, dynamic data merging optimization, cache optimization and I / O resource scheduling at the block layer to reduce storage management redundancy and improve the overall performance of the storage system. Figure 11 As shown in the figure, AdaptiDisk mainly consists of three parts: M-Tree mapping index, adaptive merge scheduler, and multi-level cache architecture, which are explained in detail below.

[0154] User mode / virtualization layer 1110: This layer is an upper-layer application or virtualization management tool in a virtualized environment. It is responsible for initiating read and write requests and is the entrance to the entire block storage system.

[0155] Kernel Block Layer 1120: As the key middle layer in the operating system for processing block device I / O requests, it receives, queues, schedules, and forwards read and write requests submitted by upper layers (such as the file system and user-mode applications), ensuring stable and efficient access to storage devices. It also provides the foundation for lower-layer mapping index management, dynamic data merging optimization, cache optimization, and I / O resource scheduling.

[0156] M-Tree mapping index 1130: Integrated in the kernel block layer, it is used to manage the mapping of logical block numbers (LBA) to physical block numbers (PBA), and implement unified scheduling of cross-layer storage mapping to reduce multi-layer redundant management and improve I / O access efficiency.

[0157] Cache layer (ring buffer) 1131: Uses a lock-free ring buffer to support multi-threaded concurrent writing. It is used to receive the latest LBA→PBA mapping or write request, temporarily aggregate a large number of small random writes, reduce the number of direct writes to the segmented memory table, and alleviate the impact pressure of random writes in the foreground. When the buffer layer is full, it will write to the segmented memory table.

[0158] Segmented memory table (skip table) 1132: It is implemented using a skip table and is designed to be multi-segmented. The segment is determined by performing hash function and modulo operations on the LBA. The segments of the segmented memory table do not interfere with each other, which improves concurrency. At the same time, hash sharding can achieve load balancing and is conducive to range queries.

[0159] Multi-layer ordered table (SSTable) 1133: Organized in a hierarchical structure, it persistently stores the mapping relationships flushed from the segmented memory table. The data in each layer is sorted by LBA. The ordered tables at each level contain corresponding Bloom filters to quickly locate data.

[0160] Adaptive Merge Scheduler 1140: Dynamically adjusts the merge strategy for high-concurrency small random writes and mixed load scenarios to reduce write amplification and I / O oscillation, thereby improving storage system stability.

[0161] Adaptive trigger strategy 1141: By collecting CPU load rate, I / O queue depth, and memory usage, it periodically evaluates memory usage and system real-time load, and dynamically modifies the trigger threshold based on real-time conditions.

[0162] Partial Merge Strategy 1142: In a Multi-layer Ordered List L n Layer L n+1 Layer merging is primarily used to reclaim space and remove invalid data. It's triggered when all ordered tables on a layer are full or when the invalid data ratio is too high. Specifically, for each ordered table in a multi-layered layer, the invalid data ratio of its key-value pairs is stored. If the invalid data ratio reaches 50%, the ordered table is merged to the next layer.

[0163] Multi-level cache structure 1150: includes a hierarchical structure of buffers, segmented memory tables, and multi-layer ordered tables, combined with cache optimization, intelligent write strategy, and crash consistency management to improve the response speed and stability of the storage system.

[0164] Write Strategy 1151: When writing, writes are only written to the M-Tree's ring buffer. A work queue thread periodically monitors the cache status in the background. When the system is idle or the cache is nearing saturation, aggregated data is transferred in batches to the segmented in-memory table, reducing the overhead of frequent small writes. In special cases, such as high-priority or urgent requests or large sequential writes, writes are bypassed by writing directly to the in-memory table segment, bypassing the in-memory buffer.

[0165] based on Figure 11 The system architecture shown, Figure 12 FIG. 1 shows a flow chart of a data processing method for block storage according to another embodiment of the present disclosure. Figure 11As shown, the data processing method includes at least steps S1210 to S1290, which are described in detail as follows.

[0166] In step S1210 , based on the received data write request, it is determined whether the data has a high priority.

[0167] In step S1220, if it is a high priority, a hash operation is performed on the data to determine the sequence number of the segment memory table to be written.

[0168] In step S1230, the data is directly inserted into the segment memory table corresponding to the determined segment memory table sequence number.

[0169] In step S1240 , if it is not a high priority, the data is written into the ring buffer.

[0170] In step S1250, it is determined whether the data in the buffer is full.

[0171] In step S1260, if the data in the buffer is full, the data in the buffer is transferred in batches to the segment memory table.

[0172] In step S1270, it is determined whether the amount of data in the segmented memory table has reached a preset threshold.

[0173] In step S1280, if the amount of data in the segmented memory table has reached a preset threshold, a flush operation is triggered to persist the data to the disk or the next-level storage structure.

[0174] In step S1290, if the amount of data in the segmented memory table does not reach the preset threshold, the relevant lock resources are released and the current writing process is completed.

[0175] Therefore, based on Figure 12 The illustrated embodiment fully utilizes the multi-level cache architecture and adaptive scheduling mechanism provided by the embodiments of the present disclosure, thereby achieving efficient data writing and storage management.

[0176] The following describes an apparatus embodiment of the present disclosure, which can be used to execute the data processing method for block storage in the above-mentioned embodiment of the present disclosure. For details not disclosed in the apparatus embodiment of the present disclosure, please refer to the above-mentioned embodiment of the data processing method for block storage in the present disclosure.

[0177] Figure 13A block diagram of a data processing device for block storage according to an embodiment of the present disclosure is shown. For ease of explanation, certain steps of the above method are described as corresponding to modules. It should be understood that the corresponding modules that perform one or more steps of the above method can be one or more hardware modules specifically configured to perform the corresponding steps, or can be implemented by a processor configured to perform the corresponding steps, or can be stored in a computer-readable medium for implementation by a processor, or can be implemented by some combination thereof.

[0178] Reference Figure 13 As shown, the data processing device for block storage according to an embodiment of the present disclosure includes a construction module 1310 , a determination module 1320 , a writing module 1330 and a processing module 1340 .

[0179] The construction module 1310 is used to construct a mapping index tree, which is used to uniformly manage the unified mapping relationship between logical block addresses and physical block addresses. The mapping index tree is a multi-level cache architecture including sequentially arranged cache layers, segmented memory tables, and multi-layer ordered tables.

[0180] The determination module 1320 is configured to determine a storage priority of the data to be written according to the received data write request, wherein the storage priority is used to represent the importance of the data to be written;

[0181] The writing module 1330 is configured to write the data to be written into the cache layer or the segmented memory table for storage according to the storage priority; and

[0182] The processing module 1340 is configured to merge and migrate the stored data to a storage space at a next level when the stored data in the segmented memory table or the multi-layer ordered table meets a merge migration rule.

[0183] In some embodiments of the present disclosure, the data to be written is written to the cache layer or the segmented memory table for storage according to the storage priority, including: when the storage priority is the first level, the data to be written is written to the cache layer for storage; when the storage priority is the second level, the data to be written is written to the segmented memory table for storage, and the importance of the second level is higher than the importance of the first level.

[0184] In some embodiments of the present disclosure, the processing module 1340 is used to: determine a first proportion of the storage data in the segmented memory table to its own total storage capacity; and merge and migrate the storage data in the segmented memory table whose first proportion is greater than or equal to the first capacity threshold to the multi-layer ordered table.

[0185] In some embodiments of the present disclosure, the processing module 1340 is further configured to: obtain system load information; and determine the first capacity threshold according to the system load information.

[0186] In some embodiments of the present disclosure, the processing module 1340 is further configured to: for a multi-layer ordered table, n The number of ordered lists of the level reaches the threshold, and L n When the second ratio of the storage data in the ordered table of the level to the total storage capacity of the level is greater than or equal to the second capacity threshold, n At least some ordered lists in the hierarchy are selected as ordered lists to be merged, where n is an integer ≥ 0; n The storage data of the hierarchical ordered table to be merged is the same as L n+1 The storage data of the hierarchical ordered table to be merged are merged and sorted in the order of the logical block address to obtain a temporary ordered table, where L n+1 The hierarchical ordered table to be merged is the key value range and L n The key value ranges of the ordered tables to be merged at the level have intersecting ordered tables; write the temporary ordered tables into L n+1 level, and discard L n Level and L n+1 The ordered list to be merged in the hierarchy.

[0187] In some embodiments of the present disclosure, the processing module 1340 is also used to: determine the data failure ratio corresponding to the ordered table based on the number of failed key-value pairs and the total number of key-value pairs in the ordered table; when the data failure ratio reaches a failure ratio threshold, merge and migrate the stored data in the ordered table to the ordered table of the next level.

[0188] In some embodiments of the present disclosure, the mapping index tree is integrated into the kernel block layer to eliminate duplicate logging and mapping management.

[0189] In some embodiments of the present disclosure, the processing module 1340 is further used to: determine the logical block address to be queried based on the received data query request; based on the logical block address, query in each storage space in the order of the cache layer, segmented memory table and multi-layer ordered table; and based on the query result, feedback the physical block address corresponding to the logical block address.

[0190] The present disclosure also provides an electronic device. Figure 14 A schematic diagram showing a hardware implementation using a processing system is shown.

[0191] like Figure 14As shown, the hardware structure of electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc. Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of illustration, the figure only uses a single connecting line, but this does not mean that there is only one bus or only one type of bus.

[0192] The present disclosure also provides a readable storage medium, in which a computer program is stored, and the computer program is used to implement the above-mentioned method when executed by a processor. "Readable storage medium" can be any device that can contain storage, communication, dissemination or transmission programs for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0193] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part using software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the process or function of the present disclosure is performed in whole or in part.

[0194] A computer program or instruction can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instruction can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any accessible medium or a data storage device such as a server or data center that integrates one or more accessible media. The accessible medium can be a magnetic medium such as a floppy disk, hard disk, or magnetic tape; an optical medium such as a digital video disk; or a semiconductor medium such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or can include both volatile and non-volatile types of storage media.

[0195] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0196] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0197] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0198] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0199] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in a suitable manner in any one or more embodiments / methods or examples. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and the features of different embodiments / methods or examples, unless they are contradictory.

[0200] Those skilled in the art will appreciate that the above embodiments are merely intended to clearly illustrate the present disclosure and are not intended to limit the scope of the present disclosure. Other changes or modifications may be made based on the above disclosure, and such changes or modifications are still within the scope of the present disclosure.

Claims

1. A data processing method for block storage, characterized in that: include: Constructing a mapping index tree, wherein the mapping index tree is used to uniformly manage the mapping relationship between logical block addresses and physical block addresses, and the mapping index tree is a multi-level cache architecture including sequentially arranged cache layers, segmented memory tables, and multi-layer ordered tables; Determining a storage priority of the data to be written according to the received data write request, wherein the storage priority is used to represent the importance of the data to be written; Writing the data to be written into the cache layer or the segmented memory table for storage according to the storage priority; as well as, When the storage data in the segmented memory table or the multi-layer ordered table meets the merge migration rule, the storage data is merged and migrated to the storage space of the next level; Writing the data to be written into the cache layer or the segmented memory table for storage according to the storage priority includes: When the storage priority is the first level, writing the data to be written into the cache layer for storage; When the storage priority is the second level, the data to be written is written into the segmented memory table for storage, and the importance of the second level is higher than that of the first level; When the stored data in the segmented memory table or the multi-layer ordered table meets the merge migration rule, the stored data is merged and migrated to the storage space of the next level, the method includes: Determining a first ratio of the storage data in the segmented memory table to its own total storage capacity; Merge and migrate the stored data in the segmented memory table whose first ratio is greater than or equal to the first capacity threshold into the multi-layer ordered table; When the stored data in the segmented memory table or the multi-layer ordered table meets the merge migration rule, the stored data is merged and migrated to the storage space of the next level, the method further includes: For multi-layer ordered tables, in L n The number of ordered lists of the level reaches the threshold, and L n When the second ratio of the storage data in the ordered table of the level to the total storage capacity of the level is greater than or equal to the second capacity threshold, n At least some ordered lists in the hierarchy are selected as ordered lists to be merged, where n is an integer ≥ 0; L n The storage data of the hierarchical ordered table to be merged is the same as L n+1 The storage data of the hierarchical ordered table to be merged are merged and sorted in the order of the logical block address to obtain a temporary ordered table, where L n+1 The hierarchical ordered table to be merged is the key value range and L n The key value ranges of the hierarchical ordered tables to be merged overlap with each other; Write the temporary ordered table into L n+1 level, and discard L n Level and L n+1 The ordered list to be merged in the hierarchy.

2. The method according to claim 1, wherein The method further comprises: Get system load information; The first capacity threshold is determined according to the system load information.

3. The method according to claim 1, wherein The method further comprises: Determine the data failure ratio corresponding to the ordered table based on the number of invalid key-value pairs and the total number of key-value pairs in the ordered table; When the data failure ratio reaches a failure ratio threshold, the stored data in the ordered table is merged and migrated to an ordered table of a next level.

4. The method according to any one of claims 1 to 3, wherein The mapping index tree is integrated into the kernel block layer to eliminate duplicate logging and mapping management.

5. The method according to any one of claims 1 to 3, wherein The method further comprises: Determine the logical block address to be queried according to the received data query request; According to the logical block address, query is performed in each storage space in the order of the cache layer, the segmented memory table and the multi-layer ordered table; According to the query result, the physical block address corresponding to the logical block address is fed back.

6. An electronic device, characterized in that: include: a memory storing execution instructions; as well as, A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the method according to any one of claims 1 to 5.

7. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images