A method and system for key-value database data layout based on computing storage device arrays

Abstract

This application relates to the field of key-value database technology, and discloses a method and system for key-value database data arrangement based on a computing storage device array. It is applied to a hardware architecture including a host CPU, host memory, and a computing storage device array. Each computing storage device in the array has a built-in merge computation acceleration kernel, including: mapping file partitions to computing storage devices at each layer based on a data placement strategy, using a sorted string table file as the smallest granularity; dynamically determining the target computing storage device range for the output sorted string table file based on the data placement strategy; and solving for the number of files written to each computing storage device based on the target computing storage device range using an integer linear programming model. This application improves the throughput of the key-value database by better utilizing the expanded computing resources under the computing storage device array, ensuring sufficient resources for front-end read and write operations.

Description

Technical Field

[0001] This application relates to the field of key-value database technology, and for example to a key-value database data arrangement method and system based on a computing storage device array. Background Technology

[0002] Key-value databases based on LSM (Log-Structured Merge-tree) are widely used in modern data-intensive systems due to their efficient data management. They first cache written data in a memtable in memory, and once a threshold is reached, convert it to an immutable memtable and flush it to disk to form a sorted string table file (SSTable), thus transforming random writes into sequential writes and improving performance.

[0003] LSM trees employ a hierarchical storage structure, distributing data hierarchically across memory and disk. The background performs compaction calculations to merge data into deeper levels, maintaining data timeliness and reducing read amplification. Shallow levels use a tiered strategy (fast writes but high read I / O overhead), while deeper levels use a hierarchical strategy (efficient reads but higher compaction overhead for writes). However, background compaction calculations consume significant system resources, leading to insufficient resources for foreground operations and severely impacting overall throughput. Currently, compute storage devices (CSDs) reduce redundant transmission due to independent internal data paths. They integrate programmable computing units within SSDs, supporting the offloading of some data processing logic to the storage end, achieving "near-data computation." However, existing research only addresses the scheduling framework for a single CSD and has not yet solved the performance scalability problem under CSD arrays. Therefore, there is an urgent need for a key-value database data arrangement method based on compute storage device arrays to solve the above problems.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0006] This disclosure provides a method and system for arranging key-value database data based on a computing storage device array, in order to solve the technical problem that the performance of computing storage device arrays in current LSM tree-based key-value databases cannot be effectively scaled.

[0007] In some embodiments, a key-value database data arrangement method based on a computing storage device array is applied to a hardware architecture including a host CPU, host memory, and a computing storage device array, wherein each computing storage device in the computing storage device array has a built-in merged computing acceleration kernel. The method includes the following steps:

[0008] Using the sorted string table file as the smallest granularity, the file partitioning and computing storage device mapping are completed at each level of the LSM tree according to the data placement strategy. The data placement strategy is to adopt a partitioning method that matches the layout strategy of different levels of the key-value database.

[0009] In response to merged computations, the target computational storage device range for the output sorted string table file is dynamically determined through a data placement strategy.

[0010] Based on the target computing storage device range, the number of sorted string table files written to each computing storage device is solved by an integer linear programming model to achieve balanced data distribution.

[0011] In some embodiments, the data placement strategy includes:

[0012] For hierarchical layouts using a layered strategy, the sorted string table file is evenly distributed to each computing and storage device in a round-robin manner.

[0013] For hierarchical layouts, partitions are created based on key ranges to ensure that different computing and storage devices do not have overlapping key value ranges at the same level.

[0014] In some embodiments, in response to merge computation, the target computational storage device range for the output sorted string table file is dynamically determined by a data placement strategy, including:

[0015] Based on the computing storage device where the existing sorted string table file in the merged input hierarchy is located, determine the preliminary computing storage device range where the output sorted string table file can be written.

[0016] If the input sort string table file holds the minimum or maximum key range in the hierarchy to be merged on the compute storage device, then extend the initial compute storage device interval boundary to the adjacent compute storage device;

[0017] The existence of the sorted string table file in the computing storage device after the extended boundary is detected, and the target computing storage device range is determined based on the detection results.

[0018] In some embodiments, the initial calculation storage device range is the union of the computing storage devices where the sorted string table files in the merged input hierarchy are located.

[0019] In some embodiments, if the input sorting string table file holds the minimum or maximum key range in the hierarchy to be merged on the compute storage device, then extending the initial compute storage device interval boundary to adjacent compute storage devices includes:

[0020] When the input sort string table file holds the largest key value in the hierarchy to be merged on the compute storage device, the initial compute storage device range is extended to the right-side adjacent compute storage device of the current compute storage device;

[0021] When the input sort string table file holds the smallest key value in the hierarchy to be merged on the compute storage device, the initial compute storage device range is extended to the left adjacent compute storage device of the current compute storage device.

[0022] In some embodiments, detecting the existence of a sorted string table file in the computing storage device after the extended boundary, and determining the target computing storage device range based on the detection result, includes:

[0023] The internal data path is used to detect whether the computing storage device after the extended boundary contains a sorted string table file in the hierarchy to be merged;

[0024] If a sorted string table file is included, stop expanding the boundary;

[0025] If the sorted string table file is not contained, the boundary continues to expand in the same direction until a computing storage device containing a sorted string table file is detected or the physical disk boundary is reached, thus obtaining a valid computing storage device range boundary.

[0026] The target computing storage device range is determined based on the final boundary.

[0027] In some embodiments, based on the target computing storage device range, the number of sorted string table files written to each computing storage device is solved using an integer linear programming model to achieve balanced data distribution, including:

[0028] A value function is established with the objective of minimizing the difference in the average load at each partition point within the target storage device interval. The value function is used to quantify the load balancing status.

[0029] To set constraints for the allocation scheme, an integer linear programming model is constructed with the value function as the optimization objective and the constraints as the boundaries.

[0030] Solve the integer linear programming model to obtain the number of sorted string table files that should be written to each computing storage device within the target computing storage device range.

[0031] In some embodiments, the value function is expressed as:

[0032]

[0033] in, Indicates computing storage device The number of sorted string table files already exists. This indicates that the data to be merged needs to be written to the computing storage device. The number of sorted string table files. Calculate the total number of storage devices within the target storage device range.

[0034] In some embodiments, the constraints include:

[0035] The number of sorted string table files received by each computing storage device within the target computing storage device range is a non-negative integer;

[0036] The total number of files in the output sorted string table is equal to the total number of files generated in this merge calculation.

[0037] Compute storage devices outside the target compute storage device range do not receive any output sorted string table files.

[0038] In some embodiments, a key-value database data arrangement system based on a computing storage device array is used to execute any of the key-value database data arrangement methods based on a computing storage device array described above.

[0039] The key-value database data arrangement method and system based on computing storage device arrays provided in this disclosure can achieve the following technical effects:

[0040] This application applies to a hardware architecture comprising a host CPU, host memory, and a compute storage device array, where each compute storage device in the array has a built-in merge acceleration kernel. Based on data placement strategies and the characteristics of different levels of the LSM tree, file partitions are mapped to compute storage devices. Different levels of the key-value database employ partitioning methods matching their respective layout strategies. When a merge operation occurs, the key ranges of the input sorted string table files are highly correlated. Through dynamic target interval decision-making, it is ensured that the sorted string table files participating in the merge are concentrated in the same or adjacent compute storage devices. Combined with independent data paths within the compute storage devices, data is directly transferred between the acceleration kernel and the storage medium, avoiding cross-disk data migration and reducing host memory bandwidth usage and I / O latency. After determining the target compute storage device interval, the number of sorted string table file writes is dynamically allocated by solving an integer linear programming model, balancing the data volume across compute storage devices and preventing some compute storage devices from becoming overloaded while others remain idle due to data skew, thus maximizing the utilization of the parallel merge acceleration kernels within the array. In summary, this application significantly reduces the CPU and memory resource consumption of the host by making better use of the computing resources expanded under the computing storage device array, while ensuring that the foreground read and write operations have sufficient resources, reducing the "write pause" phenomenon, and ultimately improving the throughput of the key-value database.

[0041] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0042] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein:

[0043] Figure 1 This is a flowchart of a key-value database data arrangement method based on a computing storage device array, provided in an embodiment of this disclosure;

[0044] Figure 2 This is a schematic diagram of data arrangement in a hierarchical strategy provided in an embodiment of this disclosure;

[0045] Figure 3 This is a schematic diagram of data arrangement in a hierarchical strategy provided in an embodiment of this disclosure;

[0046] Figure 4 This is a flowchart illustrating the generation of a target computing storage device region provided in an embodiment of this disclosure;

[0047] Figure 5 This is an overall design diagram of the framework for key-value database data arrangement based on computing storage device arrays provided in this embodiment. Detailed Implementation

[0048] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0049] The terms "first," "second," etc., used in the embodiments of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0050] Unless otherwise stated, the term "multiple" means two or more.

[0051] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0052] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0053] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0054] LSM-tree integrates programmable computing units within its SSDs via compute storage devices (CSDs), enabling the offloading of some data processing logic to the storage end, achieving "near-data computation." Since the compute accelerator and storage media are integrated within the same device, data transfer between them enjoys advantages such as high bandwidth, low latency, and low power consumption. This significantly reduces data movement overhead and minimizes the consumption of host resources. Therefore, by offloading compaction tasks from key-value databases using compute storage devices, the overall database throughput is improved; compaction can also be understood as compression and organization. Currently, existing research only addresses the scheduling framework of a single compute storage device and has not yet solved the performance scaling problem under compute storage device arrays.

[0055] To address the aforementioned problems, this disclosure provides a key-value database data arrangement method and system based on a computing storage device array. This application applies to a hardware architecture including a host CPU, host memory, and a computing storage device array. While maintaining similarity to the original LSM tree structure, this architecture replaces the traditional disk array with a computing storage device array, and each computing storage device has a merged task computing acceleration core on its internal acceleration device. The key-value database data arrangement method provided by this disclosure will be described in detail below with reference to the accompanying drawings.

[0056] Figure 1 This is a flowchart illustrating a key-value database data arrangement method based on a computing storage device array, provided in this embodiment of the disclosure. Figure 1 As shown, the key-value database data arrangement method includes the following steps:

[0057] S101: Using the sorted string table file as the smallest granularity, complete the file partitioning and computing storage device mapping at each level of the LSM tree according to the data placement strategy.

[0058] In some embodiments, the data placement strategy is to adopt a partitioning method that matches the layout strategy of different levels of the key-value database. In LSM-based key-value databases, two data organization strategies are typically included: tiering and leveling. In a tiered data organization strategy, the sorted string table files within a tier allow overlapping key-value ranges, but the key-values ​​within a single sorted string table file remain ordered. In a leveling data organization strategy, the sorted string table files within a tier do not allow overlapping key-value ranges, and the key-values ​​within each sorted string table file remain ordered. For both of these data organization strategies, the data placement strategy includes: for tiered layouts, distributing the sorted string table files evenly across computing storage devices in a round-robin manner; for leveling layouts, partitioning based on key ranges to ensure no overlap of key-value ranges across different computing storage devices at the same level, thus ensuring that key-value ranges are not only globally ordered within the tier but also ordered within a single computing storage device.

[0059] Figure 2 This is a schematic diagram of data arrangement in a hierarchical strategy provided in an embodiment of this disclosure. Combined with... Figure 2This example illustrates the data arrangement within a hierarchical structure using a layered strategy. Taking the shallow layer L0 as an example, assume there are four computing storage devices: Computing Storage Device 0, Computing Storage Device 1, Computing Storage Device 2, and Computing Storage Device 3. Five new sorted string table files (sorted string table file A, sorted string table file B, sorted string table file C, sorted string table file D, and sorted string table file E) need to be written to the L0 layer using a round-robin approach. Sorted string table file A is placed on Computing Storage Device 0, sorted string table file B on Computing Storage Device 1, sorted string table file C on Computing Storage Device 2, sorted string table file D on Computing Storage Device 3, and sorted string table file E on Computing Storage Device 0. Here, sorted string table file E returns to the first computing storage device in the round-robin process. When a new sorted string table file needs to be written, it is placed according to the same round-robin rule as sorted string table file B.

[0060] Figure 3 This is a schematic diagram of data arrangement in a hierarchical strategy provided in an embodiment of this disclosure. Combined with... Figure 3 This example illustrates the data arrangement in a hierarchical strategy. Hierarchical strategies are typically used in deep layers, such as L2. Assume there are four compute storage devices: compute storage device 0, compute storage device 1, compute storage device 2, and compute storage device 3. The key-value space ranges from 0 to 999. Existing sorted string table files in L2 have divided the entire key-value space into four consecutive, non-overlapping intervals: [0, 249], [250, 499], [500, 749], and [750, 999]. Now, three new sorted string table files need to be written to the L2 layer using a hierarchical strategy: sorted string table file A contains the key-value pair [100, 230], sorted string table file B contains the key-value pair [300, 450], and sorted string table file C contains the key-value pair [700, 749]. The string table file A, whose key value [100, 230] belongs to [0, 249], is stored in computing storage device 0. The string table file B, whose key value [300, 450] belongs to [250, 499], is stored in computing storage device 1. The string table file C, whose key value [700, 749] belongs to [500, 749], is stored in computing storage device 2.

[0061] The above ensures that the sorting string table files included during merge calculations are located in the same or adjacent CSDs as much as possible, reducing cross-disk data transfers caused by cross-CSDs and making full use of the independent data transfer paths within the compute memory.

[0062] S102: In response to merged computation, dynamically determine the target computational storage device range for the output sorted string table file based on the data placement strategy.

[0063] To maintain the data organization strategy at the corresponding level and to merge sorted string table files into deeper levels, the key-value database backend triggers a merge calculation. In the data placement strategy disclosed in this application, the merge calculation becomes the only computational process affecting the load on different computing storage devices. To maintain the globally ordered nature of the keys during the merge process, the data placement strategy also dynamically determines the target computing storage device range for the output sorted string table file.

[0064] In some embodiments, the step of dynamically determining the target computing storage device range for the output sorted string table file in response to the merge computation via a data placement strategy includes: determining a preliminary computing storage device range for which the output sorted string table file can be written based on the computing storage device where the sorted string table file already exists in the merged input hierarchy; if the input sorted string table file holds the minimum or maximum key range in the merged hierarchy on the computing storage device, then extending the boundary of the preliminary computing storage device range to adjacent computing storage devices; detecting the existence of the sorted string table file in the computing storage device after the extended boundary, and determining the target computing storage device range based on the detection result.

[0065] Figure 4 This is a flowchart illustrating the generation of a target computing storage device region provided in an embodiment of this disclosure, such as... Figure 4 As shown, the merging calculation begins first, entering Phase 1 to estimate the preliminary computational storage device range. Based on the computational storage devices where the existing sorted string table files in the input levels to be merged are located, the preliminary computational storage device range where the output sorted string table file can be written is determined. In some embodiments, the preliminary computational storage device range is the union of the computational storage devices where the sorted string table files in the input levels to be merged are located.

[0066] Next, in stage 2, interval expansion is performed based on the key range boundaries. In some embodiments, when the input sorted string table file holds the largest key value in the level to be merged on the compute storage device, the initial compute storage device interval is expanded to the right adjacent compute storage device of the current compute storage device. When the input sorted string table file holds the smallest key value in the level to be merged on the compute storage device, the initial compute storage device interval is expanded to the left adjacent compute storage device of the current compute storage device.

[0067] Then, in stage 3, it checks whether the compute storage device after the extended boundary contains a sorted string table file. In some embodiments, the internal data path is used to detect whether the compute storage device after the extended boundary contains a sorted string table file in the level to be merged. If a sorted string table file is found, the boundary extension stops. If no sorted string table file is found, the boundary continues to extend in the same direction until a compute storage device containing a sorted string table file is detected or the physical disk boundary is reached, thus obtaining a valid compute storage device interval boundary.

[0068] Finally, the target computing storage device range is determined based on the final boundary.

[0069] Figure 5 This is an overall design diagram of the framework for key-value database data arrangement based on computing storage device arrays provided in this embodiment. Figure 5 The process of determining the target computational storage device range for the output sorted string table file is visually illustrated through examples. The first stage is stage 1: estimating the initial computational storage device range. This involves determining the output range based on the input. Based on the computational storage devices where sorted string table files already exist in the merged input levels, the writeable computational storage device range for the output sorted string table file is initially determined. For example, in Example 1, the original storage locations of the sorted string table files in the merged input levels include computational storage devices 0 and 1, so the initial computational storage device range is computational storage devices 0 and 1. The second stage is stage 2: expanding the range based on key boundaries. This involves checking if the file contains boundary key values. If the input sorted string table file holds the minimum or maximum key boundary value for that level on a certain computational storage device, the output range is expanded to adjacent computational storage devices to maintain key order continuity. For example, in Example 2, sorted string table file number 3 in the merged input sorted string table files contains the largest key value in the merged level in that computational storage device. The initial computational storage device range, originally only for computational storage device 1, needs to be expanded to the right to include computational storage devices 1 and 2. Finally, there is stage 3: boundary verification. This checks whether there are no files at the current level. If stage 2 triggers boundary expansion, it determines whether the expanded compute storage device contains a sorted string table file at the corresponding level. If not, expansion continues until an adjacent compute storage device containing a sorted string table file is encountered or the physical disk boundary is reached, ultimately locking onto the valid output target compute storage device range. As shown in Example 2, after stage 2 expands to compute storage device 2, since compute storage device 2 does not contain a sorted string table file at the level to be merged, it needs to be expanded again to compute storage device 3. Compute storage device 3 contains a sorted string table file, and the expansion ends.

[0070] The following section provides a further detailed explanation of the data placement strategy, using the algorithm's pseudocode.

[0071]

[0072] In summary, this application maps file partitions to compute storage devices based on the different levels of the LSM tree according to the data placement strategy. Different levels of the key-value database employ partitioning methods that match their respective layout strategies. When a merge occurs, the key ranges of the input sorted string table files are highly correlated. Through dynamic target interval decision-making, it is ensured that the sorted string table files participating in the merge are concentrated on the same or adjacent compute storage devices. Combined with independent data paths within the compute storage devices, data is directly transferred between the acceleration kernel and the storage medium, avoiding cross-disk data migration and reducing host memory bandwidth usage and I / O latency.

[0073] S103: Based on the target computing storage device range, solve the number of sorted string table files written in each computing storage device using an integer linear programming model to achieve balanced data distribution.

[0074] In some embodiments, the step of solving for the number of sorted string table files written to each computing storage device based on the target computing storage device interval using an integer linear programming model to achieve balanced data distribution includes: establishing a value function with the objective of minimizing the difference in the average load at each partition point within the target computing storage device interval, wherein the value function is used to quantify the load balancing state; setting constraints for the allocation scheme, constructing an integer linear programming model with the value function as the optimization objective and the constraints as the boundaries; and solving the integer linear programming model to obtain the number of sorted string table files that should be written to each computing storage device within the target computing storage device interval.

[0075] In some embodiments, the expression for the value function is:

[0076]

[0077] in, Indicates computing storage device The number of sorted string table files already exists. This indicates that the data to be merged needs to be written to the computing storage device. The number of sorted string table files. The total number of computing storage devices within the target computing storage device range is calculated. For each computing storage device (numbered...), ... First calculate all devices to its left (0 to 1). -1) average load (total number of sorted string table files ÷ number of devices), then calculate the load of all devices to its right ( arrive Given the average load of -1), calculate the difference between these two averages, square the difference, and finally sum the squared differences of all storage devices. The goal is to minimize this sum.

[0078] In some embodiments, the constraints include: the number of sorted string table files received by each computing storage device within the target computing storage device range is a non-negative integer; the total number of output sorted string table files is equal to the total number of files generated in this merge calculation; and computing storage devices outside the target computing storage device range do not receive any output sorted string table files.

[0079] The following section provides a more detailed explanation of the load balancing strategy, using pseudocode as an example.

[0080]

[0081] In this application, after determining the target computing and storage device range, the number of writes to the sorted string table file is dynamically allocated by solving an integer linear programming model. This balances the data volume of each computing and storage device, prevents some computing and storage devices from becoming overloaded due to data skew while others remain idle, and maximizes the use of parallel merging within the array to accelerate the kernel.

[0082] Based on the same inventive concept as the key-value database data arrangement method based on computing storage device array, this application also discloses a key-value database data arrangement system based on computing storage device array in some embodiments. The system is used to execute the key-value database data arrangement method based on computing storage device array disclosed in any of the above embodiments, and the process will not be described in detail here.

[0083] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions executed by a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0084] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for descriptive purposes only and is not intended to limit the scope of protection. As used in the description herein, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used herein means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0085] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0086] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed between each other may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

Claims

1. A method for arranging key-value database data based on a computing storage device array, characterized in that, Applied to a hardware architecture comprising a host CPU, host memory, and an array of compute storage devices, wherein each compute storage device in the array has a built-in merged compute acceleration core, the method includes the following steps: Using the sorted string table file as the smallest granularity, file partitioning and mapping to computing and storage devices are completed at each level of the LSM tree according to the data placement strategy. The data placement strategy is a partitioning method that matches the layout strategy of different levels of the key-value database, including: for levels with a hierarchical layout strategy, the sorted string table file is evenly distributed to each computing and storage device in a round-robin manner; for levels with a hierarchical layout strategy, partitioning is performed according to the key range so that there is no overlap of key value ranges on different computing and storage devices at the same level. In response to the merge computation, the target computing storage device range for the output sorted string table file is dynamically determined through a data placement strategy. This includes: determining the initial computing storage device range within which the output sorted string table file can be written, based on the computing storage devices where existing sorted string table files in the merged hierarchy reside; if the input sorted string table file holds the minimum or maximum key range in the merged hierarchy on the computing storage device, then extending the initial computing storage device range boundary to adjacent computing storage devices; detecting the existence of sorted string table files in the extended computing storage devices, and determining the target computing storage device range based on the detection results. Based on the target computing storage device range, the number of sorted string table files written to each computing storage device is solved using an integer linear programming model to achieve balanced data distribution. This includes: establishing a value function with the objective of minimizing the difference in the average load at each partition point within the target computing storage device range. The value function is used to quantify the load balancing state, and its expression is: In the formula, Indicates computing storage device i The number of sorted string table files already exists. This indicates that the data to be merged needs to be written to the computing storage device. i The number of sorted string table files. n Define the total number of computing storage devices within the target computing storage device range; set constraints for the allocation scheme, construct an integer linear programming model with the value function as the optimization objective and the constraints as the boundaries; solve the integer linear programming model to obtain the number of sorted string table files that should be written to each computing storage device within the target computing storage device range.

2. The key-value database data arrangement method based on a computing storage device array according to claim 1, characterized in that, The preliminary computational storage device range is the union of the computational storage devices where the sorted string table files in the merged input hierarchy are located.

3. The key-value database data arrangement method based on a computing storage device array according to claim 1, characterized in that, If the input sorting string table file holds the minimum or maximum key range in the hierarchy to be merged on the computing storage device, then the preliminary computing storage device interval boundary is extended to adjacent computing storage devices, including: When the input sort string table file holds the largest key value in the hierarchy to be merged on the compute storage device, the initial compute storage device range is extended to the right-side adjacent compute storage device of the current compute storage device; When the input sort string table file holds the smallest key value in the hierarchy to be merged on the compute storage device, the initial compute storage device range is extended to the left adjacent compute storage device of the current compute storage device.

4. The key-value database data arrangement method based on a computing storage device array according to claim 1, characterized in that, The existence of the sorted string table file in the computing storage device after detecting the extended boundary is used to determine the target computing storage device range based on the detection results, including: The internal data path is used to detect whether the computing storage device after the extended boundary contains a sorted string table file in the hierarchy to be merged; If a sorted string table file is included, stop expanding the boundary; If the sorted string table file is not contained, the boundary continues to expand in the same direction until a computing storage device containing a sorted string table file is detected or the physical disk boundary is reached, thus obtaining a valid computing storage device range boundary. The target computing storage device range is determined based on the final boundary.

5. The key-value database data arrangement method based on a computing storage device array according to claim 1, characterized in that, The constraints include: The number of sorted string table files received by each computing storage device within the target computing storage device range is a non-negative integer; The total number of files in the output sorted string table is equal to the total number of files generated in this merge calculation. Computational storage devices outside the target computing storage device range do not receive any output sorted string table files.

6. A key-value database data layout system based on a computing storage device array, characterized in that, The system is used to execute the key-value database data arrangement method based on a computing storage device array as described in any one of claims 1 to 5.

Images