A method and apparatus for writing based on a computable storage device

Abstract

This invention relates to the field of database systems and storage systems, and in particular to a writing method and apparatus based on a computable storage device. The method includes receiving a write transaction for a database; when the database is under non-write-intensive load, writing the flushed data corresponding to the write transaction to the actual storage area based on a B+ tree index structure; dividing the entire key space of the database into multiple partitions according to the range of keys, and for each partition, storing the key value falling into the partition and a pointer to the actual data in the leaf nodes of an adaptive radix tree, obtaining a multi-instance adaptive radix tree index structure; when the database is under write-intensive load, writing the flushed data corresponding to the write transaction to the virtual storage area of ​​the computable storage device based on the multi-instance adaptive radix tree index structure, and having the computable storage device complete the process of migrating the flushed data from the virtual storage area to the actual storage area. This invention can improve write performance.

Description

Technical Field

[0001] This invention relates to the field of database systems and storage systems, and in particular to a writing method and apparatus based on a computable storage device. Background Technology

[0002] In existing technologies, database indexes typically employ two main structures: B+ trees or log-structured-merge-trees (LSM-trees). B+ tree indexes are widely used in relational databases. They offer stable query performance and support ordered retrieval, but they have significant drawbacks in write-intensive scenarios: random input / output (I / O) and write amplification issues. Each insert or update operation may trigger random writes such as node splits, causing a single write request to trigger multiple disk rewrites, resulting in high write amplification. In addition, B+ trees require a write-ahead log (WAL) to ensure transaction consistency for each data change. This means that writing a single piece of data often requires two write operations: one to the log and one to the index page. Therefore, under heavy and frequent writes, B+ trees often cannot fully utilize the bandwidth of modern high-speed storage media due to the performance bottleneck of random I / O. This structure is inefficient in write-heavy and read-light scenarios, and memory usage increases with data volume (index nodes need to maintain keys and pointers to record locations; as data volume increases, the index hierarchy and number of nodes increase, leading to higher memory and cache pressure).

[0003] To optimize write performance, many NoSQL databases and new storage engines have adopted LSM-Tree. LSM-Tree improves write throughput by converting discrete random writes into batched sequential writes. Specifically, new data is first written to a buffer in memory and appended to a sequential log file. Then, multiple update batches are merged and written to an ordered Sorted String Table (SSTable) file in the background. However, LSM-Tree also introduces read / write amplification and complex background maintenance overhead. To maintain data order, the LSM mechanism needs to continuously perform compaction and merging in the background to merge sorted files, resulting in the same data being repeatedly read, written, and moved multiple times throughout its lifecycle. Under high write pressure, the LSM mechanism can cause significant write amplification and background merging overhead. This background merging not only causes write amplification and device wear but also interferes with foreground operations. When the data volume is large or the write pressure is extremely high, background compression may not keep up with the foreground write speed, leading to the continuous accumulation of unmerged file segments, which can consume a large amount of storage space and increase query latency. Furthermore, to support crash recovery and consistency, LSM-Tree also needs to maintain a WAL log. Although LSM defers sorted writes to improve throughput, the system still needs to replay the log and rebuild the in-memory table upon a crash, making the recovery process time-consuming. In summary, traditional B+ trees and LSM-Trees each have their advantages and disadvantages: the former ensures strong consistency through immediate updates but has poor write performance, while the latter is write-friendly but has higher read latency and data maintenance complexity, making it difficult to guarantee timely data consistency (such as consistency maintenance during multi-version merges).

[0004] In summary, current technology lacks a method that can improve write performance while maintaining B+Tree read performance.

[0005] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention proposes a writing method and apparatus based on a computable storage device, which can improve the writing performance while maintaining the read performance of B+Tree.

[0007] The embodiments of the present invention adopt the following technical solutions: In a first aspect, the present invention provides a writing method based on a computable storage device, specifically: receiving a write transaction for a database; When the database is under non-write-intensive load, the disk flushing data corresponding to the write transaction is written to the actual storage area based on the B+ tree index structure. The entire key space of the database is divided into multiple partitions according to the range of keys. For each partition, the key value falling into the partition and the pointer to the actual data are stored in the leaf node of the adaptive radix tree, resulting in a multi-instance adaptive radix tree index structure. When the database is under write-intensive load, based on the multi-instance adaptive radix tree index structure, the disk flushing data corresponding to the write transaction is written to the virtual storage area of ​​the computable storage device, and the computable storage device completes the process of migrating the disk flushing data from the virtual storage area to the actual storage area.

[0008] Preferably, when the database is under write-intensive load, based on a multi-instance adaptive radix tree index structure, the data flushed to disk corresponding to the write transaction is written to the virtual storage area of ​​the computable storage device, and the computable storage device completes the process of migrating the data flushed to disk from the virtual storage area to the actual storage area, including: Receive the write transaction and lock it according to its isolation level; The original data is modified based on the information of the write transaction to obtain the flush data, and the B+ tree index structure is updated. Write the modification record corresponding to the write transaction to the WAL log; Submit the write transaction to write the flush data to the virtual storage area, and then migrate the flush data from the virtual storage area to the actual storage area via the migration buffer. Upon receiving the information that the disk data has been stored, update the storage status of the disk data in the WAL log and release the lock resources of the write transaction.

[0009] Preferably, submitting the write transaction, writing the flushed data to the virtual storage area, and migrating the flushed data from the virtual storage area to the physical storage area via a migration buffer includes: The disk flushing data is written into the Cuckoo Filter for recording, and the storage location of the current disk flushing data is recorded in the WAL log. The write request is routed to the corresponding adaptive radix tree index instance based on the key value range of the disk flushing data. In the corresponding adaptive radix tree index instance, an index update is performed, and the index update record is simultaneously appended to the index log of the computable storage device. The disk flushing data is written to the virtual storage area corresponding to the adaptive radix tree index instance in a sequential append manner. The disk flushing data is loaded into the migration buffer of the computeable storage device, and then written from the migration buffer into the actual storage area for permanent storage.

[0010] Preferably, the method further includes: Receive a read transaction and lock it according to its isolation level; Query the cuckoo filter to determine whether the target page corresponding to the read transaction is located in the virtual storage area; If the cuckoo filter indicates that the target page is located in the virtual storage area, a read request is sent to the computable storage device, which then locates and returns the target data page based on the multi-instance adaptive radix tree index structure. If the cuckoo filter does not indicate that the target page is located in the virtual storage area, the target data page is located and returned according to the B+ tree index structure; Receive the commit message after the read transaction is completed, and release the lock resources of the read transaction.

[0011] Preferably, the method further includes: Determine whether the data being flushed to disk belongs to a hot data item or a frequently updated item; If the disk flushing data belongs to a hot data item or a frequently updated item, the disk flushing data is copied into the data item buffer and the disk flushing data in the virtual storage area is retained; when subsequent recycling is triggered, the disk flushing data is appended to the end of the virtual storage area queue in sequence, the original disk flushing data in the virtual storage area is deleted, and the flag bit of the disk flushing data is cleared to zero. If the data being flushed to disk is not a hot data item or a frequently updated item, then the data being flushed to disk will be directly migrated to the actual storage area.

[0012] Preferably, determining whether the disk refresh data belongs to a hot data item or a frequently updated item includes: If the disk data is valid and the update flag of the disk data is not 0, then the disk data is considered to be a frequently updated item. If the disk data is valid and the access flag of the disk data is not 0, then the disk data is considered a hot data item.

[0013] Preferably, the method further includes: Recover committed but incomplete transactions in the B+ tree structure based on WAL logs; Reconstruct multi-instance ART indexes for computeable storage devices using index logs and checkpoint files; Recover the Cuckoo Filter based on log records in the WAL log that contain a field for writing to a computable storage device. Data corresponding to transactions that were committed but not yet stored before the crash are sequentially flushed to the virtual storage area, and the data corresponding to these transactions are then migrated to the physical storage area by the compute-enabled storage device.

[0014] Preferably, for transactions in a questionable state, the method further includes: If the computable storage device confirms that the data corresponding to the transaction in question has been persisted in the internal log of the computable storage device, then the redo logic is executed and the memory mapping and filter state are restored. If the storage device can be calculated to confirm the missing data corresponding to a transaction in a questionable state, then a rollback process is performed on the data corresponding to the transaction in the questionable state.

[0015] In a second aspect, the present invention provides a writing apparatus based on a computationally readable storage device, the apparatus comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the writing method based on the computationally readable storage device in the first aspect.

[0016] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the write method based on a computable storage device as described in the first aspect.

[0017] Compared with existing technologies, the advantages of this invention are as follows: It provides different write paths based on different database workloads, while preserving the read performance of the B+ tree index in the actual storage area. For write-intensive scenarios, the data is written to a computeable storage device, significantly improving write throughput by leveraging the high bandwidth between the computeable storage device and the host. Simultaneously, it implements key-value separation storage based on index design to reduce memory consumption, and introduces a multi-instance adaptive radix tree index to partition write requests, further enhancing write performance. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0019] Figure 1 This is an architecture diagram of a write system based on a computable storage device provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating a write method based on a computable storage device provided in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the write method based on a multi-instance ART index structure provided in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the specific implementation method of step 204 provided in this embodiment of the invention; Figure 5 This is a diagram of the cuckoo filter, CSD index, and storage framework provided in an embodiment of the present invention; Figure 6 This is a flowchart of the write method based on a multi-instance ART index structure provided in an embodiment of the present invention; Figure 7 This is a flowchart illustrating the reading method based on the cuckoo filter provided in an embodiment of the present invention; Figure 8 This is a flowchart of the read method based on multi-instance ART index provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the delayed migration strategy provided in an embodiment of the present invention; Figure 10 This is a flowchart illustrating the method for determining hotspot data provided in an embodiment of the present invention; Figure 11 This is a flowchart illustrating the method for post-crash recovery provided in an embodiment of the present invention; Figure 12 A schematic diagram of a write device based on a computable storage device is provided in an embodiment of the present invention; The reference numerals in the attached figures are as follows: 21: Processor; 22: Memory. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0022] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0023] In the description of this invention, the expression “A and / or B” (where A and B are used to formally represent specific features) will be used. The corresponding expression includes the following three combinations: only A, only B, and a combination of A and B.

[0024] As used in this invention, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from a particular value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0025] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0026] Example 1: Computational Storage Drives (CSDs) are devices that integrate computing power into storage hardware. Examples include intelligent solid-state drives (SSDs) with a System-on-Chip (SoC). These SSDs can perform some data processing closer to the data. This near-data-processing architecture offloads operations such as index maintenance and data filtering from the host to the computational storage device, reducing data movement and context switching between the host and storage, lowering I / O overhead, and fully utilizing the parallel computing capabilities within the device. Current advancements in technologies such as NVMExpress high-speed interfaces and ZNS partitioned storage enable CSDs to efficiently process custom commands in parallel and provide mechanisms like direct log writing and data partitioning to optimize storage management.

[0027] With the explosion of data volume and the increasing demand for high throughput and low latency, existing B+ trees and LSM-Trees (Log-Structured Merge-Trees) have shortcomings under high-intensity write loads. This invention designs a new index structure on CSD and implements write transactions based on the designed index structure to significantly improve the write performance of the database in write-intensive scenarios and enhance data consistency and crash recovery capabilities.

[0028] Specifically, the following issues need to be addressed regarding write performance in write-intensive scenarios and enhancing data consistency and crash recovery capabilities: First, how to reduce the memory footprint of indexes and write amplification; second, how to improve concurrent write throughput; and third, how to ensure the integrity of frequently updated data and rapid recovery after system crashes.

[0029] like Figure 1 As shown, this invention provides a write system based on a computable storage device that is used in conjunction with a write method based on a computable storage device. The overall architecture of the system includes a host side, a CSD side, and a physical storage area. The index structure of the CSD is a multi-instance adaptive radix tree index structure, and the index structure of the physical storage area is a B+ tree index structure. The host side includes a runtime memory pool and a common memory pool. The common memory pool includes a dictionary buffer, a log buffer, a sorting area, a hash area, a data buffer, and a write optimization area. The write optimization area is used to maintain data mapping relationships, perform database runtime load detection, and communicate asynchronously with the CSD and the physical storage area.

[0030] CSD is used to quickly write data to disk sequentially under write-intensive loads. CSD includes a memory pool and a virtual storage area. The memory pool is used for index maintenance, log maintenance, data migration, and data status monitoring. Log maintenance is used to ensure consistency during recovery in case of a crash. Data migration is used to seamlessly migrate data from the virtual storage area to the physical storage area. Data status monitoring is used to keep frequently updated data in the virtual storage area for a longer period of time to reduce migration overhead and reduce frequent erase and write operations in the physical storage area.

[0031] The actual storage area is used to store data according to the original B+ tree index structure. In the B+ tree index structure, each node represents a disk block. Internal nodes (such as disk block 1, disk block 2 and disk block 3) store XX, and leaf nodes (such as disk block 4, disk block 5, disk block 6 and disk block 7) store the index and the actual data. Adjacent leaf nodes are connected by linked list pointers.

[0032] like Figure 2 As shown, Embodiment 1 of the present invention provides a write method based on a computable storage device, which specifically includes the following steps: Step 101: Receive the write transaction for the database.

[0033] The host side detects the database's operating load, which includes non-write-intensive load and write-intensive load. In this invention, the database operates under different loads and executes different write paths when performing write transactions.

[0034] Step 102: When the database is under non-write-intensive load, the disk flushing data corresponding to the write transaction is written to the actual storage area based on the B+ tree index structure.

[0035] When the host detects that the database is under non-write-intensive load, the original write path of the B+ tree is used unchanged, and the flush data is directly written from the data buffer on the host to the actual storage area.

[0036] Here, flushed data refers to the data pages that have been modified in the data buffer on the host side.

[0037] The actual storage area in this invention still uses the B+ index structure, which retains the good read performance of the B+ index structure. When the database is under non-write-intensive load, the host side reads or writes data to the actual storage area, which can guarantee both read performance and write performance.

[0038] Step 103: Divide the entire key space of the database into multiple partitions according to the range of keys. For each partition, store the key value falling into the partition and the pointer to the actual data in the leaf node of the adaptive radix tree to obtain a multi-instance adaptive radix tree index structure.

[0039] This invention separates index storage from data storage in a computable storage device. Specifically, it uses the ART index structure as the basic index structure, stores key values ​​and references to the actual data locations only in the adaptive radix tree index nodes, and stores large data values ​​in an independent log area of ​​the computable storage device. This decouples the index structure from the actual data. Through this key-value separation design, the index structure in the computable storage device is significantly simplified compared to the B+ tree, reducing memory usage and tree node size, improving cache hit rate, and avoiding frequent movement of large data blocks in the index organization, fundamentally reducing write amplification.

[0040] Furthermore, the entire key space of the database is divided into multiple partitions according to the key range. Each partition is managed by an independent ART index instance, where each ART instance corresponds to a different key range. It can receive operations in parallel during writes, reducing lock contention and mutual blocking across partitions. This effectively utilizes the parallel capabilities of the CSD and the host's multi-core processors, improving concurrent write throughput. Compared to a single global index, this multi-instance partitioned index reduces lock contention and update conflicts in single-point hotspots, allowing insert and update operations to be distributed and executed in parallel across different instances, improving concurrent write throughput and scalability.

[0041] Step 104: When the database is under write-intensive load, based on the multi-instance adaptive radix tree index structure, the disk flushing data corresponding to the write transaction is written to the virtual storage area of ​​the computable storage device, and the computable storage device completes the process of migrating the disk flushing data from the virtual storage area to the actual storage area.

[0042] When the host detects that the database is under write-intensive load, this invention uses the computeable storage device as an intermediate high-speed disk write and management layer. The original random disk flushing process facing the actual storage layer is adjusted to a sequential flushing process facing the virtual storage area of ​​the computeable storage device. That is, instead of writing the flushed data to the actual storage area according to the method of step 102, the flushed data is first quickly persisted to the virtual storage area of ​​the computeable storage device in an append write manner, and then the flushed data is written from the virtual storage area to the actual storage area. This improves write throughput and reduces host-side I / O blocking. From the perspective of the database system, after the flushed data is written from the host-side data buffer to the virtual storage area of ​​the CSD, it represents the end of the write transaction process. This end is a virtual end and does not affect the read and write transactions of the database system. From the perspective of the storage system, the write transaction process is only truly ended after the data in the virtual storage area is migrated to the actual storage area.

[0043] When the database is under write-intensive load, the host side interacts with the actual storage area to write data. By leveraging the high bandwidth between the computing storage device and the host, the write throughput is greatly improved, ensuring write performance.

[0044] In this embodiment, different write paths are provided based on different database workloads, while preserving the read performance of the B+ tree index in the actual storage area. For write-intensive scenarios, the data is written to the computeable storage device, leveraging the high bandwidth between the computeable storage device and the host to significantly improve write throughput. Simultaneously, key-value separation storage is implemented based on index design to reduce memory consumption, and a multi-instance adaptive radix tree index is introduced to partition write requests, further enhancing write performance.

[0045] like Figure 3As shown, for databases under write-intensive load, this embodiment of the invention provides a write method based on a multi-instance ART index structure, which specifically includes the following steps: Step 201: Receive the write transaction and lock it according to its isolation level.

[0046] Isolation levels include Read Uncommitted, Read Committed, and Repeatable Read, and different isolation levels correspond to different locks. For example, when the isolation level is Read Uncommitted, the locking step is skipped; when the isolation level is Read Committed, a shared lock is applied; and when the isolation level is Repeatable Read, a range lock is applied.

[0047] Assign a unique identifier to each write transaction to facilitate subsequent recovery and backtracking, and ensure transaction consistency.

[0048] Step 202: Modify the original data according to the information of the write transaction to obtain the flush data and update the B+ tree index structure.

[0049] On the host side, the page node corresponding to the write transaction is found according to the B+ index structure. The data page corresponding to the page node is loaded from the actual storage area into the data buffer. In the data buffer, the original data of the data page is modified (added, deleted, or updated) according to the write transaction to obtain the modified data, which is the flushed data.

[0050] At the same time, update the B+ index structure in the actual storage area according to the modified content.

[0051] Step 203: Write the modification record corresponding to the write transaction to the WAL log.

[0052] WAL (Write-Ahead Log) is the core mechanism of database management systems to achieve transaction durability and fault recovery. It ensures transaction atomicity and data consistency by recording logs before modifying data.

[0053] Before committing a write transaction, the host first writes the corresponding modification record to the WAL log. To ensure data consistency in the event of a crash, the host extends the WAL log fields by adding a dedicated identifier field for whether to write to a computable storage device, which serves as the basis for subsequent database crash recovery.

[0054] Step 204: Submit the write transaction, write the flush data to the virtual storage area, and migrate the flush data from the virtual storage area to the actual storage area via the migration buffer.

[0055] The host side submits a write transaction and performs persistent disk flushing. When there is free storage space in the virtual storage area, it triggers the activation of the computable storage device for write acceleration, writes the flushed data from the host side's data buffer to the CSD's virtual storage area, and maintains the relevant data mapping for real-time updates.

[0056] Step 205: Receive the information that the disk data is stored successfully, update the storage status of the disk data in the WAL log, and release the lock resources of the write transaction.

[0057] After receiving the storage completion message from CSD, the host side marks the status field in the WAL log as csd-committed, indicating that the data to be flushed to disk has been written to the virtual storage area of ​​CSD. Then, it releases the lock and related resources, completing the write transaction process.

[0058] like Figure 4 As shown, this embodiment of the invention provides a specific implementation method for step 204, which specifically includes the following steps: Step 301: Write the disk flushing data into the Cuckoo Filter for recording, and record the storage location of the current disk flushing data in the WAL log.

[0059] The Cuckoo Filter is a space-efficient approximate set retrieval structure that supports dynamic insertion and deletion, exhibits extremely high performance in membership detection, and allows for element deletion.

[0060] This invention maintains a cuckoo filter on the host side. During the process of writing disk flushing data to the virtual storage area of ​​CSD, the corresponding disk flushing data is written to the cuckoo filter for recording. The cuckoo filter can then be used to assist in query path decision-making.

[0061] The location of the data to be flushed to disk (either stored in the virtual storage area or the physical storage area of ​​the CSD) is recorded in the WAL log as a basis for subsequent crash recovery.

[0062] like Figure 5 As shown, the Cuckoo Filter and CSD Index and Storage Framework Diagram includes: the Cuckoo Filter on the host side, the virtual storage area and the physical storage area of ​​the CSD. The CSD contains multi-instance ART indexes, each of which is stored in Dynamic Random Access Memory (DRAM). The physical data is stored in NAND flash memory, and the ART instance index contains references to the physical data.

[0063] During the process of storing data from the data buffer into the virtual storage area, on the one hand, hot data items and frequently updated items in the data buffer are temporarily stored in the data item buffer of the CSD instead of being directly migrated into the actual storage area; on the other hand, the Cuckoo Filter records which data is stored in the data buffer and which data is temporarily stored in the virtual storage area.

[0064] Step 302: Route the write request to the corresponding adaptive radix tree index instance based on the key value range of the disk flushing data.

[0065] Based on the key value range of the disk flush data corresponding to the write request, determine which ART index instance will maintain this update.

[0066] Based on data partitioning and fine-grained locking mechanisms, concurrent writes of different key value ranges are distributed in parallel to independent ART instances to perform memory updates and are simultaneously appended to the CSD internal log. This alleviates global lock contention and utilizes the multi-core parallel computing power of the CSD, reducing the waiting and lock contention overhead of the write path, improving concurrent write throughput and write latency performance.

[0067] Step 303: Perform memory index update in the corresponding adaptive radix tree index instance, simultaneously append the index update record to the index log of the computable storage device, and write the disk flushing data to the virtual storage area corresponding to the adaptive radix tree index instance in a sequential append manner.

[0068] The index log is a log specifically added in this invention to record the changes to the ART index instance each time a modification is made.

[0069] During each update operation, the CSD's logging mechanism appends critical changes (such as the address of the newly inserted key and its corresponding value, updated metadata, etc.) to the device's persistent log, achieving write-before-write log protection close to the data end. This ensures that even if the system crashes unexpectedly, the latest write operation is still recorded and available for recovery. On the other hand, the metadata of the ART index instance is periodically or on-demand persisted to the CSD's non-volatile storage, for example, by flushing incremental changes of ART leaf nodes to a dedicated persistent area.

[0070] Step 304: Load the disk flushing data into the migration buffer of the computeable storage device, and write the disk flushing data from the migration buffer into the actual storage area for permanent storage.

[0071] like Figure 6As shown, when the database is in a write-intensive scenario, the specific process of a write transaction based on a multi-instance ART index is as follows: Lock write transaction A is acquired according to its isolation level; the page node of write transaction A is determined based on the B+ tree path, and the corresponding modification information is recorded in the WAL log; on the host side, the target page corresponding to each page node is modified in the data buffer, and the corresponding modification information is recorded in the Cuckoo filter. The modified data is then flushed from the data buffer to the CSD's virtual storage area. That is, based on the key-value range of the modified data, the ART instance 1 that this modification depends on is determined, the index is updated in ART instance 1, the index log is updated, and the modified data is stored in the virtual storage area corresponding to ART instance 1. Subsequently, the CSD completes the process of flushing the modified data from the virtual storage area to the actual storage area.

[0072] During the migration phase, if the CSD has not yet migrated all the data in the virtual storage area to the physical storage area, and a read transaction occurs, the data pages required by the read transaction may exist in the virtual storage area or in the physical storage area. This invention uses a cuckoo filter to assist in query path decision-making.

[0073] like Figure 7 As shown, this embodiment of the invention provides a reading method based on a cuckoo filter, which specifically includes the following steps: Step 401: Receive a read transaction and lock the read transaction according to its isolation level.

[0074] Based on the isolation level identifier carried by the read transaction, locks are applied to the corresponding read transaction, and data visibility rules are determined based on the isolation level of the read transaction.

[0075] Step 402: Query the Cuckoo Filter to determine whether the target page corresponding to the read transaction is located in the virtual storage area.

[0076] The host side quickly determines whether the target page corresponding to the read transaction still resides in the virtual storage area by querying the cuckoo filter.

[0077] Step 403: If the cuckoo filter indicates that the target page is located in the virtual storage area, a read request is sent to the computable storage device, which then locates and returns the target data page based on the multi-instance adaptive radix tree index structure.

[0078] If the Cuckoo filter indicates that the target page is located in the virtual storage area, the host sends a read request to the compute storage device. Based on the read request, the compute storage device quickly corresponds to the ART instance based on its internal multi-instance ART index, retrieves the target page from the virtual storage area corresponding to the located ART instance, and loads the target page into the host's data buffer pool through the CSD's data buffer. The host then processes the returned target page and outputs the final result.

[0079] Step 404: If the cuckoo filter does not indicate that the target page is located in the virtual storage area, then locate the target data page according to the B+ tree index structure and return it.

[0080] If the Cuckoo filter does not indicate that the target data is located in the virtual storage area, the host locates the target page according to the traditional B+ tree read path, loads the target page from the physical storage area into the host's data buffer pool, and processes the returned target page on the host side before outputting the final result.

[0081] Furthermore, during the reading process, if concurrent modification transactions are involved, the present invention can ensure the isolation of the read operation based on the locking mechanism or version number.

[0082] Step 405: Receive the commit message after the read transaction is completed, and release the lock resources of the read transaction.

[0083] Upon receiving a successful signal that a read transaction has been completed and committed, the resource reclamation mechanism is immediately initiated to automatically release all lock resources occupied by the read transaction during its execution, thereby removing access restrictions on the relevant data objects.

[0084] In this embodiment, when a read transaction is received, the host first checks its own memory to see if there is a target data page corresponding to the read transaction. If not, it uses a cuckoo filter to assist in checking the data item buffer of the CSD to see if there is a target data page corresponding to the read transaction. If not, it further checks the virtual storage area of ​​the CSD to see if there is a target data page corresponding to the read transaction. If it still does not find it, it will then check the actual storage area.

[0085] The above-mentioned mechanism of using the cuckoo filter to assist in querying can avoid blindly searching repeatedly in multiple storage areas. Instead, it can directly determine the location of the key through the cuckoo filter, thereby selecting the most appropriate path to access the data. This mechanism can quickly divert read paths, reduce unnecessary cross-layer lookup overhead (referring to the host side searching in the actual storage area), and ensure that read operations obtain a consistent data view.

[0086] See Figure 8The read transaction process is as follows: Data visibility rules are determined based on the isolation level of read transaction B, and a lock is acquired for read transaction B; the host queries the Cuckoo Filter to determine if the target data corresponding to read transaction B might be located in the virtual storage area; if the Cuckoo Filter indicates that the target data is located in the virtual storage area, the host sends a read request to the compute storage device, which locates the target data page based on its internal multi-instance ART index and loads it from the virtual storage area of ​​the CSD to the buffer pool on the host side via the CSD's data buffer; if the Cuckoo Filter does not indicate that the target data is located in the virtual storage area, the host locates the target data page using the traditional B+ tree read path and loads the target data page from the physical storage area to the buffer pool. After the read transaction is completed, it is committed, the lock resources are released, and the result is returned.

[0087] When the database is under write-intensive load, the host side flushes the data to disk for the write transaction to the virtual storage area of ​​the CSD. Subsequently, the CSD will transfer the flushed data from the virtual storage area to the actual storage area. Considering that some data will be repeatedly updated or read within a period of time, if it is directly migrated to the actual storage area, the erase write overhead caused by frequent migration will be greatly increased.

[0088] This invention sets up a data status monitoring module on the computing storage device side to distinguish between hot data items and frequently updated items, and works in conjunction with the data migration and garbage collection process. That is, the data status monitoring module maintains the number of data accesses and updates. During the data migration stage, it determines whether the data item should be written directly back to the actual storage area or delayed in the virtual storage area based on the number of data accesses and updates. This pre-migration judgment design can reduce the erase write overhead caused by frequent migrations and take into account the access efficiency of hot data.

[0089] Specifically, it is determined whether the flushed data belongs to a hot data item or a frequently updated item. If the flushed data belongs to a hot data item or a frequently updated item, meaning that the flushed data is updated multiple times or accessed at extremely high frequency within a short period of time, the flushed data is copied into the data item buffer and retained in the virtual storage area. When subsequent recycling is triggered, the flushed data is appended to the end of the virtual storage area queue in sequence, the original flushed data in the virtual storage area is deleted, and the flag bit of the flushed data is cleared to zero. This method of temporarily delaying the migration of data from the high-speed virtual storage area to the permanent physical storage area at the back end can avoid the repeated write overhead caused by the back-and-forth migration of hot data (hot data items or frequently updated items), reduce write amplification and device wear, and ensure that hot data always remains in a higher-speed storage area for fast access. If the flushed data does not belong to a hot data item or a frequently updated item, the flushed data is directly migrated into the physical storage area.

[0090] Combination Figure 9When data A is determined to be a hot data item or a frequently updated item, the delayed migration strategy is explained in detail using the migration process of data A as an example. Specifically: Data A is stored in region 1 of the virtual storage area. A copy of data A is made and written to the data item buffer. At this time, region 1 of the virtual storage area still retains data A. When the data item buffer is full or region 1 needs to be reclaimed, data A is written from the data item buffer to the tail of the virtual storage area (i.e., region n of the virtual storage area). At the same time, all data stored in region 1 (including data A) is deleted, and the flag bit of data A is cleared to zero. When data needs to be migrated later, it is re-determined whether data A is hot data, and a new decision is made on whether to migrate data A to the actual storage area or temporarily store it in the data item buffer. Here, reclamation refers to clearing all data originally stored in region B to make room for the storage of other subsequent data.

[0091] In this embodiment, appending data to the virtual storage area instead of the physical storage area has the following main advantages: First, compared to storing data in the physical storage area, storing data in the memory of the CSD results in a faster query rate, which can improve the query rate of hot data items or frequently updated items; Second, it reduces the frequency of hot data items or frequently updated items migrating to the physical storage area, reducing erase and write operations in the physical storage area, thereby extending the lifespan of the solid-state drive. Furthermore, appending data to the virtual storage area can reduce storage fragmentation and write amplification.

[0092] This invention introduces data status and flag fields into the index nodes of multi-instance ART index instances to intelligently identify popular data (including hot data items or frequently updated items) and execute a delayed migration strategy to reduce the cost of repeatedly writing and migrating popular data, extend device life, and ensure fast access to popular data.

[0093] like Figure 10 As shown in the figure, this embodiment of the invention provides a method for determining hotspot data, which specifically includes the following steps: Step 501: If the disk data is valid and the update flag of the disk data is not 0, then the disk data is considered to be a frequently updated item.

[0094] On the CSD side, access counts and update counts are maintained in the leaf nodes of the multi-instance ART index. Specifically, a status identifier and a flag field are added to the leaf node structure of the ART index. The status identifier is used to mark the current storage location of the key value (e.g., still in the virtual storage area or log area of ​​CSD, or has been migrated to the physical storage area). The flag field includes access flags and update flags, which can characterize the popularity of data items and serve as the basis for subsequent migration strategies. The reason for using flags to characterize the popularity of data items is that subsequent verification found that using a flag to represent any data that has been accessed or updated (i.e., storing directly with bits) can save storage overhead. Storing the actual number of accesses and updates would require greater storage overhead.

[0095] A valid state refers to the latest version of a data item. Since data is appended to the virtual storage area, older versions of some data items still exist in the virtual storage area, and these data items are in an invalid state. Only the latest data item is in a valid state.

[0096] During a statistical period, when a data item is updated and written, if the leaf node is in a valid state, the update flag of the data item is set to 1, indicating that the data item was updated during the statistical period or before the migration phase.

[0097] Step 502: If the disk flushing data is in a valid state and the access flag of the disk flushing data is not 0, then the disk flushing data is identified as a hot data item.

[0098] During a statistical period, when a data item is read and accessed, if the leaf node is in a valid state, the access flag of the data item is set to 1, indicating that the data item was accessed during the statistical period or before the migration phase.

[0099] This invention uses the access flag and update flag of the above-mentioned marker to identify frequently updated or accessed hot data, and adopts a delayed migration strategy for such data.

[0100] After the statistical period ends or the migration phase ends, the update flag and access flag corresponding to the data item are both set to 0 to prevent the data access volume and update count from being reused.

[0101] Furthermore, this invention includes a collaborative crash recovery module that performs replay recovery based on WAL logs when the database crashes, and simultaneously recovers the host-side cuckoo filter and the multi-instance ART index on the computeable storage device side, thereby resolving the potential state inconsistency between the host and the computeable storage device.

[0102] Specifically, the collaborative crash recovery module utilizes index logs, migration logs, and checkpoint files on the computable storage device side to achieve rapid reconstruction. In extreme cases, it uses a metadata handshake protocol to verify and coordinate transactions that have been recorded on the host side but may be missing on the computable storage side, in order to ensure the atomicity and consistency of the database after recovery.

[0103] like Figure 11 As shown, this embodiment of the invention provides a method for recovery after a crash, which specifically includes the following steps: Step 601: Recover committed but incomplete transactions in the B+ tree structure based on the WAL logs.

[0104] During the recovery process after a crash, after the database restarts, the host side reads the snapshot of the most recent checkpoint, replays the WAL logs after the most recent checkpoint, and redoes all committed transactions, thereby ensuring the data integrity and consistency in the traditional B+ tree structure. The most recent checkpoint refers to the last point in time before the failure occurred when the database successfully and consistently flushed the data from the memory side to the actual storage area.

[0105] Step 602: Rebuild the multi-instance ART index of the computeable storage device using the index log and checkpoint file.

[0106] On the CSD side, instead of scanning large amounts of data files or replaying all logs one by one to rebuild the multi-instance ART index, it quickly rebuilds the index state using previously saved index logs and checkpoint files, thus greatly shortening the rebuild time of the multi-instance ART index. Furthermore, through hardware acceleration on the computeable storage device side, this persistent indexing mechanism has minimal impact on runtime performance, but provides guarantees for data consistency and reliability, enabling the system to quickly recover to the latest consistent state after a crash.

[0107] Step 603: Restore the Cuckoo Filter based on the log records in the WAL log that contain the field for writing to a computable storage device.

[0108] The host side identifies log records in the WAL log that contain a field for writing to the computable storage device layer, which serves as the basis for subsequent recovery of the host-side cuckoo filter.

[0109] During the WAL log replay process on the host side, if data modifications involving computable storage paths are encountered, the Cuckoo Filter is gradually restored according to the modification status recorded in the WAL log, so that the data recording status of the Cuckoo Filter is consistent with the data status confirmed by the CSD side.

[0110] Furthermore, for extreme consistency risk scenarios, namely, when the host-side WAL log is successfully written and the data is marked on the CSD side, but the data is lost during transmission or the CSD fails to complete persistence due to power failure, the host side performs a metadata handshake verification with the CSD side for transactions or data items in a questionable state.

[0111] Specifically: if the computable storage device confirms that the data corresponding to the transaction in question has been stored and persisted in the internal log of the computable storage device, then the redo logic is executed and the memory mapping and filter state are restored; if the computable storage device confirms that the data corresponding to the transaction in question is missing, then the data corresponding to the transaction in question is rolled back to ensure system atomicity and consistency.

[0112] Step 604: The data corresponding to the transactions that were committed but not yet stored before the crash are sequentially flushed to the virtual storage area, and the data corresponding to the transactions that were committed but not yet stored are migrated to the physical storage area by the computable storage device.

[0113] After the host side confirms that the memory index (i.e., the multi-instance ART index) on the CSD side has been restored, the host side can sequentially flush the dirty pages that were not stored before the crash to the virtual storage area of ​​the CSD, and the CSD will then complete the process of migrating the dirty pages that were not stored before the crash to the actual storage area.

[0114] This invention combines CSD index logs and index persistence mechanisms, making full use of the log recording and hardware-accelerated persistence capabilities provided by computeable storage devices, providing reliable and fast recovery capabilities, significantly shortening fault recovery time, and ensuring data consistency is not affected by high-concurrency writes while improving system availability.

[0115] Example 2: Based on the write method based on a computationally readable storage device provided in the foregoing embodiments, the present invention also provides an apparatus for implementing the above method based on a computationally readable storage device, such as... Figure 12 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. The write-based device for a computationally readable storage device in this embodiment includes one or more processors 21 and a memory 22. Figure 12 Take a processor 21 as an example.

[0116] Processor 21 and memory 22 can be connected via a bus or other means. Figure 12 Taking the example of a connection between China and Israel via a bus.

[0117] The memory 22, as a non-volatile computer-readable storage medium for writing based on a computationally readable storage device, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the writing method based on a computationally readable storage device in the foregoing embodiments. The processor 21 executes various functional applications and data processing of the writing device based on the computationally readable storage device by running the non-volatile software programs, instructions, and modules stored in the memory 22, thereby implementing the writing method based on the computationally readable storage device in the foregoing embodiments.

[0118] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0119] The program instructions / modules are stored in memory 22 and, when executed by one or more processors 21, perform the write method based on the computable storage device described in the foregoing embodiments.

[0120] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 12 One of the processors 21 can enable one or more of the processors to execute the write method based on the computable storage device in the foregoing embodiments.

[0121] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0122] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0123] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A write method based on a computable storage device, characterized in that, include: Receive write transactions for the database; When the database is under non-write-intensive load, the disk flushing data corresponding to the write transaction is written to the actual storage area based on the B+ tree index structure. The entire key space of the database is divided into multiple partitions according to the range of keys. For each partition, the key value falling into the partition and the pointer to the actual data are stored in the leaf node of the adaptive radix tree, resulting in a multi-instance adaptive radix tree index structure. When the database is under write-intensive load, based on the multi-instance adaptive radix tree index structure, the disk flushing data corresponding to the write transaction is written to the virtual storage area of ​​the computable storage device, and the computable storage device completes the process of migrating the disk flushing data from the virtual storage area to the actual storage area.

2. The write method based on a computable storage device according to claim 1, characterized in that, When the database is under write-intensive load, based on a multi-instance adaptive radix tree index structure, the data flushed to disk corresponding to the write transaction is written to the virtual storage area of ​​the computeable storage device. The computeable storage device then completes the process of migrating the flushed data from the virtual storage area to the actual storage area, including: Receive the write transaction and lock it according to its isolation level; The original data is modified based on the information of the write transaction to obtain the flush data, and the B+ tree index structure is updated. Write the modification record corresponding to the write transaction to the WAL log; Submit the write transaction to write the flush data to the virtual storage area, and then migrate the flush data from the virtual storage area to the actual storage area via the migration buffer. Upon receiving the information that the disk data has been stored, update the storage status of the disk data in the WAL log and release the lock resources of the write transaction.

3. The write method based on a computable storage device according to claim 2, characterized in that, The step of submitting the write transaction, writing the flushed data to the virtual storage area, and migrating the flushed data from the virtual storage area to the physical storage area via a migration buffer includes: The disk flushing data is written into the Cuckoo Filter for recording, and the storage location of the current disk flushing data is recorded in the WAL log. The write request is routed to the corresponding adaptive radix tree index instance based on the key value range of the disk flushing data. In the corresponding adaptive radix tree index instance, an index update is performed, and the index update record is simultaneously appended to the index log of the computable storage device. The disk flushing data is written to the virtual storage area corresponding to the adaptive radix tree index instance in a sequential append manner. The disk flushing data is loaded into the migration buffer of the computeable storage device, and then written from the migration buffer into the actual storage area for permanent storage.

4. The write method based on a computable storage device according to claim 1, characterized in that, The method further includes: Receive a read transaction and lock it according to its isolation level; Query the cuckoo filter to determine whether the target page corresponding to the read transaction is located in the virtual storage area; If the cuckoo filter indicates that the target page is located in the virtual storage area, a read request is sent to the computable storage device, which then locates and returns the target data page based on the multi-instance adaptive radix tree index structure. If the cuckoo filter does not indicate that the target page is located in the virtual storage area, the target data page is located and returned according to the B+ tree index structure; Receive the commit message after the read transaction is completed, and release the lock resources of the read transaction.

5. The write method based on a computable storage device according to claim 1, characterized in that, The method further includes: Determine whether the data being flushed to disk belongs to a hot data item or a frequently updated item; If the disk flushing data belongs to a hot data item or a frequently updated item, the disk flushing data is copied into the data item buffer and the disk flushing data in the virtual storage area is retained; when subsequent recycling is triggered, the disk flushing data is appended to the end of the virtual storage area queue in sequence, the original disk flushing data in the virtual storage area is deleted, and the flag bit of the disk flushing data is cleared to zero. If the data being flushed to disk is not a hot data item or a frequently updated item, then the data being flushed to disk will be directly migrated to the actual storage area.

6. The write method based on a computable storage device according to claim 5, characterized in that, The determination of whether the disk refresh data belongs to hot data items or frequently updated items includes: If the disk data is valid and the update flag of the disk data is not 0, then the disk data is considered to be a frequently updated item. If the disk data is valid and the access flag of the disk data is not 0, then the disk data is considered a hot data item.

7. The write method based on a computable storage device according to claim 1, characterized in that, The method further includes: Recover committed but incomplete transactions in the B+ tree structure based on WAL logs; Reconstruct multi-instance ART indexes for computeable storage devices using index logs and checkpoint files; Recover the Cuckoo Filter based on log records in the WAL log that contain a field for writing to a computable storage device. Data corresponding to transactions that were committed but not yet stored before the crash are sequentially flushed to the virtual storage area, and the data corresponding to these transactions are then migrated to the physical storage area by the compute-enabled storage device.

8. The write method based on a computable storage device according to claim 7, characterized in that, For transactions in a questionable state, the method further includes: If the computable storage device confirms that the data corresponding to the transaction in question has been persisted in the internal log of the computable storage device, then the redo logic is executed and the memory mapping and filter state are restored. If the storage device can be calculated to confirm the missing data corresponding to a transaction in a questionable state, then a rollback process is performed on the data corresponding to the transaction in the questionable state.

9. A write device based on a computable storage device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing a write method based on a computable storage device as described in any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the write method based on a computable storage device as described in any one of claims 1-8.

Images