Data storage method, data storage device, electronic device and computer readable storage medium

Abstract

The application relates to a data storage method, a data storage device, an electronic device and a computer readable storage medium. Embodiments of the application retain the advantages of a second storage component with a sequential file storage structure, such as good write performance, high compression rate and low storage cost, and solve the problem of poor read performance of a traditional sequential file storage structure by introducing a first storage component with a page index structure. The first storage component in the application can be constructed based on a common solid state disk, without introducing special hardware (such as a non-volatile memory), thereby effectively reducing the hardware cost; and the number of layers of the upper layer of the first storage component replacing the second storage component can be flexibly customized according to actual requirements, thereby improving the universality.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically to a data storage method, a data storage device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the continuous expansion of data scale and the increasing demand for concurrent business access, database systems have become a supporting component in information processing architecture. As the foundation for data persistence and efficient access, database storage architecture typically adopts a layered and decoupled design, divided from top to bottom into a query processing layer, a transaction management layer, and a physical storage layer. Each layer works together to complete data parsing, scheduling, reading, writing, and persistence.

[0003] The physical storage layer uses fixed-size data blocks as the basic read / write unit and achieves data scheduling between memory and disk through cache management to balance access performance and storage cost. To support fast retrieval and ordered traversal of large-scale data, the storage layer generally adopts a hierarchical indexing organization method, organizing discrete physical storage units into a storage architecture that can be efficiently traversed and located.

[0004] In real-world applications where databases need to support frequent reads and writes and have large storage capacity, existing storage architectures still have room for further optimization in terms of space utilization, range scan efficiency, and disk I / O convergence. Summary of the Invention

[0005] This application provides a data storage method, a data storage device, an electronic device, and a computer-readable storage medium.

[0006] This application provides a data storage method, comprising: receiving data to be written and storing the data to be written into a memory buffer; converting the memory buffer into a read-only buffer in response to the data volume in the memory buffer reaching a preset threshold; persistently writing the data in the read-only buffer into a first storage component, the first storage component being a paged index structure, the paged index structure including a root node, at least one layer of intermediate index nodes, and leaf nodes; and extracting a target data block from the leaf nodes in response to the first storage component meeting migration conditions, merging and sorting the target data block with data in the starting data layer of a second storage component, and writing the sorted data into the starting data layer; wherein the second storage component is a sequential file storage structure, the sequential file storage structure including multiple persistent data layers arranged in hierarchical order, the multiple persistent data layers including the starting data layer, and the number of levels of the multiple persistent data layers being associated with configuration parameters.

[0007] This application provides a data storage device, including: a memory buffer configured to receive data to be written; a read-only buffer conversion unit configured to convert the memory buffer into a read-only buffer in response to the amount of data in the memory buffer reaching a preset threshold; a first storage component, the first storage component being a paged index structure, the paged index structure including a root node, at least one layer of intermediate index nodes, and leaf nodes; a second storage component, the second storage component being a sequential file storage structure, the sequential file storage structure including multiple persistent data layers arranged in hierarchical order, the multiple persistent data layers including a start data layer, and the number of levels of the multiple persistent data layers being associated with configuration parameters; a persistent writing unit configured to persistently write data from the read-only buffer to the first storage component; and a merging unit configured to merge and sort a target data block extracted from the leaf node with data in the start data layer of the second storage component in response to the first storage component meeting migration conditions, and write the merged and sorted data to the start data layer.

[0008] This application discloses an electronic device, including: one or more processors; and one or more memories, wherein the memories store computer-executable instructions, and when the processor executes the computer-executable instructions, the above-described method is performed.

[0009] This application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the above-described method.

[0010] According to another aspect of this application, a computer program product or computer program is provided, comprising computer-executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer-executable instructions from the computer-readable medium and executes the computer-executable instructions, causing the computer device to perform the methods provided in the foregoing aspects or various alternative implementations of the foregoing aspects.

[0011] This application retains the advantages of the second storage component with a sequential file storage structure, such as good write performance, high compression ratio, and low storage cost. Simultaneously, by introducing a first storage component with a page-based index structure, it solves the problem of poor read performance in traditional sequential file storage structures. The first storage component in this application can be built based on a regular solid-state drive (SSD), eliminating the need for special hardware (such as non-volatile memory), effectively reducing hardware costs. Furthermore, the number of layers replacing the second storage component can be flexibly customized according to actual needs, improving versatility. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The accompanying drawings in the following description are merely exemplary embodiments of this application.

[0013] Figure 1 This is an example schematic diagram illustrating a scenario according to an embodiment of this application.

[0014] Figure 2 A flowchart of a data storage method according to an embodiment of this application is shown.

[0015] Figure 3 A schematic diagram of a data storage device according to an embodiment of this application is shown.

[0016] Figure 4 A flowchart illustrating the persistent writing of data in a read-only buffer to a first storage component according to an embodiment of this application is shown.

[0017] Figure 5 A schematic diagram illustrating the organization of data in a read-only buffer according to an embodiment of this application is shown.

[0018] Figure 6 A schematic diagram of multi-version data storage under a hybrid architecture according to an embodiment of this application is shown.

[0019] Figure 7 A schematic diagram illustrating the updating of a first storage component and a memory mapping table according to an embodiment of this application is shown.

[0020] Figure 8 A flowchart illustrating the migration condition determination based on invalid space according to an embodiment of this application is shown.

[0021] Figure 9 A schematic diagram is shown illustrating the writing of sorted data into the starting data layer according to an embodiment of this application.

[0022] Figure 10 A schematic diagram illustrating fault recovery using hybrid logs according to an embodiment of this application is shown.

[0023] Figure 11 A multi-level query data flow diagram according to an embodiment of this application is shown.

[0024] Figure 12 A schematic diagram of an electronic device according to an embodiment of this application is shown.

[0025] Figure 13 A schematic diagram of the architecture of an exemplary computing device according to an embodiment of this application is shown.

[0026] Figure 14A schematic diagram of a storage medium according to an embodiment of this application is shown. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0028] In this specification and accompanying drawings, operations and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these operations and elements will be omitted. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0029] To facilitate the description of this application, the following concepts related to this application are introduced.

[0030] When a database system faces the business requirements of massive data storage and efficient data access, its index structure and underlying storage architecture directly determine the overall read / write performance, resource consumption, and operational stability of the system. To address the differentiated read / write load characteristics under various business scenarios, most mainstream database storage engines are designed and implemented based on two typical data structures. One is the disk-optimized index structure, represented by B+ trees, which is more suitable for business scenarios with high requirements for query efficiency, transaction consistency, and latency stability. The other is the log-structured merged storage architecture, represented by LSM trees, which focuses more on write performance optimization in high-concurrency, high-throughput scenarios. Both improve storage access efficiency through different technical paths, such as ordered index organization and batch sequential writing, and together constitute the mainstream technical solutions in the current database storage field.

[0031] Specifically, the B+ tree is a typical self-balancing multi-way search tree, widely used in disk index design for various database systems and file systems. B+ tree nodes can be divided into three categories based on level and function: root nodes, internal nodes, and leaf nodes. The root node serves as the top-level entry point to the entire index tree, responsible for the initial route of index lookups; internal nodes only store index key values ​​and pointers to child nodes, without storing actual business record data; leaf nodes store complete key values ​​and corresponding data records, and the leaf nodes are interconnected via pointers to form an ordered linked list, thus supporting efficient range queries.

[0032] The search process in a B+ tree starts from the root node. It compares the key to be searched with the keys stored in internal nodes level by level to determine the path to the next child node, and then traverses along that path until it reaches a leaf node to locate the target key or corresponding data. The time complexity of a B+ tree search can be expressed as O(h), where h is the height of the tree. Thanks to the high fan-out characteristic of B+ trees, their height is typically kept low, usually only 3 to 5 levels, thus effectively reducing disk I / O operations and resulting in high search efficiency.

[0033] The insertion operation in a B+ tree first locates the target leaf node and inserts the data within that node. If the amount of data in the leaf node exceeds the preset capacity limit after insertion, a node split operation is triggered, migrating some data to the newly created leaf node and simultaneously updating the index information and pointers of the parent node. If the parent node also experiences capacity overflow due to an increase in index entries, the split process will recursively execute upwards until the root node. When the root node splits, the newly created root node becomes the new top-level entry point, and the overall height of the B+ tree increases by one level accordingly.

[0034] In a B+ tree, deletion first locates the target leaf node and removes the corresponding data. If the amount of data in the node after deletion is lower than the set minimum capacity threshold, it first attempts to migrate data from adjacent sibling nodes to make up the capacity. If the sibling nodes have no extra data to borrow, a node merging operation is performed, merging adjacent nodes into a single node and updating the parent node's index and pointer information. This adjustment process may also recursively propagate upwards, which in extreme cases can lead to a decrease in the height of the entire B+ tree.

[0035] B+ trees, with their high fan-out and low height, significantly reduce random disk I / O operations. They exhibit stable performance and predictable access latency in both single-point and range query scenarios, and the ordered linked list structure of the leaf nodes further enhances their range query efficiency. However, B+ trees still have significant drawbacks in write scenarios: insertion, deletion, and update operations typically require locating a specific data page and modifying it in-place, often involving reading and writing multiple data pages, which can generate a large amount of random disk I / O and affect write throughput. Furthermore, to reserve space for subsequent insertion and update operations, each data page needs to retain a certain amount of redundant space, resulting in a lower actual data fill rate and poor compression efficiency, thus increasing overall storage costs.

[0036] Log-Structured Merge-tree (LSM tree) is a hierarchical, ordered storage structure designed to address random I / O bottlenecks in high-frequency write scenarios. The core idea of ​​LSM trees is to transform numerous discrete random write operations into batch sequential writes, thereby significantly improving system write throughput. In terms of overall architecture, LSM trees are primarily divided into two parts: memory components and disk components.

[0037] The memory component comprises two types of structures: memory tables and immutable memory tables. The memory table is the active memory data structure, used to directly receive foreground write requests from the user side; all new data is preferentially written to the memory table. When the data volume in the memory table reaches a preset threshold, the storage device freezes it and marks it as an immutable memory table, ceasing to receive new write operations, and simultaneously creating a new memory table to accommodate subsequent data. In terms of implementation, memory tables typically employ a skip list structure because it supports efficient lookup and insertion operations, and its implementation logic is simple and easy to maintain.

[0038] The disk component, serving as the persistent data layer of the LSM tree, employs a multi-level hierarchical design, consisting of levels 0 to n, each containing several ordered string table files. After data is persisted to disk via an immutable in-memory table, it is first written to the ordered string table at level 0. When the number of files or the total data size at a certain level reaches a trigger threshold, the storage device initiates a compression and merging process, merging and sorting the files at that level with those at the next lower level to generate new ordered files, which are then written to the next lower level. Simultaneously, the merged old files are deleted. As the number of levels increases, the amount of data each level can hold expands exponentially, ensuring that the vast majority of business data ultimately resides at the lowest storage level.

[0039] Using a log-structured merge tree as the storage structure has the following technical drawbacks: poor and unstable read performance. Because the data is distributed across multiple ordered string table files at multiple levels, queries need to be performed sequentially in the memory table, the immutable memory table, and the ordered string table files at each level. In the worst case, all levels need to be traversed, making read latency unpredictable. Compression operations consume a large amount of input / output resources. During large-scale data merging, the system's input / output bandwidth is heavily occupied, leading to a significant increase in the response time of front-end read / write requests. At the same time, the compression process requires repeated reading and writing of a large amount of data, causing write amplification problems.

[0040] First, refer to Figure 1 This application describes the application scenarios corresponding to the data storage method and data storage device according to the embodiments of this application. Figure 1 A schematic diagram of an application scenario 100 according to an embodiment of this application is shown, wherein a server 110 and multiple user terminals 120 are schematically illustrated.

[0041] User terminal 120 can be an electronic device with data interaction capabilities, such as a mobile phone, tablet computer, laptop computer, desktop computer, smart TV, smart glasses, smart speaker, or smartwatch, but is not limited to these.

[0042] Server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. User terminals and servers can be directly or indirectly connected via wired or wireless communication; this application does not limit this.

[0043] In actual business operations, multiple user terminals 120 can simultaneously send various data read and write requests to server 110, including data query, insertion, content update, and record deletion operations. Upon receiving these concurrent read and write requests, server 110 uses its built-in database storage engine to complete request parsing, permission verification, and task scheduling, and then performs data read and write, index updates, and persistent storage on the physical storage medium. Especially in high-concurrency business scenarios serving a massive number of online users, the concentrated delivery of a large number of intensive read and write requests places extremely high demands on the database storage engine's read and write efficiency, I / O resource consumption, storage space utilization, and service stability. The quality of the underlying data organization and index structure directly affects the overall business response latency and operational reliability.

[0044] Specifically, using B+ trees as the underlying storage structure also has significant drawbacks: write performance is a bottleneck, as insertion, deletion, and update operations all require locating a specific data page for in-place modification, generating a large amount of random disk I / O, and frequent disk head seeks severely limit write throughput; storage space utilization is low, as each data page of the B+ tree needs to reserve redundant capacity to reserve space for subsequent data operations, resulting in an actual effective data volume far lower than the physical storage capacity, and B+ trees do not natively support efficient embedded compression, with a data compression rate far lower than that of log structure merge trees, further increasing storage deployment costs.

[0045] Using a log-structured merge tree as the underlying storage structure may present the following problems: 1) Poor read performance and insufficient stability. Since the data is scattered across multiple ordered string table files at multiple levels, a single query requires sequentially searching for matches in the memory table, the immutable memory table, and the ordered string table files at each level. In the worst case, it is necessary to traverse all storage levels, resulting in uncontrollable read latency and significant query jitter; 2) Compression and merging operations consume a large amount of I / O resources. During large-scale data merging, the system I / O bandwidth is heavily occupied by background tasks, directly leading to a significant increase in the response latency of front-end business read and write requests. At the same time, repeated read and write operations on data during the merging process will further exacerbate the write amplification problem.

[0046] Currently, some hybrid storage solutions combining B+ trees and log structure merged trees have emerged in the industry, but they still have many limitations: most solutions only use B+ trees to replace the level 0 storage structure of LSM trees, which has limited effect on optimizing overall read and write performance; some optimization solutions, in order to ensure performance, need to rely on dedicated hardware such as non-volatile memory to carry B+ tree leaf nodes, which not only significantly increases hardware deployment costs, but also results in poor versatility and portability of the solutions, making it difficult to widely deploy and implement them in general-purpose servers and conventional hardware environments.

[0047] To this end, this application provides a data storage method, comprising: receiving data to be written and storing the data to be written into a memory buffer; in response to the data volume in the memory buffer reaching a preset threshold, converting the memory buffer into a read-only buffer; persistently writing the data in the read-only buffer into a first storage component, the first storage component being a paged index structure, the paged index structure including a root node, at least one layer of intermediate index nodes, and leaf nodes; and in response to the first storage component meeting migration conditions, extracting a target data block from the leaf nodes, merging and sorting the target data block with data in the starting data layer of a second storage component, and writing the sorted data into the starting data layer; wherein the second storage component is a sequential file storage structure, the sequential file storage structure including multiple persistent data layers arranged in hierarchical order, the multiple persistent data layers including the starting data layer, and the number of levels of the multiple persistent data layers being associated with configuration parameters.

[0048] This application also provides a data storage device, comprising: a memory buffer configured to receive data to be written; a read-only buffer conversion unit configured to convert the memory buffer into a read-only buffer in response to the amount of data in the memory buffer reaching a first preset threshold; a first storage component, the first storage component being a paged index structure, the paged index structure including a root node, at least one layer of intermediate index nodes, and leaf nodes; a second storage component, the second storage component being a sequential file storage structure, the sequential file storage structure including multiple persistent data layers arranged in hierarchical order, the multiple persistent data layers including a start data layer, and the number of levels of the multiple persistent data layers being associated with configuration parameters; a persistent writing unit configured to persistently write the data in the read-only buffer to the first storage component; and a merging unit configured to merge and sort the target data block extracted from the leaf node with the data in the start data layer of the second storage component in response to the first storage component meeting migration conditions, and write the merged and sorted data to the start data layer.

[0049] This application retains the advantages of the second storage component with a sequential file storage structure, such as good write performance, high compression ratio, and low storage cost. Simultaneously, by introducing a first storage component with a page-based index structure, it solves the problem of poor read performance in traditional sequential file storage structures. The first storage component in this application can be built based on a regular solid-state drive (SSD), eliminating the need for special hardware (such as non-volatile memory), effectively reducing hardware costs. Furthermore, the number of layers replacing the second storage component can be flexibly customized according to actual needs, improving versatility.

[0050] This application retains the memory buffer and several underlying layers of the sequential file storage structure, replacing the upper layers of the disk structure of the second storage component with a first storage component (page index structure). This integrated design allows this application to simultaneously possess the write performance of a sequential file storage structure and the query performance of a page index structure.

[0051] In one optional aspect, embodiments of this application can be widely applied to key-value databases or relational databases based on solid-state drives. Embodiments of this application are particularly suitable for business systems with high read loads, scenarios with high requirements for query performance and concerns about storage costs, and enterprise-level applications that need to achieve efficient data storage in ordinary hardware environments.

[0052] Compared to existing sequential file storage structures, where all data above the initial data layer is an ordered string table, the lack of global ordering necessitates searching through multiple files, resulting in poor and unstable read performance. This application replaces the initial data layer and the first layer with a first storage component (page-based index structure). Since the page-based index structure inherently possesses global ordering, queries only require searching a few files, significantly improving read performance. Secondly, existing page-based index structures typically require specialized hardware such as non-volatile memory to achieve optimal performance, leading to high hardware costs and poor versatility. This application, however, can be implemented using ordinary solid-state drives (SSDs), and the number of replaced layers can be flexibly configured according to actual needs, demonstrating strong versatility and adaptability. Therefore, this application achieves a good balance in terms of write performance, query performance, and storage cost.

[0053] The converged storage architecture of this application is divided into the following layers from top to bottom: the top layer is a memory buffer for receiving data written from the foreground; the middle layer is the first storage component for storing recently active data, providing efficient point and range query performance; the bottom layer is the second storage component for storing historical and cold data, providing high compression ratio and low-cost storage. This layered architecture allows data with different frequencies of activity (i.e., data with different read / write frequencies) to be stored in the most suitable layer, thereby achieving optimal overall performance.

[0054] The following reference Figures 2 to 14 This application describes data storage methods, electronic devices, and computer-readable storage media according to embodiments of the present application.

[0055] Figure 2 A flowchart of a data storage method 20 according to an embodiment of this application is shown, wherein method 20 includes the following operations S201 to S204. Of course, method 20 may also include more or fewer operations, and this application is not limited thereto.

[0056] In operation S201, the data to be written is received and stored in the memory buffer.

[0057] Optionally, the data to be written refers to various data records submitted by the user to the server via the terminal that need to be persistently stored. Examples include user information, business transaction records, and system logs. Optionally, the data to be written is organized and stored in key-value pairs, where the key is an identifier used to uniquely identify each record (such as user ID or transaction number), and the value is the detailed data content associated with that key (such as user name, transaction amount, or transaction time). The format of the data to be written can be a simple key-value pair or more complex structured data (such as business form data containing multiple fields), depending on the requirements of the application scenario. This application does not impose any special restrictions on the specific format of the data to be written, as long as the data can be serialized and stored, ensuring stable persistence and subsequent normal reading and parsing.

[0058] Optionally, a memory buffer (also known as a memory table) is the first-level storage structure located in random access memory (RAM) and is used to temporarily store recently received data to be written. As an active memory data structure receiving write requests, the memory buffer's role is to quickly handle foreground write requests, avoiding the latency caused by direct disk writes and improving write response speed. Due to the limited storage capacity of memory, the capacity of a memory buffer is usually much smaller than that of disk storage, but its read and write speed is much faster than that of disk, enabling efficient handling of batch write requests in high-concurrency scenarios and ensuring the continuity of business writes.

[0059] In one alternative embodiment, a skip list data structure can be used to implement the in-memory table. A skip list is a multi-level linked list structure that maintains multiple levels of indexes through randomization, with each level being a subset of the indexes of the next level. Skip lists achieve near logarithmic time for lookup and insertion, and offer the following advantages for adapting to in-memory tables: Compared to arrays, skip lists do not require pre-allocating a fixed amount of space and can dynamically grow based on the amount of data written, avoiding space waste. Compared to ordinary linked lists, skip lists skip some nodes through multi-level indexes, resulting in higher lookup efficiency and faster data location. Compared to paged index structures, skip lists offer lighter-weight in-memory operations, eliminating the need for disk I / O, making them more suitable for use as in-memory tables for temporary storage of data to be written.

[0060] In operation S202, in response to the amount of data in the memory buffer reaching a preset threshold, the memory buffer is converted into a read-only buffer.

[0061] Optionally, a freeze operation can be automatically triggered when the amount of data in the memory buffer (memory table) reaches a preset first threshold. This preset threshold can be flexibly configured according to the hardware configuration and business write volume of the actual application scenario. A typical value is 4 megabytes (MB) or 1 million records. The purpose is to avoid memory overflow caused by excessive memory buffer capacity, while ensuring the efficiency of subsequent data persistence.

[0062] Optionally, the process of converting the memory buffer into a read-only buffer is as follows: First, stop writing new data to the current memory buffer (memory table) and mark the memory buffer as frozen to ensure that the data in the buffer is no longer modified; then, convert the frozen memory buffer into a read-only buffer (i.e., an immutable memtable). After the conversion, the read-only buffer no longer accepts any write requests and is only used for subsequent data reading and persistence operations; finally, immediately create a new memory buffer (memory table) to continue receiving subsequent write requests from users. This mechanism ensures the continuity of write operations. When the old memory buffer is frozen and converted, the new memory buffer immediately takes over the write service, preventing write interruptions and improving the stability of business writes.

[0063] Optionally, during the conversion of the memory buffer to a read-only buffer, the data storage device executing data storage method 20 automatically performs key-value sorting on the data in the buffer. Specifically, the data storage device sorts all data to be written according to ascending key-value order (or a preset sorting rule), and then stores it in the read-only buffer after sorting. This ordered characteristic has significant advantages: on the one hand, it avoids additional sorting when writing data to the first storage component, improving persistent write efficiency; on the other hand, ordered data can better adapt to the paged index structure of the first storage component, reducing the index adjustment overhead during data insertion. Once the data in the read-only buffer is sorted, it will maintain a fixed order until all data is persistently written to the first storage component, during which time no data corruption or order change will occur.

[0064] In operation S203, the data in the read-only buffer is persistently written to the first storage component, which is a paged index structure, including a root node, at least one layer of intermediate index nodes, and leaf nodes.

[0065] For example, since the data in the read-only buffer (immutable memory table) is already sorted by key value, no additional sorting processing is required, and it can be directly and efficiently written to the first storage component. The first storage component uses a page-based index structure, which is a disk-optimized ordered index structure. In practical applications, it can be implemented using a B+ tree, whose structure includes a root node, at least one level of intermediate index nodes, and leaf nodes. That is, the first storage component may contain several levels of intermediate index nodes and one level of leaf nodes. Of course, this application is not limited to this.

[0066] Furthermore, the paged index structure has the following characteristics: all actual business data is stored in the leaf nodes, while intermediate index nodes and the root node only store key values ​​and pointers to child nodes for index location; leaf nodes are interconnected by pointers to form an ordered linked list, enabling fast support for range query operations; each node can store multiple key values, resulting in a larger fan-out ratio and a lower overall index tree height, thereby reducing disk I / O operations and improving query efficiency. Write operations in a paged index structure typically involve page allocation, data insertion, and, if necessary, page splitting. However, since the data in the read-only buffer is already ordered, it can be directly inserted in batches according to the node rules of the paged index structure without additional sorting and adjustment, thus significantly improving write performance and reducing resource consumption during the write process.

[0067] In operation S204, in response to the first storage component meeting the migration conditions, a target data block is extracted from the leaf node, the target data block is merged and sorted with the data in the starting data layer of the second storage component, and the sorted data is written into the starting data layer.

[0068] Optionally, when the amount of data in the first storage component reaches a certain value, the data storage device will trigger migration conditions and initiate a data migration operation to merge the data in the first storage component into the second storage component. Migration conditions can be flexibly configured according to actual business needs, including but not limited to: the number of records stored in the first storage component reaching a preset second threshold, the disk capacity occupied by the data reaching a specified upper limit, or reaching a preset time period (such as triggering migration once per hour or per day). Details of the data migration process will be described in detail later, and will not be repeated here.

[0069] Optionally, the second storage component is a sequential file storage structure, which includes multiple persistent data layers arranged in hierarchical order. The multiple persistent data layers include the initial data layer, and the number of levels in the multiple persistent data layers is associated with configuration parameters. Of course, this application is not limited to this.

[0070] Optionally, the first storage component corresponds to a plurality of consecutive high-level persistent data layers in the sequential file storage structure, the hierarchy of the plurality of consecutive high-level persistent data layers being determined by the configuration parameters, and the starting data layer being the first persistent data layer after the plurality of consecutive high-level data layers. Of course, this application is not limited thereto.

[0071] Specifically, in the converged storage architecture of this application, the first storage component assumes the functions of multiple consecutive high-level layers in a traditional sequential file storage structure, replacing the storage role of these high-level layers. A traditional sequential file storage structure starts from the initial data layer (e.g., layer 0) and ascends layer by layer, with data gradually merging from upper layers to lower layers. In contrast, this application uses the first storage component to replace the initial data layer, the first layer, and other upper-level structures of the sequential file storage structure (the number of layers replaced is determined by configuration parameters). Because the first storage component itself possesses globally ordered characteristics, its function is completely equivalent to multiple consecutive high-level layers in a traditional sequential file storage structure, providing both high-efficiency query performance and connecting to the lower-level second storage component.

[0072] The initial data layer of the second storage component is the first persistent data layer after the first storage component replaces multiple consecutive high-level layers, and it is also the direct target layer for data migration from the first storage component. During data merging, the ordered data in the first storage component is merged and sorted with the existing ordered data in the initial data layer to ensure that the data written to the initial data layer remains globally ordered. This guarantees data consistency and fully leverages the high compression ratio and low cost advantages of the second storage component. This hierarchical correspondence allows the converged architecture of this application to retain the efficient query performance of the first storage component while enjoying the high compression ratio and low storage cost advantages of the second storage component, achieving a balance between performance and cost.

[0073] Optionally, the configuration parameters are configurable parameters used to adapt to the needs of different business scenarios. They can be flexibly set according to the actual read / write load, storage capacity, and performance requirements of the business. Specifically, they include three aspects: First, the number of layers above the second storage component replaced by the first storage component, for example, it can be configured to replace 1 layer (only the initial data layer) or 2 layers (initial data layer + first layer) to adapt to different query performance requirements; Second, migration condition-related parameters, including the record count threshold, capacity threshold, and timing period of the first storage component, can be adjusted according to the write frequency and data growth rate to avoid migration operations being too frequent or too delayed; Third, the total number of layers of the second storage component can be configured according to the overall data storage volume to ensure that the underlying layer can accommodate sufficient historical data and cold data, further optimizing storage costs. By flexibly configuring these parameters, the storage method of this application can adapt to business scenarios of different scales and different read / write characteristics, greatly improving the versatility and adaptability of the solution.

[0074] Therefore, this application retains the advantages of the second storage component with a sequential file storage structure, such as good write performance, high compression ratio, and low storage cost. Simultaneously, by introducing a first storage component with a page-based index structure, the problems of poor read performance and unstable queries in traditional sequential file storage structures are effectively solved. The first storage component in this application can be built on a regular solid-state drive, eliminating the need for special hardware such as non-volatile memory, effectively reducing hardware deployment costs. Furthermore, by flexibly configuring relevant parameters, the number of layers above the second storage component replaced by the first storage component can be customized according to actual needs, further improving the versatility and practicality of the solution.

[0075] This application embodiment retains the memory buffer (memory table) and several underlying layers of the sequential file storage structure, and replaces several upper layers of the disk structure of the second storage component with the first storage component (page index structure). This integrated design enables the data storage method 20 to simultaneously possess the excellent write performance of the sequential file storage structure and the efficient query performance of the page index structure, achieving a good balance between write throughput, query efficiency and storage cost.

[0076] Next, refer to Figure 3 The data storage device 30 according to an embodiment of this application is described below. Figure 3 A schematic diagram of a data storage device 30 according to an embodiment of the present application is shown, wherein the device 30 is used to implement a data storage method 20 according to an embodiment of the present application.

[0077] Optionally, the device 30 described in this application embodiment includes multiple functional components that work together, and each component executes the data processing flow in a predetermined order. Specifically, after a user initiates a data write request to the server through a terminal, the data to be written is first written to a first log, an example of which is a write-ahead log. The first log is written to the disk in an append-only manner and returns confirmation information after the writing is completed. Therefore, when the device 30 experiences a power outage or abnormal interruption, the device 30 can perform log replay based on the first log to recover data that has not yet been persisted. Subsequently, the data that has been written to the log is written to a memory buffer and forms a queryable structure in memory. In addition, when the data in the memory buffer reaches a preset condition, the device 30 triggers a freeze operation and converts it into a read-only buffer, while creating a new memory buffer to continuously receive write requests. Afterward, the data in the read-only buffer is persisted to the first storage component through sequential reading. Further, when the migration condition is met, the data in the first storage component is extracted and migrated to the second storage component, and stored long-term through a hierarchical structure.

[0078] Optionally, memory buffer 301 is configured to receive data to be written after the first log pre-persistence. This memory buffer 301 resides in memory and maintains a set of key-value pairs in an ordered data structure. Although Figure 3 Only one memory buffer 301 is shown, but those skilled in the art will understand that multiple memory buffers 301 can actually exist, and this application does not limit the number of memory buffers 301. The memory buffer 301 can optionally adopt a skip list structure or a memory B+ tree structure to adapt to different access modes. In the skip list implementation, the device 30 achieves fast positioning through multi-level forward pointers. In the memory B+ tree implementation, the device 30 maintains structural balance through node splitting and merging. Simultaneously, the memory buffer 301 supports a sequential iteration interface so that batch reading can be performed during the subsequent freezing phase. Furthermore, the memory buffer capacity is set through configuration parameters and associated with a first preset threshold. When the data volume reaches the threshold, a freezing process is triggered, thereby avoiding resource contention caused by continuous memory growth.

[0079] Optionally, the read-only buffer conversion unit 302 is configured to perform a freeze and conversion operation when the amount of data in the memory buffer 301 reaches a preset threshold, so as to convert the memory buffer 301 into a read-only buffer. Although Figure 3 Only one read-only buffer is shown, but those skilled in the art will understand that multiple read-only buffers can actually exist, and this application does not limit the number of read-only buffers. Specifically, the read-only buffer conversion unit 302 records the current log position when a freeze is triggered and uses this position as a consistency boundary, thereby ensuring that the data in the read-only buffer corresponds to the log state. Subsequently, the read-only buffer conversion unit 302 stops writing and applies read-only constraints to the data structure, while blocking concurrent modification operations. In addition, the read-only buffer conversion unit 302 can also perform a full traversal of the data in the buffer and generate an ordered view according to the key-value order, thereby ensuring that subsequent write processes do not depend on additional sorting operations. At the same time, the read-only buffer conversion unit 302 creates a new memory buffer and initializes the same data structure and configuration parameters, enabling seamless switching of write paths. Furthermore, the read-only buffer conversion unit 302 can optionally include a data freeze timestamp for subsequent migration strategies to make time-dimensional judgments. Therefore, this conversion process maintains the system's continuous write capability while ensuring consistency.

[0080] Optionally, the first storage component 303 is implemented as a page-based index structure, and optionally as a B+ tree structure. The first storage component 303 is organized on disk in units of pages, and pointer relationships between nodes are established through page numbers or offsets. Specifically, the root node resides in memory or cache to accelerate access, intermediate index nodes are used to narrow the search range, and leaf nodes are used to store complete key-value data. Furthermore, leaf nodes form sequential access paths through linked pointers, thereby supporting range scan operations. During writing, ordered data from the read-only buffer is inserted into the leaf nodes sequentially, and sequential writing is used as much as possible to reduce disk addressing costs. For example, when consecutive key values ​​fall within the same leaf node range, batch writing can be completed within a single page, thereby reducing the number of page splits. When the node capacity reaches its limit, a split operation can be performed, and intermediate keys are pushed up to the parent node, thereby maintaining the balance of the tree structure. In addition, the device 30 optionally maintains a page cache and dirty page marking, and optimizes disk I / O behavior through a delayed write-back mechanism.

[0081] Optionally, the second storage component 304 is implemented as a sequential file storage structure and used to store data with low access frequency. This structure employs a multi-level organization, with each level differentiated by capacity and priority. Specifically, each level consists of multiple ordered files, referred to as SSTables, whose internal data is arranged in key-value order and includes auxiliary structures such as sparse indexes or Bloom filters to support fast location. For example, during a query, a Bloom filter can be used to determine if the target key exists in the file, thereby reducing unnecessary disk reads. Furthermore, data is consolidated between levels using a merging strategy, gradually pushing data down to lower levels. As the number of levels increases, the capacity of a single level expands while the update frequency decreases. Simultaneously, the sequential write method reduces random I / O overhead and allows for compression encoding during the write process, further reducing storage footprint.

[0082] Optionally, the persistent write unit 305 is configured to write data from the read-only buffer to the first storage component 303 and write page changes and index structure adjustments of the first storage component to the second log. Specifically, when performing a flush operation, the persistent write unit 305 reads data from the read-only buffer in batch mode and outputs the key-value sequence through a sequential scan interface. Subsequently, the persistent write unit 305 performs a write operation based on the target leaf node range and maintains the context information of the current write position. For example, the persistent write unit 305 may check the remaining space of the current page to determine whether to trigger a page switch or node split. In addition, the persistent write unit 305 may optionally generate verification information, such as checksums or version flags, during the write process for subsequent consistency verification. After the write is completed, the persistent write unit 305 compares the statistical information of the source data and the target data, such as the number of data and the key range, to confirm the integrity of the write. Finally, the persistent write unit 305 releases the memory occupied by the read-only buffer and updates the relevant status flags, thereby completing a full flush cycle.

[0083] Optionally, the merging unit 306 is configured to perform a data migration operation when the first storage component 303 meets the migration conditions. Specifically, this unit determines the migration trigger timing based on data volume, storage space occupancy, or a time strategy, and locks the corresponding data range after triggering to avoid interference from concurrent modifications. Subsequently, the unit sequentially reads data from the leaf node linked list and constructs a continuous data stream. Then, the unit performs a merge process with the data in the starting data layer of the second storage component 304, performing deduplication, overwriting, and deletion mark cleanup operations during the merge process. For example, when multiple versions of the same key exist, the system only retains the latest version of the data. Furthermore, after the merge is completed, the unit writes the results to the target layer in sequential file format and generates corresponding indexes and auxiliary structures. Finally, the unit deletes the migrated data pages in the first storage component and reclaims disk space, thereby maintaining system storage balance.

[0084] Optionally, the device 30 also includes additional functional components to support overall operation. For example, a log storage unit maintains a first log file and performs log truncation and archiving operations; a monitoring unit collects operational metrics of each component, such as memory usage, disk throughput, and migration progress; and a query unit performs lookup path selection in the multi-layered storage structure and accesses the memory buffer, the first storage component, and the second storage component according to priority. Furthermore, the query unit may optionally combine caching mechanisms and filtering structures to reduce access latency. Therefore, the components work collaboratively to form a stable data storage and access system, which can be adjusted according to configuration parameters in different embodiments.

[0085] Next reference Figure 4 and Figure 5Describe some optional details of operation S203. Figure 4 A flowchart illustrating the persistent writing of data in a read-only buffer to a first storage component according to an embodiment of this application is shown. Figure 5 A schematic diagram illustrating the organization of data in a read-only buffer according to an embodiment of this application is shown.

[0086] As previously mentioned, the data in the read-only buffer originates from the immutable memory table frozen in memory buffer 301 and has been sorted by key-value during the conversion phase. Therefore, during subsequent disk flushing, the data is processed sequentially in ascending key-value order, forming a continuous, ordered input sequence. Based on this ordered characteristic, device 30 can reduce repeated path search operations when writing to the first storage component (B+ tree structure), thereby reducing disk access overhead. Furthermore, device 30 introduces a cursor structure, which is used to determine the leaf page information corresponding to the current data write position and locates the target page after a complete search from the root node to the leaf node. Therefore, when processing subsequent data, device 30 prioritizes determining the insertion position based on the cursor, thereby avoiding re-execution of the complete search process in most cases.

[0087] Optionally, operation S203 as described above may include, but is not limited to, the following operations S401 to S406. First, in operation S401, device 30 iterates over each piece of data in the read-only buffer. Since the data in the immutable memory table is already sorted by key value, device 30 processes the data one by one in ascending order of key value.

[0088] For example, such as Figure 5 As shown, the read-only buffer can contain sets of data with keys of 5 and 7, where each data entry includes fields such as key, value, and version number. For example, one data entry can be represented as the value "abc" for key 5, with a version number of 9; another data entry can be represented as the value "de" for key 7, with a version number of 6. Therefore, device 30 processes the data with key 5 first, and then processes the data with key 7, thereby ensuring the order of the input sequence.

[0089] Optionally, in operation S402, device 30 determines whether the data for the current iteration exists on the page where the current cursor is located. Specifically, operation S402 includes sub-operations S4021 to S4023 (sub-operations S4021 to S4023 are not shown).

[0090] In operation S4021, device 30 compares the key value of the current data with the set of key values ​​of the data already existing in the page where the current cursor is located. Since the key values ​​within the page are also arranged in ascending order, device 30 can perform a fast comparison operation based on the ordered structure without traversing all the data.

[0091] Next, in operation S4022, in response to the current data's key value matching a key value on the page, device 30 determines that the data already exists on the current page. In operation S4023, in response to the current data's key value not matching any key value on the page, device 30 determines that the data does not exist on the current page.

[0092] For example, when the cursor is currently pointing to leaf page P1, and page P1 already contains data with key value 5, when device 30 processes data with key value 5, it determines the matching relationship through operations S4021 and S4022, thereby skipping the data and returning to operation S401 to continue processing the next piece of data. Therefore, this operation is used to avoid duplicate writing and maintain data uniqueness.

[0093] Optionally, in operation S403, in response to determining that the data for the current iteration already exists on the page where the current cursor is located, device 30 skips the current data and continues iterating over the next data.

[0094] Optionally, in operation S404, in response to determining that the data of the current iteration does not exist on the page where the current cursor is located, the device 30 further determines whether the data belongs to the key value range of the current page. Specifically, operation S404 includes sub-operations S4041 to S4043 (sub-operations S4041 to S4043 are not shown).

[0095] In operation S4041, device 30 reads the maximum key value field of the page where the current cursor is located and compares the maximum key value with the key value of the current data. In operation S4042, in response to the current data's key value being numerically less than the maximum key value, device 30 determines that the data still belongs to the key value range of the current page. In operation S4043, in response to the current data's key value being greater than or equal to the maximum key value, device 30 determines that the data does not belong to the current page's range.

[0096] For example, when the maximum key value of page P1 is 10, and the current data key value is 8, device 30 determines through operation S4042 that the data can still be inserted into page P1; when the current data key value is 12, device 30 determines through operation S4043 that the data should be inserted into another page. Therefore, this operation uses key value range judgment instead of a complete search, thereby reducing computational overhead.

[0097] Optionally, in operation S405, in response to determining that the key value of the current data is less than the maximum key value of the page where the current cursor is located, device 30 directly inserts the data into the current page. Since the input data is generally ordered, and the current data is usually within the range of the current page, this direct insertion operation is successful in most cases. For example, when the consecutive data key values ​​are 5, 6, 7, and 8 and the current page range is [1, 10], device 30 continuously executes operation S405 to complete the insertion without performing a tree structure search. Therefore, this operation utilizes local ordering to improve write efficiency.

[0098] Optionally, in operation S406, in response to determining that the key value of the current data is not less than the maximum key value of the page where the current cursor is located, device 30 performs a search in the first storage component to locate the insertion position. Specifically, device 30 starts from the root node of the B+ tree, traverses intermediate nodes layer by layer, and finally locates the target leaf page. After the insertion is completed, device 30 updates the cursor to the new page, thereby providing a new reference position for subsequent data. For example, when the maximum key value of the current page is 10, and the subsequent data key values ​​are 12 and 15 respectively, device 30 performs a complete search and updates the cursor in operation S406. Afterwards, the data with key value 15 can continue to perform local judgment based on the new cursor. Therefore, operation S406 is used to handle cross-page write scenarios and complements operation S405, thereby reducing the overall number of searches while ensuring correctness.

[0099] Next, refer to Figure 6 Some optional details of embodiments of this application are described below. Figure 6 A schematic diagram of multi-version data storage under a hybrid architecture according to an embodiment of this application is shown.

[0100] like Figure 6 As shown, the first storage component 303 and the second storage component 304 employ different processing architectures for multi-version data. The first storage component 303 includes, but is not limited to, the target leaf node 601 and an external version list 603 independent of the target leaf node 601. This solution aims to address the efficiency issues of multi-version storage and retrieval when the same data record is updated multiple times in a high-concurrency database system.

[0101] In the actual operation of a database system, the same data record (i.e., the same key value) often undergoes multiple update operations, resulting in multiple copies of data at different times. "Multiple versions of the same key value" refers to a sequence of values ​​with the same primary key (e.g., user ID) but modified at different points in time. For example, a user configuration might be modified multiple times within a short period, resulting in version 1, version 2, and version 3. To efficiently manage this multi-version data, in response to determining that the data in the current iteration includes multiple versions of the same key value, device 30 performs a set of chained separation storage steps when inserting the data of the current iteration into the page where the current cursor is located or the page where the insertion position is located.

[0102] Specifically, device 30 first identifies and extracts the "latest version data" 602 from these multiple versions. The latest version data refers to the record with the latest timestamp or the largest transaction sequence number, representing the current effective business state of that key value. Device 30 treats this latest version data 602 as the only valid entity and directly inserts it into the target leaf node 601 of the page where the current cursor is located or the page where the insertion position is located. The target leaf node refers to the physical storage block in the first storage component (i.e., the paged index structure) that actually carries the business data record. Simultaneously, device 30 extracts the "historical version data" 604 (i.e., the aforementioned version 1 and version 2) in addition to the latest version, and allocates new linked list node space in memory or a separate disk log area, outside the physical storage space of the main data page. Device 30 stores the historical version data 604 of the same key value in a version linked list 603 outside the paged index structure. The version linked list is a one-way or two-way pointer structure specifically used to chain the historical states of the same key value in reverse chronological order. To associate master data with historical states, device 30 records a pointer to the head node of the version list 603 at a fixed offset of the latest record in the target leaf node 601. The pointer is a memory address or file offset identifier, serving as the sole entry point for accessing historical data.

[0103] Compared to the traditional approach of storing all new and old versions of data contiguously in B+ leaf nodes, the solution described in device 30 offers significant advantages. In traditional methods, when performing range queries (e.g., scanning the latest status of a large number of users), the scanning mechanism inevitably reads a large number of expired historical versions. This invalid data occupies valuable page space, drastically reducing the number of valid latest records that can be read in a single disk I / O, resulting in a severe "read amplification" effect. Device 30, through the aforementioned implementation steps, ensures that the target leaf node 601 always contains only the purest latest version data, allowing any range scan of the current status to be completed with minimal page reads. This achieves the technical effect of minimizing read amplification in range queries and maximizing the performance of hot data scanning.

[0104] Furthermore, as the data lifecycle progresses and the device 30 merges the data from the first storage component 303 into the second storage component 304, different processing steps are employed for multiple versions of data to adapt to underlying characteristics. Specifically, in response to the existence of multiple versions of data with the same key value in multi-way merge sort (e.g., ... Figure 6 The second storage component 304 writes the historical versions (key 1, version 2) and the latest versions (key 1, version 3) of the data in the initial data layer 610 of the second storage component in version order, adjacent and consecutively. "Adjacent and consecutive writing" means that in the generated persistent physical file, each version is tightly connected in byte offset, without mixing with data from other key values. The device 30 outputs these sorted versions sequentially to the file buffer and finally writes them to disk through a streaming write interface. Unlike the first storage component 303, which aims to improve scanning performance, the second storage component 304 is mainly used to store cold data to reduce storage costs. Since multiple versions of the same key value typically only have a few fields that change, they have a high degree of similarity. By storing them physically adjacently, the device 30 allows the underlying compression engine to fully utilize prefix compression and dictionary matching techniques to eliminate a large number of redundant duplicate bytes. This achieves the technical effect of significantly improving data compression ratio in the cold data layer and significantly reducing the physical storage cost of massive amounts of data.

[0105] Next, refer to Figure 7 Some optional details of embodiments of this application are described below. Figure 7 A schematic diagram illustrating the updating of the first storage component 303 and the memory mapping table 710 according to an embodiment of this application is shown.

[0106] like Figure 7As shown, the first storage component 303 according to the embodiments of this application can be updated through "append write" to solve the two core problems of disk random write performance bottleneck and cascading write amplification caused by node updates in traditional paged index structures (such as B+ trees) under high-concurrency write scenarios. Figure 7 The following objects are shown: physical data file 701 corresponding to the first storage component 303, memory mapping table 710 residing in memory, logical identifier 711 representing the logical identity of the page, and physical offset 712 representing the physical location of the page.

[0107] Traditionally, in database storage engines built on page-based index structures such as B+ trees, data write, update, and delete operations ultimately modify index pages. These index pages include both leaf nodes carrying user business data and intermediate index nodes responsible for routing. In high-concurrency database operations, frequent transaction writes continuously trigger data insertion, updates, and deletions in leaf nodes. When the remaining storage space on a page is insufficient to accommodate newly written data, node splitting operations are triggered, generating entirely new index pages. Traditional relational databases and storage engines generally employ an "in-place overwrite" update mechanism for these page modification and addition scenarios. This mechanism has a fatal performance flaw under high-concurrency write conditions.

[0108] Specifically, in traditional solutions, when the content of an index page changes, traditional database storage engines directly overwrite the original physical storage location of that page on the disk with the new page data. When a node splits to create a new page, it also searches for a random location in the free blocks of the disk file to write to. This update mechanism, on the one hand, transforms a large number of scattered page modification operations into random disk I / O requests. The hard disk drive head needs to frequently jump between different tracks. Even for solid-state drives, random write performance is lower than sequential write, ultimately leading to a decrease in database write throughput and an increase in write latency. On the other hand, in the index nodes of traditional B+ trees, the parent node achieves routing addressing through the physical offset of the child node. When the physical location of the child node changes due to in-place overwriting or splitting, the pointer recorded in the parent node must be updated synchronously. The update of the parent node will cause its own physical location to change, thereby triggering a chain update from the grandparent node to the root node, i.e., "cascading update". This cascading update will produce a significant write amplification effect—a single modification to a leaf node may trigger the rewriting of multiple layers of index nodes, greatly amplifying the amount of disk writes. This not only consumes the IO bandwidth of the storage medium but also shortens the lifespan of the flash memory medium, making it difficult to adapt to high-concurrency write business scenarios.

[0109] To address the aforementioned issues of random write and cascade write amplification, the apparatus 30 of this application embodiment provides the following end-to-end append write and indirect addressing mechanism for the page persistence write process of the first storage component 303.

[0110] First, during the data persistence writing process, when device 30 detects that the first storage component 303 generates a "new page" 703 to be written or updates a page in the first storage component (hereinafter referred to as "updated page 703"), it no longer performs the traditional in-place overwrite operation. Instead, it calls the append write interface of the underlying file system to append the new page or updated page to the end of the persistent physical data file 701 corresponding to the first storage component. Here, a new page refers to an index page generated by the B+ tree index due to node splitting or the creation of new branches, including new leaf node pages and intermediate index node pages. An updated page refers to a page copy containing the latest data state generated after the content of the original index page has changed due to data insertion, modification, deletion, or other operations.

[0111] According to the "append-write method" in the embodiments of this application, regardless of the offset position of the original version of the page to be written in the physical data file, all newly generated page data is appended to the end of the physical data file as a continuous byte stream, without modifying or overwriting the historical data area in the file, thus avoiding random write operations. Through this step, the device 30 transforms the performance-intensive random disk write in the traditional solution into a more physical storage medium-friendly sequential disk write, improving the write bandwidth utilization of the storage medium.

[0112] However, the append-only mechanism may cause the physical storage location of the same logical page to change with each update. If the traditional physical addressing method is used, the problem of cascading updates of index nodes still cannot be solved. To address this, this application provides a two-layer decoupled addressing architecture of "logical-physical" for all index pages, and performs update and maintenance steps for the memory mapping table 710 accordingly.

[0113] Specifically, after completing the append write to a page, device 30 synchronously obtains the "logical identifier" 711 corresponding to the page, and the "physical offset" 712 newly allocated in the physical data file 701 after this append write. The logical identifier is a globally unique and fixed virtual numerical ID assigned to the page by device 30 when the index page is first created. This identifier is bound to the page's logical function and its identity in the index topology, and will not change with modifications to the page content or changes in physical storage location. The physical offset refers to the starting absolute byte address of the data block written to the page in this append write in the physical data file. The latest complete data of the page can be read from the physical data file using this offset.

[0114] The device 30 adds or updates the logical identifier 711 in the memory-mapped table 710 residing in memory. Compared with the in-situ update + physical addressing scheme of the corresponding physical offset and the traditional B+ tree, the above architecture of this application embodiment has significant performance advantages. In the traditional scheme, a single leaf node update may trigger cascading rewrites of multiple layers of index nodes, and the write amplification effect will increase with the number of layers of the B+ tree, which will consume a lot of storage I / O resources in high-concurrency write scenarios; while the device 30, through the above combination of append write + logical addressing + memory-mapped table, controls the disk write volume corresponding to a single data update to only write the modified page itself, and no additional index node write overhead will be generated regardless of the number of layers of the B+ tree, effectively reducing the write amplification effect.

[0115] Meanwhile, this embodiment improves write performance while reducing the impact on data read efficiency. When device 30 receives a data query request, it first uses the intermediate index node of the B+ tree to complete routing navigation based on the logical identifier, locating the logical identifier corresponding to the target leaf node; then, it obtains the latest physical offset corresponding to the logical identifier through a hash lookup of the memory-mapped table; finally, it reads the latest complete data of the target page from the physical data file using this physical offset. The entire read process only adds one memory hash lookup operation, which has a small impact on read performance, while ensuring that the read operation can obtain the latest version of the page data, thus guaranteeing data consistency.

[0116] Therefore, through the above-described architecture design, this application transforms random writes into sequential writes, improving the write bandwidth utilization of the storage medium and increasing the high-concurrency write throughput of the database. Simultaneously, by decoupling the logical topology from the physical storage, the write amplification problem caused by index cascading updates is avoided, while simultaneously balancing data read performance and data consistency, making it suitable for high-concurrency, low-latency business scenarios.

[0117] Next, refer to Figure 8 Some optional details of embodiments of this application are described below. Figure 8 A flowchart illustrating the migration condition determination based on invalid space according to an embodiment of this application is shown.

[0118] like Figure 8 As shown, this application embodiment addresses the problem of continuous expansion of storage space caused by the aforementioned append-write architecture by designing an adaptive migration condition determination and space reclamation triggering mechanism to improve the disk space utilization caused by the accumulation of historical invalid data in append-write mode, and at the same time achieve a dynamic balance between foreground write performance and background space reclamation overhead.

[0119] As mentioned earlier, the append-only mechanism transforms random disk writes into sequential writes, resolving performance bottlenecks and index cascading write amplification issues in high-concurrency write scenarios. However, in append-only mode, whenever an index page is updated, device 30 writes the complete new version of the page to the end of the physical data file 701, rather than overwriting the original page's physical location, and simultaneously updates the physical offset corresponding to the page's logical identifier in the memory-mapped table. This means that the same logical page will generate multiple historical versions of physical data blocks in the physical data file 701 as update operations continue to execute; while the memory-mapped table 710 only retains the latest physical offset corresponding to each logical identifier. Only the latest version of the page data will be queried by the business and referenced by the index routing. The old version of the page data in the physical data file 701 will lose its valid pointer reference, becoming redundant and discarded data that cannot be accessed by the business or participate in any data processing. If this discarded data is not managed and cleaned up, the size of the physical data file will continue to expand with page updates, continuously occupying a large amount of disk storage space even if the amount of valid business data does not increase, resulting in serious resource waste.

[0120] In order to manage redundant and obsolete data, the device 30 can identify the physical storage space in the physical data file 701 that is not pointed to by the physical offset currently recorded in the memory mapping table due to page append writing as "invalid space", also known as data hole or garbage data area. All bytes in this area are obsolete data that are no longer referenced by any logical process.

[0121] Therefore, device 30 maintains a global invalid space byte accumulator. The update of the accumulator's value is bound to the update of the mapping relationship in the memory mapping table. The entire process is completed in memory, without generating additional disk I / O overhead, and has no impact on the performance of the front-end business. Specifically, whenever the physical offset corresponding to any logical identifier in the memory mapping table is updated, device 30 first obtains the original physical offset of the logical identifier and the fixed byte length of the index page, calculates the complete byte range occupied by the old version page, and then accumulates the byte length value corresponding to the page into the global invalid space byte accumulator. At the same time, when a page is deleted or a logical identifier is destroyed, device 30 also synchronously accumulates the number of bytes occupied by the latest version page corresponding to the logical identifier into the invalid space byte accumulator to ensure the completeness and accuracy of invalid space statistics.

[0122] After completing the statistics of the total amount of invalid space, device 30 calculates the proportion of invalid space in the entire physical data file in real time or periodically. Specifically, device 30 first obtains the current total number of bytes of the physical data file corresponding to the first storage component, which is the actual disk space occupied by the file at the operating system level; then, it divides the total number of bytes of invalid space recorded by the aforementioned global accumulator by the total number of bytes of the data file to calculate the corresponding percentage value, i.e., the proportion of invalid space. This proportion value will fluctuate dynamically with page updates and data write operations, which can intuitively reflect the proportion of discarded data in the current data file, and also reflect the degree of waste and fragmentation of storage space. The higher the proportion, the lower the effective utilization rate of disk space and the more redundant historical version data there is.

[0123] Subsequently, device 30 can calculate the percentage of invalid space in real time and continuously compare it with the system's preset percentage threshold to determine the migration conditions. The preset percentage threshold is a configurable core control parameter that can be flexibly adjusted based on the disk resource situation of the business scenario, the frequency of data write and update, and the sensitivity of front-end business performance. For example, for business scenarios with limited disk space, a lower threshold can be set to trigger space reclamation earlier, ensuring effective utilization of disk space; for business scenarios with extremely high write and update frequencies and sensitivity to front-end read / write latency, a slightly higher threshold can be set to reduce the frequency of background space reclamation operations, avoiding impact on front-end business performance.

[0124] When the proportion of invalid space calculated in real time reaches or exceeds the preset proportion threshold, the device 30 determines that the current first storage component 303 meets the triggering condition for data merging and migration to the second storage component 304. Once the migration condition is met, the device 30 will start the merging and writing process from the first storage component to the second storage component (that is, extract the target data block from the leaf node, merge and sort the target data block with the data in the starting data layer 610 of the second storage component 304, and write the sorted data into the starting data layer 610).

[0125] Optionally, during the merging process, the device 30 will only extract the latest valid page data corresponding to all current logical identifiers in the memory mapping table 710, merge them in an orderly manner, and write them down to the second storage component; after all valid data has been migrated, persisted to disk, and passed the data consistency check, the device 30 will directly delete or truncate the old physical data file corresponding to the first storage component, release all invalid space in the file to the operating system at once, and complete the entire space reclamation process.

[0126] It is worth noting that the writing process from the initial data layer 610 to other persistent data layers of the second storage component 304, and the writing process from any persistent data layer in the second storage component 304 to its lower persistent data layer are similar, and will not be described in detail here.

[0127] Compared to the space reclamation triggering mechanisms commonly used in traditional databases, the embodiments of this application have stronger adaptability. Traditional solutions typically employ two types of fixed triggering logic: one is timed triggering based on fixed time intervals, which executes space cleanup at the preset time regardless of the amount of invalid space in the data file. This approach is prone to two extreme situations: either frequent business updates result in severely expanded invalid space, but the triggering time has not yet arrived, leading to continuous waste of disk space; or there are almost no data updates and very little invalid space, yet space cleanup is triggered on time, resulting in unnecessary consumption of system resources and even preempting IO resources of foreground services, affecting read and write performance. The other type is triggering based on a fixed file size, which executes cleanup when the total size of the data file reaches a preset threshold. This approach completely ignores the proportion of invalid space within the file: for example, the business continuously writes new data with very few updates, the total file size grows rapidly, but the proportion of invalid space is extremely low, making it completely unnecessary to trigger cleanup; conversely, the business updates frequently with a small total amount of effective data, but the proportion of invalid space is already extremely high, but because the total file size has not reached the threshold, space reclamation cannot be triggered, leading to continuous disorderly expansion of disk space.

[0128] In the optional embodiments of this application, if a large number of high-frequency data updates occur in a short period of time, although the total amount of effective business data is not large, the repeated appending and writing of pages generates a large number of historical invalid versions, and the proportion of invalid space will rise rapidly. Device 30 can detect this change in real time and immediately trigger migration and space reclamation to avoid invalid occupation of disk space. However, if the system only continuously writes new data and rarely updates pages, even if the total size of the data file continues to grow, the proportion of invalid space will always remain at a low level. Device 30 will not trigger unnecessary migration and reorganization operations, minimizing the impact of background operations on the performance of the front-end business, while avoiding unnecessary consumption of system resources.

[0129] Through the aforementioned adaptive migration condition determination mechanism based on the proportion of invalid space, device 30 achieves fine-grained management of storage space under the append-write architecture. It can accurately control the triggering time of space reclamation, achieve the optimal balance between disk space utilization and system resource overhead, solve the problem of disordered disk space expansion caused by the append-write mechanism, and at the same time, it will not affect the high-concurrency write performance advantage brought by the append-write architecture, ensuring the long-term stable operation of the database in high-concurrency update scenarios.

[0130] In addition to the core migration triggering condition based on the proportion of invalid space mentioned above, the device 30 can also be configured with migration conditions in other dimensions.

[0131] For example, device 30 can pre-configure a maximum physical capacity threshold for the first storage component. This threshold can be flexibly set based on the available disk space of the server, the storage quota of the database instance, the growth pattern of business data, and the write capacity of the second storage component, thereby limiting the maximum disk space that the first storage component can occupy. During operation, device 30 will monitor the current total number of bytes of the physical data file corresponding to the first storage component in real time or periodically. When the total number of bytes of the file reaches or exceeds the preset maximum physical capacity threshold, regardless of whether the current invalid space ratio reaches the preset percentage threshold, device 30 will directly determine that the first storage component meets the migration conditions and immediately trigger the merge migration process to the second storage component.

[0132] Device 30 also supports customizable timed migration strategies. Users can preset time windows during off-peak hours (such as the period with the lowest business traffic in the early morning) based on business traffic patterns, and simultaneously configure two supporting control parameters: the maximum migration interval threshold and the maximum page dwell time threshold. When the system time reaches the preset off-peak time window, device 30 will verify the status of the first storage component: if the time interval since the last full merge migration exceeds the preset maximum migration interval threshold, or if there are valid data pages in the first storage component that have exceeded the preset dwell time, even if the current invalid space ratio has not reached the core threshold and the physical file size has not reached the capacity limit, device 30 will determine that the migration conditions are met and trigger the lightweight merge migration process.

[0133] In addition, the device 30 can monitor the lifecycle events of the database instance in real time. When it detects that the instance has received instructions for normal shutdown, service restart, or primary / standby role switching, it will directly determine that the migration conditions are met, regardless of whether other migration conditions are met, and forcefully trigger the full merge migration process from the first storage component to the second storage component.

[0134] Next, refer to Figure 9 Some optional details of embodiments of this application are described below. Figure 9 A schematic diagram is shown illustrating the writing of sorted data into the starting data layer according to an embodiment of this application.

[0135] like Figure 9As shown, the components involved in writing sorted data into the starting data layer include, but are not limited to, a frozen first storage component 901, a first storage component 902 for transfer, leaf node groups 903 logically divided based on key value ranges, the starting data layer 904 in the second storage component that receives the data, overlapping data blocks 905 that match the key value range of the current group, and a multi-way merge sorter 906 for locally ordered data merging. Although different reference numerals are used, those skilled in the art will understand that the functions of the above components are the same as or similar to the components described above.

[0136] In response to the first storage component meeting the aforementioned migration triggering conditions, device 30 marks the first storage component 901, which is currently serving front-end read and write operations, as read-only frozen by modifying the status flag bit in the memory metadata, thus forming a frozen first storage component 901. This freezing operation only modifies the status bit in memory and will not block ongoing read requests. After freezing, the internal B+ tree structure, all page data, and memory mapping table of the component are completely locked, and no new insert, update, or delete write requests are accepted. It can only respond to front-end read requests, ensuring that the data source to be merged remains static during the merging process and avoiding consistency issues caused by data changes during the merging process.

[0137] Within the same transaction that performs the freeze operation, device 30 initializes a brand new, empty first storage component in memory, forming a first storage component 902 for receiving newly written data as a transfer. This first storage component 902 for transfer completely inherits the configuration parameters, index structure, page specifications, and memory mapping table mechanism of the original first storage component. It is a complete and usable paged index storage instance. The initialization process is completed only in memory, without the need to create physical files or pre-allocate disk space, and can be completed synchronously with the freeze operation.

[0138] After completing the above two steps, device 30 immediately performs atomic redirection of front-end business read and write traffic, redirecting all new write requests to the newly initialized first storage component 902 used for relay. For front-end read requests, device 30 adopts a two-layer merge query mechanism, first querying the new relay component, and if it does not find the target, then continuing to query the frozen first storage component 901, ensuring that read requests can obtain the full amount of the latest data and that there will be no data loss or inability to read the data.

[0139] Next, device 30 logically decomposes the originally one-time full merge task into multiple independent, lightweight local merge tasks. Device 30 first traverses the frozen B+ tree metadata of the first storage component 901 to obtain the total number of all leaf nodes, the key value range corresponding to each leaf node, and the global sorting information of the key values. The leaf nodes of the B+ tree are globally ordered in ascending order of key values, and the key value ranges of adjacent leaf nodes are continuous and non-overlapping, providing a basis for the orderliness of subsequent logical grouping. Subsequently, device 30 dynamically determines the specifications and number of groups based on the current real-time operating status. Key reference indicators include the system's current available memory space, CPU idle rate, disk I / O bandwidth margin, and read / write pressure of front-end services. It also combines preset thresholds for the maximum data volume of a single group and the longest time threshold for a single group to merge. For example, when the front-end service is in a peak period and CPU and I / O resources are low, device 30 will reduce the key value range and data volume of a single group, splitting it into more groups, so that each small merging task takes less time and consumes less resources. When the front-end service is in an off-peak period and system resources are sufficient, the data volume of a single group can be appropriately increased, the number of groups can be reduced, and the overall merging progress can be accelerated. After determining the grouping specifications, device 30 can logically divide consecutive leaf nodes into multiple leaf node groups 903 according to the ascending key value order of B+ leaf nodes. This grouping operation is only a logical key value range division and will not physically move, modify or copy the data in the frozen first storage component. It only sets a corresponding start key and end key for each group and clarifies the key value range covered by the group. It will not generate any disk I / O overhead and will not occupy additional memory space.

[0140] After the grouping and decomposition are completed, device 30 starts an independent background merging thread, strictly following the ascending key-value order, iteratively processing each leaf node group and performing lightweight local merging operations. For the target leaf node group being processed in the current iteration, device 30 first obtains the start key and end key corresponding to the group, clarifying the key value range it covers. Then, based on this key value range, it performs interval filtering on the file list of the starting data layer 904 of the second storage component, extracting all data blocks 905 in the starting data layer 904 whose key value range overlaps with the key value range of the current group. The starting data layer of the second storage component is the first layer in the LSM tree architecture that carries the upper-level data. The data blocks in it are all arranged globally in key-value order, and each data block records its own key value range. By extracting overlapping data blocks through interval filtering, it can ensure that the upper-level data of the current group can be completely merged with the existing data of the same range at the lower level, ensuring that the data written after merging still maintains a globally strict order, while avoiding reading data blocks unrelated to the current group and reducing unnecessary disk I / O overhead.

[0141] After data block extraction is completed, device 30 initializes the multi-way merge sorter 906 in memory and simultaneously enables input of multiple ordered data streams. The core input streams include ordered data streams read in ascending order of key values ​​from the leaf node group of the current iteration, and ordered data streams read in ascending order of key values ​​from the extracted overlapping data blocks 905. Subsequently, the multi-way merge sorter 906 performs multi-way merge sorting operations on the multiple ordered data streams. This operation is implemented based on a min-heap data structure. By comparing the key values ​​at the heads of the multiple input streams simultaneously, the data record corresponding to the globally smallest key value is continuously popped, and finally, a globally strictly ordered data stream is output. During the merging process, for multiple versions of data with the same primary key, the sorter automatically retains the latest version of the data and discards expired historical versions, thereby achieving data deduplication and version merging, further reducing data redundancy.

[0142] The device 30 encapsulates the globally ordered data stream output by the multi-way merge sorter 906 into fixed-size ordered data blocks according to the standard data block specifications of the second storage component. The data blocks are then written into the starting data layer 904 of the second storage component via a streaming sequential write method. After the write is completed, the device 30 will synchronously update the metadata of the second storage component, recording information such as the key value range, physical location, and data checksum of the newly written data blocks. At the same time, it will mark the key value range that has been completed in this merge process. Subsequent read requests from the foreground will directly read the data within the corresponding range from the new data blocks in the starting data layer, and will no longer access the corresponding group data in the frozen first storage component.

[0143] During the process of processing groups one by one, after completing the merging operation of each group, device 30 can proactively release the memory space, CPU resources, and corresponding files in the starting data layer 904 occupied by the merge. At the same time, during the interval between two group merges, it proactively yields system resources to foreground business requests. This segmented merging mechanism also has fault tolerance capabilities. If an abnormal interruption occurs during the merging process of a group, device 30 only needs to re-execute the merging task of that single group, without having to re-execute the full merge. After all group merging operations are completed and all data is successfully persisted to the starting data layer of the second storage component, device 30 will perform a merge cleanup operation, destroying the frozen first storage component 901, releasing its corresponding physical data files and memory space, and completing the entire merge sinking process. At the same time, foreground read requests no longer need to access the frozen storage component and are all redirected to the new first storage component (e.g., the first storage component 902 used for relay) and the second storage component, completing a smooth switch of the entire storage architecture.

[0144] Compared with traditional storage engine merging schemes, the segmented progressive merging architecture of this application embodiment has significant performance and availability advantages. In database storage engines designed based on LSM tree architecture, traditional storage engine merging schemes generally adopt a full locking plus one-time full merging approach, which may lead to the following problems: 1) Full merging will consume a large amount of CPU, memory, and disk I / O resources in a short period of time, competing with front-end business requests for system resources, resulting in a significant increase in the latency of front-end read and write requests and causing serious system jitter; 2) If an instance crashes, process is interrupted, or other abnormal situation occurs during the full merging process, the entire merging task needs to be completely re-executed, which will not only prolong the merging time but also further aggravate the unavailability time of front-end services.

[0145] The solution in this application eliminates write blocking during the merging process through dual-component atomic switching, ensuring the continuous availability of front-end services and avoiding write interruptions and latency peaks in traditional solutions. By breaking down the full merging into small group merging tasks, the originally intensive system resource consumption is smoothly distributed over a longer time window, completely eliminating system jitter caused by the merging operation and ensuring the stability of latency for high-concurrency business requests. At the same time, the progressive merging process can dynamically adjust the merging speed according to the current business pressure of the system, achieving dynamic resource balance between background merging tasks and front-end services. Without affecting the performance of front-end services, the data merging and sinking and space reclamation process is completed smoothly. In addition, the single-group-level fault tolerance capability also greatly reduces the impact of abnormal scenarios on the merging progress and avoids repeated full resource consumption.

[0146] Next, refer to Figure 10 Some optional details of embodiments of this application are described below. Figure 10 A schematic diagram illustrating fault recovery using hybrid logs according to an embodiment of this application is shown.

[0147] like Figure 10 As shown, for the multi-layer hybrid storage architecture of this application embodiment, in one aspect of this application, an improved hybrid log checkpointing and fault recovery method is provided, the components of which include, but are not limited to, a first log 1001 that records changes to user business data and a second log 1002 that records changes to cache index structure.

[0148] As mentioned earlier, in the multi-layered hybrid storage architecture of this application, the first storage component may adopt an append-write logical addressing architecture, a multi-version storage design with hot and cold tiers, and a dual-component merging and switching mechanism. These mechanisms can improve read and write performance, but they also bring higher complexity to data consistency assurance and fault recovery. Traditional single log write-ahead schemes either cannot balance write performance and data persistence security, or require full replay of massive historical logs during fault recovery, resulting in extremely long recovery times and even serious problems such as duplicate data writing and inconsistent index structures. Therefore, device 30 further designs a hybrid log and fault recovery mechanism, as detailed below.

[0149] To achieve the optimal balance between extreme write performance and absolute data security, device 30 can use a dual-log scheme that separates business data from index structure to adopt differentiated persistence strategies for two different types of change operations.

[0150] Specifically, when receiving write, update, and delete transaction requests initiated by front-end users, device 30 follows the core protocol of the database write-ahead log, synchronously persisting the first log 1001, which records data changes in the memory buffer, to non-volatile disk storage in real time. The core content of the first log is the original key-value pair insert, update, and delete instructions of the user's business, completely recording the complete operation semantics and full data content of the transaction. Device 30 waits for the corresponding transaction record in the first log to be accurately written to disk and receives a persistence confirmation acknowledgment from the disk before returning a successful write confirmation signal to the user client. This step is the bottom line to ensure that user business data is absolutely not lost. Even if the system experiences a sudden crash, all transaction data that has been successfully confirmed to the user can be completely recovered through the first log, and no data loss will occur.

[0151] During the background data persistence process, when device 30 persists the data in the memory read-only buffer to the first storage component, the B+ tree index structure undergoes frequent internal adjustments, including page updates of leaf nodes, node splitting, routing information adjustments of intermediate index nodes, and mapping changes between page logical identifiers and physical offsets. If these high-frequency index structure change operations were also synchronously flushed to disk in real time like the first log, it would generate a large amount of random disk I / O, resulting in excessive performance overhead. Therefore, device 30 temporarily stores the second log 1002, which records page changes and index structure adjustments in the first storage component, in a cache dedicated to the second log 1002 in memory. It is not written to disk in real time. Only when the accumulated data volume of the cached second log reaches a preset batch flush threshold, or when the system enters an idle state or a checkpoint process is triggered, is the cached second log batch and sequentially flushed to the disk for persistent storage. The second log records the redo instructions for physical pages, fully recording the changes to the index pages, logical identifiers, and previous and subsequent physical offsets, which can be directly used for page redo reconstruction.

[0152] Therefore, as long as the first log fully persists the user's original business data, even if the system crashes suddenly and all the second logs that have not been flushed to disk in memory are lost, the device 30 can reconstruct the complete index structure and page data by fully replaying the first log 1001, without any data loss. At the same time, the batch flushing of the second log 1002 significantly reduces the number of disk I / O operations and greatly improves the system's write performance.

[0153] To avoid the need to replay the entire historical log during system failure recovery and significantly shorten the recovery time, device 30 implements a dynamically advancing checkpoint process, continuously updating the safe replay starting point for failure recovery and reducing the amount of logs that need to be replayed. Device 30 continuously monitors internal system trigger signals, including preset checkpoint timed trigger signals, system idle signals during off-peak hours, and signals indicating completion of batch flushing of the second log cache to disk. Upon receiving an internal trigger signal to perform a storage status check, device 30 initiates the checkpoint advancement process.

[0154] In this process, the first step is to perform checkpoint calculations based on the first storage component. Device 30 first scans the list of dirty pages in memory for the first storage component. A dirty page refers to a cached page whose content has been modified in memory but has not yet been persisted to disk via append-only. Each dirty page corresponds to one or more log records that have not been persisted, and each log record has a globally unique and continuously increasing log sequence number. Device 30 then extracts and calculates the minimum log sequence number from all the log sequence numbers corresponding to all dirty pages. This minimum log sequence number represents the oldest memory state in the current system that has not yet been fully persisted, which is the absolutely safe starting point for fault recovery operations. By replaying the logs from this starting point, all unpersisted memory states can be fully recovered without data loss.

[0155] After confirming that a read-only buffer has been written to the first storage component, the device 30 will immediately force a batch flush of all second log records in the memory second log cache that have not yet been flushed to disk to the disk persistent storage, ensuring that all index structure change records after this safe starting point have been completely and safely stored in non-volatile storage, avoiding log loss caused by a crash during the checkpoint advancement process.

[0156] Under this mechanism, device 30 maintains dual checkpoints to finely control the replay starting point. First, the second checkpoint is defined as the smallest log sequence number corresponding to all dirty pages in the first storage component, and it is synchronously persisted to disk when the first storage component performs a checkpoint operation. Second, when the buffer completes the checkpoint operation, i.e., the second log completes a forced flush and receives a persistence confirmation from the disk, device 30 triggers an update of the first checkpoint. Each read-only buffer corresponds to a safe starting point in the first log. At this time, records in the first log corresponding to the safe starting point before this safe starting point are no longer needed, and log space in the first log earlier than the safe starting point can be released. At this time, device 30 uses this safe starting point as the latest system replay point, writes it to a separate metadata file on the disk for persistence, and overwrites the old replay records. Each checkpoint advancement substantially moves the log starting point that needs to be replayed during fault recovery backward, thereby significantly reducing the log volume for replay and shortening the recovery time. If the system crashes and restarts, the first and second storage logs will start collaborative replaying from their respective persisted replay points to ensure the complete recovery of the system state.

[0157] When the system encounters a sudden power outage, process crash, or other fault and restarts, the device 30 responds to the fault recovery trigger signal and executes the full-link consistency recovery steps in the order of physical structure first and then business data.

[0158] First, since the underlying index physical structure is intact and usable, it is the basis for the recovery of upper-layer business data logic. Device 30 will first read the second log 1002 persisted in the disk, and at the same time read the second checkpoint, locate the log record position corresponding to the second checkpoint in the second log, and start from that position to perform the redo replay operation of the second log.

[0159] Device 30 parses the physical page redo instructions in the second log line by line, reallocates the corresponding page storage space, adjusts the pointer relationship of the index nodes, and restores the complete data content of the page, thereby recovering all data pages in the first storage component that were damaged or not written to disk due to the fault, until the end of the second log, thus completing the full recovery of the physical index structure of the first storage component.

[0160] After completing the playback of the second log 1002, confirming that the physical index structure of the first storage component has been fully restored to consistency, and that all page data is complete and available, the device 30 will begin reading the first log 1001 from the disk and performing the playback operation of the business data. The purpose is to reload the business data that was in the memory buffer when the fault occurred and had not yet been persisted to the first storage component into the memory buffer, and restore it to the business state before the fault.

[0161] During this process, if some business data may have already been fully persisted from the memory read-only buffer to the first storage component before the system crash, but the historical record cleanup operation corresponding to the first log was not executed in time due to system scheduling delays or the checkpoint not yet being implemented, these persisted data records will still remain in the first log file. If the first log is blindly replayed in full during fault recovery, this persisted data will be repeatedly inserted into the memory buffer, causing data redundancy and potentially leading to primary key conflicts, memory overflows, and errors in business data statistics. To address this, device 30 can also synchronously perform an existence check for each record to be replayed during the first log replay process.

[0162] Specifically, device 30 first extracts the primary key value corresponding to the current playback record, performs a precise key value retrieval in the full index of the first storage component that has been restored, and checks whether a persistent version of the current playback record already exists in the first storage component, and whether the transaction sequence number of the persistent version is not less than the transaction sequence number of the current playback record.

[0163] In response to the confirmation that a valid persistent version of the record already exists in the first storage component, the device 30 determines that the record has been persisted and does not need to be replayed. When restoring the memory buffer, the write operation of the data record is skipped directly. Only when a valid persistent version of the key value does not exist in the first storage component will the record be written to the memory buffer to complete the data recovery.

[0164] Through this existence detection mechanism, device 30 ensures that all business data confirmed to users is lost without loss, and perfectly avoids problems such as duplicate data writing, primary key conflicts, and memory overflow caused by repeated log replay, thus guaranteeing the consistency and accuracy of business data after fault recovery.

[0165] Furthermore, in the architecture of this application, the memory-mapped table used to implement the mapping between logical identifiers and physical offsets resides entirely in the memory cache. If the system suddenly crashes, the entire contents of the memory-mapped table will be completely lost. Since the memory-mapped table is the core foundation for index addressing of the first storage component, it must be completely rebuilt during fault recovery in order to restore normal read and write services.

[0166] To this end, device 30 fully utilizes the physical characteristics of append-write files to perform reverse-order scan reconstruction. Specifically, after restoring the physical structure of the first storage component, device 30 initiates the memory-mapped table reconstruction process. First, it positions the file read pointer directly to the end of the physical data file corresponding to the first storage component, which is the location of the latest written data. Then, starting from this position, it performs continuous, page-aligned reverse-order read operations towards the beginning of the file, scanning the physical pages in the file one by one.

[0167] When reading the metadata information of the scanned page header one by one, the device 30 will simultaneously extract the globally unique logical identifier corresponding to the page and the physical offset corresponding to the current scan position of the page. When processing the mapping information of each page, the device 30 will first check whether the mapping record corresponding to the logical identifier already exists in the currently reconstructed memory mapping table.

[0168] If the memory-mapped table does not contain a record for the logical identifier being read, it means that this is the first time the page corresponding to that logical identifier has been encountered in the current reverse timeline. Since the reverse scan proceeds from the end of the file (the latest write position) to the beginning of the file (the historical write position), the first page scanned must be the latest written version corresponding to that logical identifier. The device 30 will immediately establish a mapping relationship between the logical identifier and the currently scanned physical offset in the memory-mapped table. Conversely, if the memory-mapped table already contains a mapping record for that logical identifier, it means that the device 30 has already captured the updated version corresponding to that logical page in the previous reverse scan, and the currently scanned page is an expired historical version. In this case, the device 30 will directly skip the mapping relationship establishment operation for the currently scanned page and continue to perform the reverse scan forward.

[0169] For example, if the same logical page has the latest version 3 at the end of the file, a historical version 2 in the middle, and the earliest version 1 at the beginning, a reverse scan will first encounter version 3, establishing a corresponding mapping relationship in the memory mapping table. Continuing the forward scan, when encountering versions 2 and 1, if a record for that logical identifier already exists in the memory mapping table, it will be ignored without additional comparison or storage operations. Through this reverse scan reconstruction mechanism, device 30 eliminates the overhead of additional logging for memory mapping table changes, eliminating the need to maintain separate logs for changes. Furthermore, it only requires a single reverse scan of the physical data file, without needing to store all version mapping relationships, to restore the latest full mapping relationship of the memory mapping table. This shortens fault recovery time and improves instance fault recovery efficiency and business availability.

[0170] Compared to traditional database storage engine log and fault recovery solutions, the hybrid log checkpointing and fault recovery mechanism of this application embodiment has significant advantages in security, performance, and recovery efficiency. Traditional database storage engines mostly employ a single log write-ahead scheme, uniformly recording and flushing logs of business data changes and index structure changes to disk in real time. While this scheme ensures data security, it generates significant disk I / O overhead, severely limiting database write performance. Some solutions employ asynchronous batch flushing of logs to improve performance, but this faces the risk of log loss and incomplete data recovery during system crashes, failing to achieve a balance between performance and data security. Furthermore, traditional checkpointing mechanisms mostly use fixed time intervals for triggering, unable to dynamically adjust based on the system's dirty page state. Fault recovery still requires replaying a large amount of logs, resulting in extremely long recovery times. During log replay, the lack of an effective data existence detection mechanism makes it prone to data duplication and primary key conflicts, leading to a lack of consistency in the recovered data. Furthermore, for addressing architectures with logical-physical separation, traditional solutions mostly require maintaining change logs for mapping tables separately, which introduces log recording overhead on regular write paths and additionally handles checkpoints for mapping tables, making them complex and inefficient.

[0171] This application's embodiment employs a dual-log mechanism that separates business data from the index structure. This mechanism safeguards the bottom line of zero user data loss through real-time flushing of the first log, while simultaneously maximizing system write performance through batch asynchronous flushing of the second log, achieving an optimal balance between performance and security. A dynamically advancing checkpoint mechanism continuously updates system replay points, significantly reducing the amount of logs required for replay during fault recovery and effectively shortening recovery time. An orderly recovery process, prioritizing physical structure over business data, coupled with a key-value existence detection mechanism, ensures index structure consistency and perfectly avoids data redundancy and conflicts caused by repeated log replay. Furthermore, leveraging the physical characteristics of append-only files, a single-pass scan is sufficient to rapidly rebuild the memory-mapped table, eliminating additional log recording overhead and significantly improving the table reconstruction efficiency. This further shortens the overall fault recovery time and ensures high availability of the database instance.

[0172] Next, refer to Figure 11 Some optional details of embodiments of this application are described below. Figure 11 A multi-level query data flow diagram according to an embodiment of this application is shown.

[0173] like Figure 11 As shown, in the cold and hot tiered hybrid storage architecture of this application embodiment, data may be distributed across multiple components. This architecture can improve read and write performance, but it also brings higher complexity to the query process. To address this, device 30 further designs a query processing mechanism adapted to the characteristics of tiered storage, so as to compress the query chain time while ensuring the global consistency and accuracy of query results, and to balance low-latency response for hot data and efficient retrieval of massive amounts of cold data.

[0174] As an example, in the hybrid storage architecture of this application, data will continuously flow from the upper hot storage to the lower cold storage as the life cycle progresses. Data corresponding to the same primary key may have copies of different time versions in different storage components and different levels. Moreover, the data version in the upper storage component is always newer than that in the lower storage component. Therefore, the device 30 has designed differentiated query execution processes for different types of query requests.

[0175] For point query requests that precisely match a single primary key, device 30 executes a top-down retrieval scheme based on data time frequency and an immediate short-circuit return scheme. Specifically, a point query refers to a precise search operation that retrieves a single target data record corresponding to a unique primary key identifier. It is the most common query type in online transaction processing scenarios and has extremely high requirements for response latency.

[0176] In response to receiving a point query request 1101 initiated by the user, the device 30 searches each storage level sequentially from top to bottom according to the order of data popularity from high to low and data version from newest to oldest. It starts the search from the memory level, which represents the latest and hottest data status, and performs target key value matching search in the memory buffer and read-only buffer in turn. The memory buffer is used to hold the latest write transaction data, and the read-only buffer is used to store static data to be persisted to the first storage component. Both are stored in memory, and the search time is extremely low, which is the optimal path for querying hot data.

[0177] If the target key is not found in either of the two buffers at the memory level, device 30 then performs a fast key-value lookup in the first storage component using a B+ tree global index. The first storage component, as a persistent storage layer for hot data, stores frequently accessed valid data and achieves extremely fast conversion between logical and physical addresses through a memory mapping table, resulting in retrieval efficiency far exceeding that of the underlying cold data layer. If the target key is not found in the first storage component and the system background is performing a merge and sinking operation, and a frozen first storage component exists, device 30 will continue to perform the target key-value search in that frozen first storage component to avoid data query gaps during the merging process and ensure the integrity of the query results during the merging process.

[0178] If the target key is not found in any of the upper-level hot data storage structures, indicating that the target data belongs to historical cold data that has been sunk to the lower levels, the device 30 will then sequentially delve into each persistent data layer of the second storage component, performing the search for the target key layer level by level in ascending order of data age, until the target data is found or the full-level search is completed. Throughout this top-down search chain, the device 30 implements a short-circuit return principle: as soon as valid data corresponding to the target key is found in any storage component, the query result is immediately returned to the user, while all subsequent level search operations are interrupted, eliminating the need to continue searching lower-level storage structures.

[0179] In the storage architecture of this application, the data version in the upper-layer storage component is always newer than the data version in the lower-layer storage component. The data retrieved first is necessarily the latest effective business data corresponding to the primary key, which can fully meet the user's query needs, without needing to continue searching for historical versions in the lower layer. Through this mechanism, device 30 maximizes the use of the principles of temporal and spatial locality of data, allowing frequently accessed hot data queries to be quickly hit and returned in the uppermost memory or hot data storage layer without accessing the lower-level cold data layer, providing a low-latency response for frequent hot data queries.

[0180] For range query requests that require scanning continuous key-value intervals, one aspect of this application provides an alternative query mechanism. Specifically, because data within the same key-value interval is very likely to be simultaneously stored in various persistent data layers of the memory buffer, read-only buffer, first storage component, frozen first storage component, and second storage component, the interval data within each storage component is locally ordered, but the global data across components is not naturally ordered, and different versions of the same primary key may be scattered across different components, global ordered merging and version deduplication are required to output query results that meet the requirements.

[0181] Therefore, in response to receiving the range query request 1102 initiated by the user, the device 30 first parses the query start key and end key in the request to determine the target key value scanning range. Then, in each of the above-mentioned storage components containing valid data, including the memory buffer, read-only buffer, new first storage component, frozen first storage component, and second storage component, the device calls the range positioning interface at the bottom layer of each component to initialize an independent range scanning iterator for each component.

[0182] An iterator is a software structure with a built-in cursor state that can continuously output ordered data records within its component and the target key value range in ascending order of key value. It also supports core operations such as cursor stepping forward and obtaining the key value and data content currently pointed to by the cursor. Each iterator is only responsible for scanning the range data of its component and is completely independent of each other.

[0183] After initializing all iterators, device 30 connects all initialized iterators to a min-heap structure deployed in memory to perform multi-way merge sort. The core characteristic of the min-heap structure is its ability to maintain the top element of the heap as the globally smallest key value among all iterators' current output data at all times.

[0184] Device 30 continuously pops the data record corresponding to the minimum key value at the top of the min-heap and outputs it to the query result set. At the same time, it moves the iterator corresponding to the popped data forward one position, reads the next ordered data, and re-inserts it into the min-heap to complete the rebalancing of the heap structure. This process is repeated in a loop, interweaving and merging multiple locally ordered data streams from different components into a globally strictly ordered output data stream in real time, until all interval data of all iterators have been scanned.

[0185] Throughout the multi-way merge process, since old and new versions of the same primary key may exist simultaneously in different iterator output streams, device 30 synchronously performs a strict version deduplication filtering step. When it detects that the output data from different iterators contains records with the same primary key, device 30 compares the version information of multiple records, including transaction sequence number and timestamp, and only retains the latest version of the data to output to the query result set, automatically discarding all outdated historical version data. This ensures that in the final output query result, each primary key contains only one latest valid data, in order to meet the user's consistent snapshot reading requirements.

[0186] Through this multi-iterator collaborative multi-way merge mechanism, device 30 does not need to load the full range of data into memory to perform full sorting. It only needs to maintain a min-heap structure that matches the number of iterators. With extremely low memory usage, it can complete the global ordered merging of massive range data across components, while strictly ensuring that users read the latest consistent snapshot data.

[0187] Furthermore, when any of the above-mentioned query operations locate the target leaf node within the first storage component using a B+ tree index and require reading the specific data content within the node, device 30 must execute a multi-version retrieval process that spans the boundaries between logical addressing and physical storage, adapting to the append-write architecture and the chained multi-version storage. Because the first storage component employs an append-write architecture that decouples logical identifiers from physical offsets, and a chained separation storage mechanism that stores the latest version in leaf nodes and historical versions in external version linked lists, the physical storage location of the target data is not statically fixed. A conversion from logical address to physical address and a collaborative retrieval of the main page data and the external version linked list data are required to accurately obtain the data content that meets the query requirements.

[0188] Specifically, in response to a query request involving data in a target leaf node, device 30 first obtains the globally unique logical identifier corresponding to the target leaf node from the routing information of the intermediate index node of the B+ tree, and at the same time reads the head pointer of the target primary key in the target leaf node, which points to the external version linked list, from the metadata of the target leaf node.

[0189] Next, the device 30 quickly queries the memory mapping table in memory based on the logical identifier of the target leaf node, obtains the latest physical offset corresponding to the logical identifier, and completes the key conversion from the B+ tree logical route to the file physical address that the operating system can recognize, ensuring that the latest version page data of the leaf node can be read.

[0190] Subsequently, based on the acquired physical offset, device 30 reads the complete page content of the target leaf node from the physical data file of the first storage component to obtain the latest version data corresponding to the primary key.

[0191] If the user initiates a historical snapshot query for a specific point in the past, rather than a query for the latest status, after reading the latest version data from the main page, device 30 will first compare the timestamp of the latest version with the target time point of the snapshot query. If the latest version does not meet the snapshot time requirement, it will then, based on the previously obtained version list head pointer, search for the data blocks of each historical version sequentially along the version list outside the main data page, comparing the timestamp and transaction sequence number of each historical version one by one, until a specific version data that meets the snapshot query conditions is found, or the entire version list is traversed.

[0192] This seamlessly connects the logical-physical addressing of the master data page with the linked list traversal process of external historical data, adapting to the addressing characteristics of the append-write architecture while fully utilizing the structural advantages of the chaining method for separate storage.

[0193] Compared to traditional solutions that cram all historical version data into the main data page, resulting in a bulky page size and low effective data density, the solution in device 30 ensures that the main data page always stores only the latest version data, maintaining a compact structure and extremely high effective data density. Regular queries for the latest status do not require traversing historical versions, enabling faster response times. For infrequent historical snapshot queries, the faster addressing of the memory-mapped table and the targeted traversal of the external version list enable accurate tracing of historical versions without impacting the performance of regular queries.

[0194] Through this retrieval mechanism, device 30 can support complex multi-version concurrency control and historical state backtracking capabilities without affecting the performance of reading and writing current hot data, thus balancing performance and functional integrity.

[0195] Compared with the query processing schemes of traditional database storage engines, the multi-level query processing mechanism of this application embodiment has significant performance, efficiency and functional advantages.

[0196] Traditional single-tier storage architecture query solutions cannot adapt to the hybrid storage characteristics of hot and cold tiers. For point query requests, it is often necessary to penetrate all storage tiers to complete the retrieval, failing to achieve short-circuit return based on data popularity. This results in significantly increased query latency for hot data, failing to meet the low-latency requirements of high-concurrency online transaction processing. For range query requests, traditional solutions involve multiple data tiers, requiring the maintenance of an iterator for each data tier and using the iterator as input to a min-heap. This leads to an excessive number of data sources for the min-heap input, a complex sorting process, and significant resource consumption. For historical snapshot queries with multi-version concurrency control, traditional solutions mostly store all historical versions and the latest version mixed in the main data page. This results in extremely low effective data density on the main page, causing severe read amplification during range scans and a significant drop in regular query performance. Separating the storage of historical versions, however, presents problems such as complex addressing paths and low historical version retrieval efficiency.

[0197] The solution in this application, for point query scenarios, maximizes the use of the locality of data principle through a top-down retrieval and return mechanism, allowing hot data queries to be quickly returned at the top level, thus drastically reducing the response latency of point queries. For range query scenarios, the first storage component replaces multiple layers of the LSM structure, greatly simplifying the sorting complexity. Through multi-component independent iterators and a min-heap multi-way merge mechanism, it achieves globally ordered merging of massive amounts of data across components with extremely low memory usage, while simultaneously completing version deduplication to ensure the global orderliness and consistency of query results. For multi-version data retrieval scenarios, through the logical-physical addressing conversion adapted to the append-write architecture and the version linked list traversal mechanism of separate storage using the chaining method, it not only ensures the ultimate performance of regular latest data queries but also supports the ability to query historical snapshots. This solves the contradiction between multi-version storage and query performance in traditional solutions, achieving a comprehensive technical effect of low latency for hot data, high retrieval efficiency for cold data, and accurate backtracking of multi-version data.

[0198] According to another aspect of this application, an electronic device is also provided for implementing the methods according to embodiments of this application. Figure 12 A schematic diagram of an electronic device 2000 according to an embodiment of this application is shown. The electronic device may be... Figure 1 The user terminal 120 or server 110 shown Figure 3 Data storage device 30 in the middle.

[0199] like Figure 12 As shown, the electronic device 2000 may include one or more processors 2010 and one or more memories 2020. The memories 2020 store computer-executable instructions, which, when run by the one or more processors 2010, can execute the methods described above.

[0200] The processor in this application embodiment can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, operations, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 architecture or an ARM architecture.

[0201] Generally, the various exemplary embodiments of this application can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of the embodiments of this application are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0202] For example, the method or apparatus according to embodiments of this application can also be used by means of Figure 13 The architecture of the computing device 3000 shown is used for implementation. For example, Figure 12 The electronic device 2000 in the device can adopt the architecture of the computing device 3000.

[0203] like Figure 13 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the methods provided in this application, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 10 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 13 One or more components in the computing device shown.

[0204] According to another aspect of this application, a computer-readable storage medium is also provided. Figure 14 A schematic diagram of a computer-readable storage medium 4020 according to this application is shown.

[0205] like Figure 14 As shown, the computer-readable storage medium 4020 stores computer-executable instructions 4010. When the computer-executable instructions 4010 are executed by a processor, the methods according to embodiments of this application described with reference to the above figures can be performed. The computer-readable storage medium in the embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0206] This application also provides a computer program product or computer program that includes computer-executable instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the computer device to perform a method according to an embodiment of this application.

[0207] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer-executable instructions.

[0208] Generally, the various exemplary embodiments of this application can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of the embodiments of this application are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0209] The exemplary embodiments of this application described in detail above are merely illustrative and not restrictive. Those skilled in the art should understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of this application, and such modifications should fall within the scope of this application. For example, the following provides an overview of some aspects of this application, which can be combined with any other aspects.

[0210] Aspect 1: A data storage method is provided, comprising: receiving data to be written and storing the data to be written into a memory buffer; in response to the data volume in the memory buffer reaching a preset threshold, converting the memory buffer into a read-only buffer; persistently writing the data in the read-only buffer into a first storage component, the first storage component being a paged index structure, the paged index structure including a root node, at least one layer of intermediate index nodes, and leaf nodes; and in response to the first storage component meeting migration conditions, extracting a target data block from the leaf nodes, merging and sorting the target data block with data in the starting data layer of a second storage component, and writing the sorted data into the starting data layer; wherein the second storage component is a sequential file storage structure, the sequential file storage structure including multiple persistent data layers arranged in hierarchical order, the multiple persistent data layers including the starting data layer, and the number of levels of the multiple persistent data layers being associated with configuration parameters.

[0211] Aspect 2: The first storage component corresponds to multiple consecutive high-level persistent data layers in the sequential file storage structure. The hierarchy of the multiple consecutive high-level persistent data layers is associated with the configuration parameters, and the starting data layer is the first persistent data layer after the multiple consecutive high-level data layers.

[0212] Aspect 3: The persistent writing of data in the read-only buffer to the first storage component includes: iterating through each piece of data in the read-only buffer; determining whether the currently iterated data exists on the page where the current cursor is located; in response to determining that the currently iterated data exists on the page where the current cursor is located, skipping the currently iterated data and continuing to iterate through the next piece of data; in response to determining that the currently iterated data does not exist on the page where the current cursor is located, determining whether the key value of the currently iterated data is less than the maximum key value of the page where the current cursor is located; in response to determining that the key value of the currently iterated data is less than the maximum key value of the page where the current cursor is located, inserting the currently iterated data into the page where the current cursor is located; in response to determining that the key value of the currently iterated data is not less than the maximum key value of the page where the current cursor is located, searching in the first storage component to locate the insertion position, inserting the currently iterated data into the page where the insertion position is located, and updating the cursor to a new page containing the insertion position.

[0213] Aspect 4: Determining whether the data of the current iteration exists on the page where the current cursor is located includes: comparing the key value of the data of the current iteration with the key value of the data already existing on the page where the current cursor is located; in response to the key value of the data of the current iteration matching the key value of the data already existing, determining that the data of the current iteration exists on the page where the current cursor is located; in response to the key value of the data of the current iteration not matching the key value of any of the data already existing, determining that the data of the current iteration does not exist on the page where the current cursor is located.

[0214] Aspect 5: Determining whether the key value of the currently iterated data is less than the maximum key value of the page where the current cursor is located includes: reading the maximum key value field of the page where the current cursor is located and comparing it with the key value of the currently iterated data; in response to the key value of the currently iterated data being numerically less than the maximum key value, determining that the key value of the currently iterated data is less than the maximum key value of the page where the current cursor is located; in response to the key value of the currently iterated data being greater than or equal to the maximum key value, determining that the key value of the currently iterated data is not less than the maximum key value of the page where the current cursor is located.

[0215] Aspect Six: The data of the current iteration includes multiple versions of data with the same key value. Inserting the data of the current iteration into the page where the current cursor is located or the page where the insertion position is located includes: inserting the latest version of the data with the same key value into the target leaf node of the page where the current cursor is located or the page where the insertion position is located; storing the historical version data with the same key value in a version linked list outside the page of the page index structure; and recording a pointer to the head node of the version linked list in the target leaf node.

[0216] Aspect 7: The method of persistently writing data in the read-only buffer to the first storage component further includes: when a new page of the first storage component is generated or a page in the first storage component is updated during the persistent writing process, writing the new page or updated page to the end of the data file corresponding to the first storage component in an append-only manner; obtaining the logical identifier of the appended page and its physical offset in the data file; and recording or updating the mapping relationship between the logical identifier and the physical offset in a memory mapping table, wherein the pointer to the child node recorded in the intermediate index node of the first storage component is the logical identifier of the child node.

[0217] Aspect 8: In response to the first storage component meeting the migration conditions, extracting the target data block from the leaf node, merging and sorting the target data block with the data in the starting data layer of the second storage component, and writing the sorted data into the starting data layer, includes: determining the physical space occupied by the physical offset of the data file that is not currently recorded in the memory mapping table due to page append writes as invalid space; determining the proportion of the invalid space in the total capacity of the data file; and in response to the proportion reaching a preset proportion threshold, determining that the first storage component meets the migration conditions, extracting the target data block from the leaf node, merging and sorting the target data block with the data in the starting data layer of the second storage component, and writing the sorted data into the starting data layer.

[0218] Aspect 9: The step of extracting a target data block from the leaf node in response to the first storage component meeting the migration conditions, merging and sorting the target data block with the data in the starting data layer of the second storage component, and writing the sorted data into the starting data layer includes: marking the current first storage component as read-only to freeze it, and creating a first storage component for transit to receive data subsequently persisted from the read-only buffer; obtaining the total number of leaf nodes in the frozen first storage component, dividing the leaf nodes into multiple groups; and iterating through each group in a predetermined order, and writing the currently iterated group into the starting data layer.

[0219] Aspect 10: Writing the current iteration group into the starting data layer includes: extracting data blocks in the starting data layer that overlap with the key value range of the current iteration group; merging and sorting the current iteration group with the extracted data blocks, and writing the sorted data blocks into the starting data layer.

[0220] Aspect 11: Writing the sorted data blocks into the starting data layer includes: in response to the existence of multiple versions of data with the same key value in the merge sort, arranging the multiple versions of data adjacently and continuously in the starting data layer according to the version order.

[0221] Aspect Twelve: The provided method further includes: when receiving data to be written, persisting a first log recording changes in the memory buffer to disk; during the process of persisting data in the read-only buffer to the first storage component, caching a second log recording page changes in the first storage component in memory, and flushing it to disk when the amount of data in the cached second log reaches a preset threshold; in response to receiving a trigger signal for performing a storage state check, calculating the minimum log sequence number of the currently modified dirty page in the first storage component; recording the minimum log sequence number as the replay point of the second log to disk, and releasing the log space in the second log earlier than the minimum log sequence number; in response to determining that the writing of data from the read-only buffer to the first storage component is complete, forcibly flushing the second log cached in memory to disk; and after the second log is flushed, recording the safe starting point corresponding to the read-only buffer as the first log replay point to disk, and releasing the log space in the first log earlier than the safe starting point.

[0222] Aspect Thirteen: The provided method further includes: in response to receiving a fault recovery trigger signal, reading the second log in the disk, replaying from the replay point of the second log to recover data in the first storage component; after completing the recovery of the first storage component, reading the first log and replaying from the system replay point, and detecting whether a persistent version of the current replay record already exists in the recovered first storage component during the replay process; in response to the existence of a persistent version of the current replay record in the first storage component, skipping the writing of the data record when recovering the memory buffer.

[0223] Aspect Fourteen: The provided method further includes: in response to receiving a fault recovery trigger signal, restoring the memory mapping table, wherein restoring the memory mapping table includes: performing a reverse scan from the end of the data file of the first storage component to the beginning, reading the logical identifier and corresponding physical offset of each scanned page; in response to the absence of a record for the read logical identifier in the memory mapping table, establishing a mapping relationship between the logical identifier and the physical offset in the memory mapping table; and in response to the existence of a record for the read logical identifier in the memory mapping table, determining that the recorded physical offset is the latest version, and skipping the establishment of the mapping relationship for the currently scanned page of the logical identifier.

[0224] Method 15: The provided method further includes: in response to receiving a point query request, sequentially searching for a target key value in the memory buffer, the read-only buffer, and the first storage component for relay; in response to not finding the target key value in the first storage component for relay, and the existence of the currently frozen first storage component, searching for the target key value in the frozen first storage component; and in response to not finding the target key value in the frozen first storage component, sequentially searching in each persistent data layer of the second storage component, and immediately returning the query result when the target data is found in any component.

[0225] Method 16: The provided method further includes: in response to receiving a range query request, initializing respective iterators in the memory buffer, the read-only buffer, the first storage component for transfer, the frozen first storage component, and the persistent data layer of the second storage component; connecting all the initialized iterators into a min-heap structure to perform multi-way merge sort; and in response to data from different iterators containing the same key value, retaining only the latest version of the data in the query results based on version information.

[0226] Method 17: The provided method further includes: in response to a query request involving data in the target leaf node, reading a pointer in the target leaf node pointing to the version list; querying the memory mapping table to locate the physical offset of the page in the data file based on the logical identifier of the page containing the target leaf node; and reading the page content based on the physical offset, and sequentially retrieving data of each version along the version list outside the page based on the pointer, until a version data that meets the query conditions is found.

[0227] Method 18: A data storage device is provided, comprising: a memory buffer configured to receive data to be written; a read-only buffer conversion unit configured to convert the memory buffer into a read-only buffer in response to the amount of data in the memory buffer reaching a preset threshold; a first storage component, the first storage component being a paged index structure, the paged index structure including a root node, at least one layer of intermediate index nodes, and leaf nodes; a second storage component, the second storage component being a sequential file storage structure, the sequential file storage structure including multiple persistent data layers arranged in hierarchical order, the multiple persistent data layers including a start data layer, and the number of levels of the multiple persistent data layers being associated with configuration parameters; a persistent writing unit configured to persistently write data in the read-only buffer to the first storage component; and a merging unit configured to merge and sort a target data block extracted from the leaf node with data in the start data layer of the second storage component in response to the first storage component meeting migration conditions, and write the merged and sorted data to the start data layer.

[0228] Aspect 19: An electronic device comprising: one or more processors; and one or more memories, wherein the memories store computer-executable instructions that, when executed by the processor, perform the method described in any aspect.

[0229] Aspect 20: A computer-readable storage medium having stored thereon computer-executable instructions which, when executed by a processor, implement the method described in any aspect.

[0230] Aspect 21: A computer program product comprising computer-executable instructions which, when executed by a processor, are used to implement the method described in any aspect.

Claims

1. A data storage method, characterized in that, The method includes: Receive the data to be written and store the data to be written into a memory buffer; In response to the data volume in the memory buffer reaching a preset threshold, the memory buffer is converted into a read-only buffer; The data in the read-only buffer is persistently written to a first storage component, the first storage component being a paged index structure, the paged index structure including a root node, at least one level of intermediate index nodes, and leaf nodes; and In response to the first storage component meeting the migration conditions, the target data block is extracted from the leaf node, the target data block is merged and sorted with the data in the starting data layer of the second storage component, and the sorted data is written into the starting data layer. The second storage component is a sequential file storage structure, which includes multiple persistent data layers arranged in hierarchical order. The multiple persistent data layers include the starting data layer, and the number of levels of the multiple persistent data layers is associated with configuration parameters. Persistently writing the data in the read-only buffer to the first storage component includes: When a new page is generated in the first storage component or a page in the first storage component is updated during the persistent writing process, the new page or the updated page is written to the end of the data file corresponding to the first storage component in an append-only manner. Obtain the logical identifier of the page to be appended and its physical offset in the data file; and The mapping relationship between the logical identifier and the physical offset is recorded or updated in the memory mapping table, wherein the pointer to the child node recorded in the intermediate index node of the first storage component is the logical identifier of the child node.

2. The data storage method according to claim 1, characterized in that, The first storage component corresponds to a plurality of consecutive high-level persistent data layers in the sequential file storage structure. The hierarchy of the plurality of consecutive high-level persistent data layers is associated with the configuration parameters. The starting data layer is the first persistent data layer after the plurality of consecutive high-level persistent data layers.

3. The method according to claim 1, characterized in that, The step of persistently writing the data in the read-only buffer to the first storage component further includes: Iterate through each piece of data in the read-only buffer; Determine if the data for the current iteration exists on the page where the current cursor is located; In response to determining that the data for the current iteration exists on the page where the current cursor is located, skip the data for the current iteration and continue iterating to the next data item; In response to determining that the data of the current iteration does not exist on the page where the current cursor is located, determine whether the key value of the data of the current iteration is less than the maximum key value of the page where the current cursor is located; In response to determining that the key value of the data in the current iteration is less than the maximum key value of the page where the current cursor is located, the data in the current iteration is inserted into the page where the current cursor is located; In response to determining that the key value of the data in the current iteration is not less than the maximum key value of the page where the current cursor is located, the system searches in the first storage component to locate the insertion position, inserts the data of the current iteration into the page where the insertion position is located, and updates the cursor to the new page containing the insertion position.

4. The method according to claim 3, characterized in that, Determining whether the data for the current iteration exists on the page where the current cursor is located includes: Compare the key value of the data in the current iteration with the key value of the data already existing in the page where the current cursor is located; In response to the key value of the data in the current iteration matching the key value of existing data, it is determined that the data in the current iteration exists on the page where the current cursor is located; If the key value of the data in the current iteration does not match the key value of any existing data, it is determined that the data in the current iteration does not exist on the page where the current cursor is located.

5. The method according to claim 3, characterized in that, Determining whether the key value of the data in the current iteration is less than the maximum key value of the page where the current cursor is located includes: Read the maximum key value field of the page where the current cursor is located, and compare it with the key value of the data in the current iteration; In response to the fact that the key value of the data in the current iteration is numerically less than the maximum key value, it is determined that the key value of the data in the current iteration is less than the maximum key value of the page where the current cursor is located; In response to the fact that the key value of the data in the current iteration is greater than or equal to the maximum key value, it is determined that the key value of the data in the current iteration is not less than the maximum key value of the page where the current cursor is located.

6. The method according to claim 3, characterized in that, The data of the current iteration includes multiple versions of data with the same key value, and inserting the data of the current iteration into the page where the current cursor is located or the page where the insertion position is located includes: Insert the latest version of the same key value into the target leaf node of the page where the current cursor is located or the page where the insertion position is located. The historical version data of the same key value is stored in a version list outside the pages of the paged index structure; and The target leaf node records a pointer to the head node of the version linked list.

7. The method according to claim 1, characterized in that, In response to the first storage component meeting the migration conditions, the process of extracting the target data block from the leaf node, merging and sorting the target data block with the data in the starting data layer of the second storage component, and writing the sorted data into the starting data layer includes: The physical space occupied by physical offsets that are not currently recorded in the memory mapping table due to page append writes in the data file is determined as invalid space; Determine the percentage of the invalid space in the total capacity of the data file; and In response to the percentage reaching a preset threshold, the first storage component is determined to meet the migration conditions, and a target data block is extracted from the leaf node. The target data block is then merged and sorted with the data in the starting data layer of the second storage component, and the sorted data is written into the starting data layer.

8. The method according to claim 1, characterized in that, The step of extracting the target data block from the leaf node in response to the first storage component meeting the migration conditions, merging and sorting the target data block with the data in the starting data layer of the second storage component, and writing the sorted data into the starting data layer includes: The current first storage component is marked as read-only to be frozen, and a first storage component for relay is created to receive subsequent data written persistently from the read-only buffer; Obtain the total number of leaf nodes in the first storage component that is in a frozen state, and divide the leaf nodes into multiple groups; and Iterate through each group in a predetermined order and write the group of the current iteration into the starting data layer.

9. The method according to claim 8, characterized in that, The step of writing the group of the current iteration into the starting data layer includes: Extract data blocks from the initial data layer that overlap with the key value range of the group in the current iteration; and The group of the current iteration is merged and sorted with the extracted data block, and the sorted data block is written into the starting data layer.

10. The method according to claim 9, characterized in that, The step of writing the sorted data blocks into the starting data layer includes: In response to the existence of multiple versions of data with the same key value in the merge sort, the multiple versions of data are written in a sequential order, adjacent and consecutively, in the starting data layer.

11. The method according to claim 1, characterized in that, Also includes: When receiving data to be written, the first log recording the changes to the data in the memory buffer is persisted to disk; During the process of persistently writing the data in the read-only buffer to the first storage component, the second log recording the page changes of the first storage component is cached in memory, and when the amount of data in the cached second log reaches a preset threshold, it is flushed to the disk; In response to receiving a trigger signal to perform a storage status check, calculate the minimum log sequence number of the currently modified dirty page in the first storage component; Record the minimum log sequence number as the replay point of the second log to the disk, and release the log space in the second log that is earlier than the minimum log sequence number; In response to determining that writing data from the read-only buffer to the first storage component is complete, the second log cached in memory is forcibly flushed to disk; as well as After the second log is refreshed, the safe starting point corresponding to the read-only buffer is recorded as the system replay point to the disk, and the log space in the first log that is earlier than the safe starting point is released.

12. The method according to claim 11, characterized in that, The method further includes: In response to receiving a fault recovery trigger signal, the second log in the disk is read and replayed from the replay point of the second log to recover the data in the first storage component; After the recovery of the first storage component is completed, the first log is read and replayed from the system replay point, and during the replay process, it is detected whether a persistent version of the data record being replayed already exists in the recovered first storage component; In response to the existence of a persistent version of the data record being played back in the first storage component, writing to that data record is skipped when restoring the memory buffer.

13. The method according to claim 1, characterized in that, The method further includes: in response to receiving a fault recovery trigger signal, restoring the memory mapping table, wherein restoring the memory mapping table includes: Perform a reverse scan from the end of the data file of the first storage component to the beginning, and read the logical identifier and corresponding physical offset of each scanned page one by one; In response to the absence of a record for the read logical identifier in the memory mapping table, a mapping relationship between the logical identifier and the physical offset is established in the memory mapping table; and In response to the existence of a record for the read logical identifier in the memory mapping table, the recorded physical offset is determined to be the latest version, and the mapping relationship for the current scanned page of the logical identifier is skipped.

14. The method according to claim 8, characterized in that, The method further includes: In response to receiving a point query request, the target key value is searched sequentially in the memory buffer, the read-only buffer, and the first storage component used for transfer. In response to the fact that the target key value is not found in the first storage component used for transit, and that a frozen first storage component currently exists, the target key value is searched for in the frozen first storage component; and In response to the failure to find the target key value in the frozen first storage component, the search is sequentially performed in each persistent data layer of the second storage component, and the query result is returned immediately when the target data is found in any component.

15. The method according to claim 8, characterized in that, The method further includes: In response to receiving a range query request, each iterator is initialized in the memory buffer, the read-only buffer, the persistent data layer of the first storage component for transfer, the frozen first storage component, and the second storage component, respectively. Connect all initialized iterators to a min-heap structure and perform multi-way merge sort; and In response to data with the same key value in the output from different iterators, only the latest version of the data is retained in the query results based on the version information.

16. The method according to claim 6, characterized in that, The method further includes: In response to a query request involving data in the target leaf node, read the pointer to the version list in the target leaf node; Based on the logical identifier of the page containing the target leaf node, query the memory mapping table to locate the physical offset of that page in the data file; and The page content is read based on the physical offset, and the data of each version is retrieved sequentially along the version list outside the page based on the pointer until the version data that meets the query conditions is found.

17. A data storage device, characterized in that, The device includes: A memory buffer is configured to receive data to be written. A read-only buffer conversion unit is configured to convert the memory buffer into a read-only buffer in response to the amount of data in the memory buffer reaching a preset threshold. The first storage component is a paged index structure, which includes a root node, at least one layer of intermediate index nodes, and leaf nodes. The second storage component is a sequential file storage structure, which includes multiple persistent data layers arranged in hierarchical order. The multiple persistent data layers include a starting data layer, and the number of levels of the multiple persistent data layers is associated with configuration parameters. The persistent write unit is configured to persistently write the data in the read-only buffer to the first storage component; The persistent write unit is further configured to, during the persistent write process, when a new page of the first storage component is generated or a page in the first storage component is updated, append the new page or updated page to the end of the data file corresponding to the first storage component; obtain the logical identifier of the appended page and its physical offset in the data file; record or update the mapping relationship between the logical identifier and the physical offset in a memory mapping table, wherein the pointer to the child node recorded in the intermediate index node of the first storage component is the logical identifier of the child node; and The merging unit is configured to, in response to the first storage component meeting the migration conditions, merge and sort the target data block extracted from the leaf node with the data in the starting data layer of the second storage component, and write the merged and sorted data into the starting data layer.

18. An electronic device, characterized in that, The electronic device includes: Memory; and A processor for executing a program in the memory to implement the method as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the method of any one of claims 1 to 16.

Images