Data processing method and system, electronic equipment, storage medium and program product

Abstract

The data processing system provided by the invention comprises a main node, a standby node, a meta-information node and a data storage node. The master node additionally writes the target data to the first active data file based on meta-information of the first active data file in the data storage node read from the first active meta-information file in the meta-information node. By adopting the storage mode of combining the meta-information nodes and the data storage nodes, the problems that the meta-information nodes are not suitable for storing a large amount of data, and the data storage nodes are high in access delay and not suitable for managing a large amount of meta-information can be solved. The log of the additional write operation performed on the target data comprises the write site and the write data information. And after the rewriting is completed, the main node also verifies the meta-information recorded by the first active meta-information file based on the write-in site, and updates part of the meta-information recorded by the first active meta-information file according to the write-in information only when the verification is passed, so that the writing atomicity and the data correctness under the multi-activity scene can be ensured through the write-in site.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data processing method, system, electronic device, storage medium, and program product. Background Technology

[0002] Column-based storage engines store data in a columnar format, meaning all data items within the same column are stored together. Specifically, when retrieving and updating data in column-based storage, an additional secondary index structure is needed to store the primary key and its corresponding data storage location. This ensures fast location of the column-based data. The primary key is a combination of one or more columns in a database table; its value uniquely identifies a row in the table, allowing for quick retrieval and updating of its value. Therefore, the secondary index structure used by column-based storage engines can be simply referred to as the primary key index.

[0003] Currently, in columnar storage engines, primary key indexes face the problem of being unable to store all primary key index information in memory when the data volume is large. Furthermore, the recovery speed of primary key indexes after restarts or migrations is limited by memory, and the storage cost is relatively high. Summary of the Invention

[0004] In view of the problems mentioned in the background art above, this application is made to provide a data processing method, system, electronic device, storage medium and program product that solves or at least partially solves the above problems.

[0005] In a first embodiment, this application provides a method for processing primary key index data, applied to a master node in a data processing system. The data processing system further includes a backup node, a metadata node, and a data storage node. The metadata node stores metadata corresponding to each data file in the data storage node. The metadata includes usable metadata that ensures the atomicity of data writes. The method also includes a backup node switching to become the master node when the master node fails.

[0006] Read the first active metadata file in the metadata node, where the first active metadata file contains the metadata of the first active data file in the data storage node;

[0007] Based on the metadata of the first active data file, the target data to be submitted is appended to the active data file; wherein, the append write operation log for the target data includes: write location and write data information;

[0008] Based on the write location, verify the metadata recorded in the first active metadata file;

[0009] When the verification passes, update part of the metadata recorded in the first active metadata file according to the written data information.

[0010] In a second embodiment, this application provides another data processing method applied to a backup node. The method includes:

[0011] When the compression and merging trigger conditions are met, the compression and merging task to be executed is determined based on the metadata of each data file in the data storage node recorded by the metadata node. The compression and merging task includes at least one subtask that can be executed in parallel. The subtask involves a first data file at the upper layer and N second data files at the adjacent lower layer in the data storage node.

[0012] When executing the subtask, consistent hashing is used to distribute all key-value data in the first data file involved in the subtask to the corresponding N second data files for persistent storage;

[0013] After the distribution is completed, update the metadata of the N second data files recorded in the metadata node;

[0014] Wherein, N is the value of the data distribution parameter of the consistent hash, which is used to indicate that data in one data file of the upper layer is distributed to N data files of the lower layer, and N is an integer greater than or equal to 2.

[0015] In a third embodiment, this application provides a data processing system. The system includes:

[0016] A data storage node includes multiple data files with a hierarchical relationship; the multiple data files include a first active data file and multiple ordered data files;

[0017] Meta-information nodes are used to record the metadata of each data file in the plurality of data files; wherein, the currently active first metadata file in the metadata node is used to record the metadata of the first active data file;

[0018] The master node contains an invisible layer and a complete layer in its memory; the master node is used to implement the data processing method provided in the first embodiment of this application.

[0019] Backup nodes are used to implement the data processing method provided in the second embodiment of this application.

[0020] In a fourth embodiment, this application provides an electronic device. The electronic device includes a memory and a processor, wherein the memory is used to store a program; the processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the various method embodiments provided in this application;

[0021] In a fifth embodiment, this application provides a computer-readable storage medium. This computer-readable storage medium stores a computer program; when executed by a processor, the computer program is capable of implementing the steps in the various method embodiments provided in this application described above.

[0022] In a sixth embodiment, this application provides a computer program product. This computer program product includes a computer program that, when executed by a processor, causes the processor to implement the steps described in the various method embodiments of this application.

[0023] The technical solution provided in this application embodiment includes a data processing system comprising a master node, backup nodes, metadata nodes, and data storage nodes. Each metadata file in the metadata node records metadata information of the corresponding data files in the data storage node. The master node, based on the metadata information read from the first active metadata file in the metadata node, appends the target data to be submitted to the first active data file. The append operation information for the target data includes the write position and the written data information. This data storage method combining metadata nodes and data storage nodes solves the problems that metadata nodes are unsuitable for storing large amounts of data, and that data storage nodes have high access latency and are unsuitable for managing large amounts of metadata. Furthermore, after the master node writes the target data to the first active data file, it verifies the metadata recorded in the first active metadata file based on the write position. Only when the verification passes does it update a portion of the metadata recorded in the first active metadata file according to the write information. Thus, the "write position" ensures the atomicity of the write operation and the data correctness in multi-active scenarios (such as concurrent writes). In addition, this solution also reduces the write load of the primary node by using a backup node, so that the compression and merging operation is performed on the backup node, which reduces the load on the primary node (such as the pressure on the processor (CPU)). Specifically, the implementation process of the backup node performing the compression and merging operation is as follows: when the compression and merging trigger condition is met, the compression and merging task to be executed is determined according to the metadata of each data file in the data storage node recorded by the metadata node. The compression and merging task includes at least one subtask that can be executed in parallel. Each subtask involves a first data file at the upper level and N adjacent second data files at the lower level in the data storage node. When executing the subtask, consistent hashing is used to distribute all key-value data in the first data file involved in the subtask to the corresponding N second data files for persistent storage. After the distribution is completed, the metadata of the N second data files recorded in the metadata node is updated. Here, N is the value of the consistent hashing data distribution parameter, which is used to indicate that the data in the upper-level data file is distributed to the N lower-level data files, and N is an integer greater than or equal to 2. This scheme employs a consistent hashing data partitioning scheme with a fixed data distribution parameter fanout. This strictly limits the number of data files at each level in the data storage node and also strictly limits the scale of merged data. In addition, it facilitates efficient parallel traversal and querying of data in the future. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A schematic diagram illustrating the principle of the Delete-Insert model provided in one embodiment of this application;

[0026] Figure 2 A schematic diagram of the principle of an existing log structure merging tree is provided for one embodiment of the application;

[0027] Figure 3a This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;

[0028] Figure 3b An example diagram of the log structure merge tree corresponding to the metadata node and data storage node provided in this application;

[0029] Figure 3c A schematic diagram of the data structure in a data storage node that allows appending log files, provided in an embodiment of this application;

[0030] Figure 4a A schematic diagram illustrating the principle of a search acceleration module provided in one embodiment of this application;

[0031] Figure 4b A schematic diagram illustrating the principle of a write operation provided in one embodiment of this application;

[0032] Figure 4c A schematic diagram illustrating the principle of a repair operation provided in one embodiment of this application;

[0033] Figure 4d-A and Figure 4d-B A schematic diagram illustrating the principle of data querying provided in one embodiment of this application;

[0034] Figure 4e A schematic diagram illustrating the principle of a clearing operation provided in an embodiment of this application;

[0035] Figure 5a , Figures 5b-1 to 5b-4 as well as Figure 5c A schematic diagram illustrating the principle of compression and merging operations provided in the embodiments of this application;

[0036] Figures 6a to 6c A schematic diagram illustrating the principle of aligning the memory states of the backup node and the master node as provided in an embodiment of this application;

[0037] Figure 7A flowchart illustrating the method for processing primary key index data by the master node as provided in this embodiment of the application;

[0038] Figure 8 A flowchart illustrating the data processing method performed by the backup node provided in this application embodiment;

[0039] Figure 9 and Figure 10 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0040] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] Currently, columnar storage engines use key-value columnar storage, where the key includes the primary key (PK) and its corresponding value. The primary key is a combination of one or more columns in a database table, and its value uniquely identifies the corresponding row. Specifically, when retrieving and updating key-value data in columnar storage, an additional index structure is needed to store the primary key and its corresponding data storage location information to ensure fast location of the columnar storage data. Therefore, this additional auxiliary index structure used by columnar storage engines is often called the primary key index. The primary key index solves the problem of columnar storage engines based on the delete-insert model struggling to handle large-scale data and find the location of data that needs updating. However, it also presents the following problems: with large datasets, it's difficult to store all primary key index information in memory; the lack of persistent primary key index capabilities makes it difficult to guarantee fast recovery of the primary key index after restarts or migrations; furthermore, it's difficult to ensure data integrity, and storage costs are relatively high. The primary key index is the only stateful part in a columnar storage engine, requiring low-cost storage to adapt to the ecosystem of object-oriented programming languages ​​such as Java.

[0042] For example, there is currently a high-speed, full-scenario MPP database whose architecture uses a fully vectorized engine and also employs a primary key index. However, this primary key index is a pure in-memory structure, loaded on demand, and scans columnar data to rebuild the primary key index during loading, lacking persistence capabilities. While the aforementioned primary key index, built on demand and in pure in-memory, has advantages such as simple implementation, high query efficiency due to its on-demand construction and full-memory structure, it also suffers from the following drawbacks: long recovery time, the need to scan all valid data locations within the shard, and the shard size being limited by memory size, requiring shards to be smaller than a certain threshold; furthermore, the requirement that the primary key index fit into memory restricts the definition and use of data shards.

[0043] Therefore, in order to solve the above problems, this application provides a data processing technology solution that can better ensure the correctness of stored data and persist the columnar storage primary key index, thereby significantly improving the restart speed of the block columnar storage construction node, and also allowing the range of data shards to be processed to be unrestricted by memory size.

[0044] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0045] In some processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. These operations may be executed out of order or in parallel. Operation numbers such as 101, 102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the terms "first," "second," etc., used herein are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. The term "or / and" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A or / and B means that A can exist alone, A and B can exist simultaneously, or B can exist alone. The character " / " in this application generally indicates that the preceding and following related objects have an "or" relationship.

[0046] First, the terminology used in the embodiments of this application will be explained. It should be understood that this explanation is for the purpose of making the embodiments of this application clearer and does not necessarily constitute a limitation on the embodiments of this application.

[0047] A columnar storage engine, also known as a column-oriented storage engine, is a database storage system that stores data column-wise. This column-wise storage method contrasts with the traditional row-based storage engine, which stores data row-wise. In row-based storage, all fields of a data row are stored contiguously, while in columnar storage, the data in each column is stored separately. Both columnar and row-based storage engines can be considered key-value (KV) storage engines. A KV storage engine, also a database storage system, stores data in key-value format. In this storage engine, each key corresponds to a unique value, which can be used to quickly retrieve or update its corresponding value. In this application, the key in the key-value data is referred to as the primary key (PK).

[0048] A primary key index (or primary index) is a special type of index in a database used to ensure the uniqueness of each row of data in a table and to provide a fast way to access data. In column-oriented storage, the primary key (PK) is a combination of one or more columns in a database table, whose value uniquely identifies the corresponding row in the table. Therefore, a primary key index is an index built on the primary key column.

[0049] The column-oriented storage solution based on the delete-insert model: This real-time column-oriented storage solution is a database architecture specifically designed for handling real-time data updates. In column-oriented storage engines, data is stored column-wise rather than row-wise. This approach is highly efficient for certain types of queries (especially those requiring only a few columns of data). However, updating data in a column-oriented storage engine can be complex, especially in scenarios requiring real-time processing. In the delete-insert model storage scenario, data update operations do not directly update the original data. Instead, a corresponding operation record (such as delete, modify, or insert operation record) is written to memory. When the number of operation records stored in memory reaches a certain amount, they are then matched and written to disk. Specifically, this delete-insert model is implemented through the following two steps:

[0050] 1) Delete: Deletion is also a write operation. When a deletion operation is performed on data, a record with a deletion identifier is created for that data. This record could be named: Key:delete, where Key is the key to the deleted data. Therefore, Delete is used to mark deleted records. In columnar storage, this typically involves marking the valid bit of the old record as invalid or removing the data directly from the stored column.

[0051] 2) Insert: Inserts updated new data records into the database. In columnstore, this means adding new data values ​​to the end of the relevant column.

[0052] The key advantage of this delete-insert model is that it simplifies the update logic of column storage while maintaining dense column storage, thereby preserving query performance and data compression ratio. Real-time column storage solutions are typically used in data warehousing and real-time analytics, such as analyzing transaction or call records in the financial services or telecommunications industries.

[0053] In the Delete-Insert model described above, the primary key index serves to locate the old data. For a more comprehensive understanding of the Delete-Insert model, please refer to [link to relevant documentation]. Figure 1 As shown. Figure 1 As shown, a delete-insert model scheme introduces a primary key index and a delete bitmap to achieve a high-efficiency, real-time update storage engine. The delete bitmap is used to mark which data has been deleted; for example, in the delete bitmap, 0 indicates that the corresponding data has been deleted and does not exist, while 1 indicates that the corresponding data exists. Figure 1 Optimized Row Columnar (ORC) is a columnar storage format that improves data read and write performance. In this article, files with a columnar storage format are referred to as columnar storage files.

[0054] Log-Structured Merge-tree (LSM-tree): A data structure used for efficiently implementing insert (write), update, and delete operations. Specifically, it is a multi-tiered storage structure combining memory and disk (such as a hard disk), commonly used in key-value data storage systems. LSM-tree optimizes write performance by using a log-structured approach while still providing good read performance. See also... Figure 2The illustrated diagram illustrates the principle of an LSM-tree. The core of an LSM-tree is that data is first written to a data structure in memory layer L0, typically an ordered data structure (such as a skip list or red-black tree). This data structure in memory layer is called a "memTable." Specifically, data is first written to the active memTable in memory. The data in the active memTable is mutable and supports read and write operations. To prevent system crashes and other failures before the data in the active memTable is persisted to a file on the storage disk, data can be written to a write-ahead log (WAL) file on the storage disk simultaneously with the writing to memory for fault recovery. The storage disk is not limited to a hard disk; it can include a solid-state drive (SSD). The WAL needs to be cleared before the data in the active memTable is persisted to a file on the storage disk. The files on the storage disk are Sorted Strings Tables (SSTs). SSTs are persistent data files on the storage disk, typically containing data arranged in key-order. That is, an SST is an immutable and ordered file, belonging to static files. When the amount of data in the active memory table exceeds a set value, it will freeze in memory and automatically transform into an immutable memory table. Simultaneously, a new active memory table will be generated for continued data writing. The aforementioned immutable memory table only supports read operations and not write operations. The data in this immutable memory table is transferred to an SST file on the storage disk for persistent storage through a compaction operation. It can be understood that the storage disk has a persistent storage structure containing multiple persistent layers, such as the first persistent layer L1 to the Zth persistent layer LZ. Each persistent layer contains multiple SST files, and the total number of SST files in a single layer increases exponentially with the number of layers. In specific implementation, the data in the immutable memory table is transferred to several corresponding SST files on the first persistent layer of the storage disk. Figure 2 In this context, the "files" shown in each persistent layer are SST files.

[0055] Write-Ahead Log (WAL) files are a technique used in database management systems to ensure the durability and recoverability of transactional data. It's a logging method that requires all modifications (such as inserts, updates, and deletions) to be written to a special file called the WAL log (or redo log) before the data is actually written to the database file. In the event of a failure (such as a system crash or power outage), the WAL mechanism allows the database to use these logs to redo or undo transactions after restarting, ensuring the atomicity and durability of transactions are maintained, and also protecting the database from inconsistent states due to partially written data.

[0056] Compaction is a critical background maintenance operation in LSM-trees, helping to optimize stored data and improve read / write performance. Because LSM-trees are primarily designed for write-intensive applications, they generate a large number of SST files over time. The compaction operation is responsible for organizing these files to ensure system efficiency.

[0057] Database paging: This generally refers to the process in a database management system (DBMS) that manages and optimizes memory usage. Because databases are typically much larger than available memory, a DBMS needs a mechanism to efficiently move data between memory and disk. Paging is this mechanism, which involves dividing data into fixed-size blocks called pages.

[0058] Object Storage Service (OSS) is a massive, secure, low-cost, and highly reliable cloud storage service suitable for storing files of any type.

[0059] The technical solutions provided by the various embodiments of this application will be described and explained below. In the description and explanation, the above will be combined with... Figure 2 When describing an LSM-tree, the active memtable is represented as an active log file, the immutable memtable as an immutable log file, and the SST file as an ordered file.

[0060] The following method embodiments provided in this application are applied to the data processing system provided in this application. The data processing system is a storage engine, which can be, but is not limited to, a KV storage engine. A KV storage engine is implemented based on an LSM-tree and is mainly used to store key-value data. In this application, the key in the key-value data is referred to as the primary key. In specific implementations, the KV storage engine can be a columnar storage engine or a row-based storage engine. In this application, the data processing system is a columnar storage engine, and this columnar storage engine implements a persistent primary key index.

[0061] Figure 3a A block diagram of the data processing system architecture provided in this application is shown. Figure 3a As shown, the data processing system includes: a master node 11, a backup node 12, a metadata node 13, and a data storage node 14 (e.g., OSS). The data structures of the metadata node 13 and the data storage node 14 are as follows: Figure 2 The LSM-tree design pattern is shown.

[0062] Before detailing the functions of the master node 11, backup node 12, metadata node 13, and data storage node 14, please refer to the figure. Figure 3b The section on the LSM-tree shown for data storage node 14 and metadata node 13 provides a general overview of the data processing system implemented based on the LSM tree concept. Specifically, the implemented features mainly include the following:

[0063] 1) Use an appendable log file to handle WAL functionality to achieve high-performance persistent storage. The appendable log file is the currently active log file in the memory layer (e.g., ...). Figure 3b The first active data file and the first active metadata file shown in the figure)

[0064] 2) A layered structure enables the separation of hot and cold data. For example, the L0 layer in data storage node 14 is a memory layer (also known as the log layer). The data stored in its data structure often overlaps and is unordered, and is written sequentially by appending. Specifically, it is written in batch mode.

[0065] 3) To facilitate compaction operations without affecting write operations, the memory layer includes two types of log files: active and immutable. There is only one active log file at any given time, and new data can only be appended to it. The immutable log file is generated during both fix and compaction operations. The implementation details of fix and compaction operations will be provided below.

[0066] To facilitate differentiation, different representations are used for active log files in data storage node 14 and metadata node 13. Specifically, in data storage node 14, active log files are represented as active data files, while in metadata node 13, they are represented as active metadata files. The active metadata file records the metadata information of the corresponding active data file.

[0067] For example, referring to the LSM-tree shown for data storage node 14 in Figure 5b, there is currently only one active log data file in the memory layer L0 of data storage node 14, such as the second active log data file; in addition, there may be an immutable log data file, which may be formed by changing the previous first active log data file when the compression and merge operation is triggered.

[0068] Furthermore, other layers on the storage disk (such as an SSD) (the first persistent data layers L1 to LZ shown in Figure 5b) are persistent layers used for persistent data storage. Data in files on these layers does not overlap, and the integer primary key (Key) within a file is sorted by value, the variable-length primary key (Key) is sorted by its hash value, and if hash values ​​are the same, they are sorted by their binary values. In other words, the files on each layer of the storage disk are ordered files (which can be denoted as SST).

[0069] To facilitate differentiation, different expressions are used for the ordered file SST in data storage node 14 and metadata node 13. Specifically, in data storage node 14, the ordered file in the storage disk is expressed as an ordered data file (SST data, abbreviated as SSTd), while in metadata node 13, the ordered file is expressed as an ordered metadata file (SST meta, abbreviated as SSTm). The ordered metadata file is used to record the metadata of the corresponding ordered data file.

[0070] In data storage nodes such as OSS, the files within them store densely arranged, ordered variable-length insert / delete items, and the end of the file stores a sparse index (integer as the primary key value, variable-length as the hash value).

[0071] 4) Data storage sharding adopts a consistent hashing (rehash) scheme, so that a file in the upper layer strictly corresponds to N adjacent files in the lower layer, where N is the value of the data distribution parameter fanout.

[0072] Therefore, for multiple layers on the storage disk, the topmost layer, the first persistent data layer L1 as shown in Figure 5b, has a maximum of fanout files, and the last layer, the Zth persistent data layer LZ as shown in Figure 5b, has a maximum of (fanout)^Z files.

[0073] Furthermore, each time a file from the upper layer is merged into a file from the lower layer, it involves reading at most 1+fanout files (including one file from the upper layer and fanout files from the lower layer) and writing fanout files. The number of reads and writes is strictly controllable, and all files within the same layer are merged without any overlap, which is beneficial for achieving parallel tasks.

[0074] The specific functions of the master node 11, backup node 12, metadata node 13, and data storage node 14 (if it is OSS) are described in detail below.

[0075] The aforementioned metadata node 13, also known as the metadata database (MetaDB), records the metadata of each file in the data storage node 14. Metadata can be understood as file descriptions, including but not limited to: file identifier (file_id), file name (file_iname), file length (valid_length), total number of rows (total_rows), latest update time (latest_tso), hash bucket information, verification information (such as CRC32), Bloom filter information, etc. A Bloom filter, similar to a hash table, is a probability-based data structure that indicates whether a piece of data is definitely not in the set or may be in the set.

[0076] See also Figure 3b As shown, the first active metadata file in metadata node 13 is used to record the metadata of the first active data file in data storage node 14. In addition, the ordered metadata files SSTm1-1 to SSTm1-4 in metadata node 13 are used to record the metadata of the ordered data files SSTd1-1 to SSTd1-4 in data storage node 14, and so on.

[0077] Furthermore, the aforementioned data storage node 14 is a remote storage node. Remote storage nodes can be, but are not limited to, Object Storage Service (OSS), Network File System (NFS), or Simple Storage Service (S3). OSS is a massive, secure, low-cost, and highly reliable cloud storage service suitable for storing files of any type. NFS is a shared storage service. By using the NFS protocol, clients can access shared resources on a remote server as if they were local directories. Clients can be, for example, smartphones, desktop computers, tablets, etc. S3 is scalable cloud storage (also known as bucket storage). Specifically, S3 is a network storage service that provides cloud storage services, supporting the storage of various types of data, including digital assets such as images, videos, music, and documents.

[0078] The master node 11 is primarily responsible for writing and retrieving (querying) data for the metadata node 13 and the data storage node 14. Specifically, the master node 11 includes a write module 111 and a read module 112. The write module 111 is responsible for write operations to write the objects to be stored into the corresponding nodes. The objects to be stored include the target data to be submitted and the metadata of the file. In specific implementation, the write module 111 can be used to write the target data to be submitted into a file in the data storage node 14, and after completing the write, update the metadata of the file stored in the data storage node 14 in the metadata node 13. After the metadata is updated, the writing and submission of the target data can be considered complete. The target data can be batch key-value data, specifically, the key-value data can be column-stored key-value data. The primary key in the column-stored key-value data is a combination of one or more columns in the table, and the value of the primary key identifies the corresponding row in the table. Of course, the key-value data can also be in other forms, such as row-stored key-value data, which is not limited here. In this application, the key-value data is preferably column-stored key-value data. The write module 111 performs an append operation. The read module 112 is responsible for reading operations to achieve data retrieval and querying.

[0079] To facilitate understanding of the read and write functions of master node 11, the following will combine... Figure 4a and Figure 4b First, the write operation process performed on master node 11 will be described in detail.

[0080] See also Figure 4bAs shown: Master node 11 contains target data to be committed. This target data may contain multiple key-value pairs stored column-wise, such as Key1:Value1, Key2:Value2, Key3:delete, Key4:Value4, etc. Assuming the target data has 2 rows and a length of 20 bytes, master node 11 will perform the following write operation on this target data:

[0081] Step 11: Read Meta Information. Reading meta information involves performing a read operation on meta information node 13 to read the meta information of the first active data file (active log data) recorded in data storage node 14, which is the current first active meta information file (active log meta) in meta information node 13. For example, the meta information of the first active data file includes a file ID (file_id) of 1 (file_id:1), a file name (file_name) of abc (file_name:abc), a file length (valid length) of 100 (valid_length:100), a total number of rows (total_rows) of 10 (total_rows:10), and a latest update time (latest_tso) of 123 (latest_tso:123). In addition, the meta information of the first active data file may also include other information such as hash bucket information. For more detailed information on what information the meta information may include, please refer to the relevant descriptions in other embodiments. The file identifier `file_id` of 1 is reserved for the active data file by default. There is only one active data file in data storage node 14, and by default, the file identifier `file_id` assigned to the active data file is always 1. Other file identifiers besides 1 can be used by immutable data files (such as immutable log data). Similarly, there is only one active metadata file in metadata node 13. The file name is `abc`, the valid length `valid_length` is 100, and the total `rows` is 10. This means that the first active data file `abc` is expected to already store 10 rows of data with a valid length of 100 bytes (original log data).

[0082] Step 12: Upload Data. After reading the metadata of the first active data file in data storage node 14, a write operation can be performed remotely on data storage node 14 to write the target data to be submitted into the first active data file. Specifically, the first active data file can be found in data storage node 14 based on the file identifier and / or file name contained in the metadata; then, the write position can be determined based on the file length contained in the metadata, and the target data can be written to the first active data file according to that write position. The write method can be append-only.

[0083] For example, continuing with the target data example given above, based on the file length of the first active data file abc, the target data will be appended to the end of the original log data in the first active log data file, starting from position 100 of the first active data file abc. After the append is completed, the file length of the first active data file abc will increase by 20, and its valid length valid_length will change to 120.

[0084] It should be noted that the writing of the target data described above is a batch submission, meaning the target data contains a batch of key-value data, and this key-value data type includes insert items and delete items. Insert items are exemplified by the previously given example Key1:Value1, and delete items are exemplified by the previously given example Key3:delete. Here, "Key" represents the primary key of the key-value data, and "Value" represents the value corresponding to the primary key. When the key-value data in the target data is submitted and written to data storage node 14, it will be compressed and encoded according to the characteristics of the data. For example, the primary key and its corresponding value in the key-value data can support the storage types shown below:

[0085] 1) The primary key in key-value data can support the following two storage types:

[0086] ① Integer primary key: The key represents the primary key. The integer primary key is zigzag encoded and then compressed into a variable-length integer type (varint). That is, the storage representation of the integer primary key is: integer key "varint(zigzag(Key))". Zigzag encoding maps signed integers to unsigned integers, and then the varint encoding rules are used to achieve data compression.

[0087] ② Other primary key types, such as variable-length primary keys (varlen Key). Variable-length primary keys use a variable-length integer type (varint) to record the primary key length, followed by the binary-encoded primary key, and support any user-defined type. That is, one storage representation of a variable-length primary key can be: varlen Key "varint(len).K raw data", where len represents the primary key length and K raw data represents the binary-encoded primary key.

[0088] 2) In key-value data, the primary key value (Value) is the location of the column data, supporting the following two storage types:

[0089] ① The corresponding columnar data is valid, which is the file (file id) and offset (pos) of the corresponding row data in the columnar storage. For an example of insert entry storage given below, please refer to the following:

[0090] Insert item:

[0091] Key encoded varint(file id) varint(pos)

[0092] Here, "Key encoded" indicates the compressed encoding of the primary key, as described above for the two storage types for the primary key. "varint(file id)varint(pos)" represents the valid storage format for the column data corresponding to the primary key value.

[0093] ② The corresponding column data is deleted. In this case, the Value can be 0, and the corresponding storage format can be found in the following example of deleted entry storage:

[0094] Delete item:

[0095] Key encoded varint(0)

[0096] Among them, "varint(0)" is the storage format in which the column corresponding to the value of the primary key, Value, is deleted.

[0097] Furthermore, when batch writing target data, operations with the same primary key in the batch key-value data included in the target data will be compressed into a single record (either an insert record or a delete record), and all will be encoded according to the methods described in 1) and 2) above. These records will then be densely arranged and stored in the log block (also called a storage block) contained in the appendable log file on the data storage node 14, such as OSS, according to the order of operations. It is understood that the appendable log file is the active data file on the memory layer of the data storage node 14 in other embodiments, such as the first active data file. Figure 3c An example diagram of an appendable log file is shown; see [link / reference]. Figure 3c As shown, an appendable log file contains multiple log blocks. Each log block includes a header and a data area. The header stores the block length, verification information (such as CRC), commit timestamp, primary key type, etc., of the log block. The data area is used to store data, such as the raw data shown. Specifically, the data types stored in this data area can include inserted entries and deleted entries.

[0098] Furthermore, it should be noted that the append write operation described above is atomic on data storage node 14. To ensure the atomicity of the append write operation, the write operation process will be interrupted and terminated if any error is detected before the target data is successfully committed. Such errors may include, but are not limited to, other concurrent append write operations, program crashes, or master node failures. For example, if both the primary node 11 and the backup node 12 unexpectedly believe they are the primary node with write permissions, they might concurrently write data to the first active data file. In this case, when the primary node 11 appends the target data to the first active data file "abc" and prepares to update the metadata recorded in the first active metadata file in the metadata node 13, the verification of the metadata recorded in the first active metadata file before the update determines that another concurrent write operation performed by the backup node 12 has already appended the effective length of the first active data file "abc" to 110. At this point, the write operation performed by the primary node 12 will trigger an error, thus interrupting the entire write operation and causing it to fail. Subsequently, a fix process needs to be executed so that the write operation can be re-executed for the target data using the fixed first active data file and the first active metadata file. The fix process will be described in detail below. The implementation of the verification of the metadata recorded in the first active metadata file will be described in detail in Step 13 below.

[0099] Step 13: Perform a commit operation on metadata node 13 to commit the metadata of the first active data file to the first active metadata file in metadata node 13 for recording, thereby updating the first active metadata file. The specific implementation of this commit operation includes:

[0100] 1) Determine the append write operation information; wherein, the append write operation information may include, but is not limited to, the write position, the write data information of this write operation, the write position may include the file identifier of the first active data file (which may include the file name file_name such as abc, and may further include the file identifier number fild_id such as 1), and the original data information in the first active data file before the append write operation (which may include the valid length of the original data valid_length such as 100, and may further include the total number of rows of the original data total_rows such as 10);

[0101] 2) Based on the write positions contained in the append write operation information, a verification is performed on the first active metadata file in metadata node 13 to check whether at least one of the metadata items originally recorded in the first active metadata file, such as file name, file identifier, effective length, and total number of lines, has changed during the intermediate process from step Step 11 "reading metadata" to step Step 13. If a change has occurred, it indicates that the first active data file has been unexpectedly modified (e.g., modified due to unexpected concurrent write operations), and the first active metadata file will not be updated successfully. Conversely, if no change has occurred, the effective length, total number of lines, and latest update time recorded in the first active metadata file will be updated based on the write data information contained in the append write operation information.

[0102] For example, following the target data example given in Steps 11 and 12 above, the append write operation information (also known as append write operation log) for the target data includes: fild_id:1 (file identifier is 1), file_name:abc (file name is abc), last_pos:100 (last position, i.e., the effective length of the original data), last_rows:10 (last row number, the total number of rows of the original data), written_rows:2 (number of rows of data written is 2), written_bytes:20 (number of bytes of data written is 20), tso:124 (write time is 124). Based on this append operation information, the "updada+where" statement can be used to verify and update the first active metadata file, as exemplified below:

[0103] updata "meta" set

[0104] "valid_length" = 120 "total_rows" = 12 "latest_tso" = 124

[0105] where

[0106] "file_id" = 1 and "file_name" = abc and "valid_length" = 100 and "total_rows" = 10.

[0107] The "update + where" statement above means: First, it checks whether the filename (file_name), fileid (file_id), valid length (valid_length), and total number of rows (total_rows) recorded in the first active metadata file have changed according to the conditions after the where statement. If none of them have changed, the update statement is triggered to update the valid length (valid_length), total number of rows (total_rows), and latest update time (latest_tso) recorded in the first active metadata file. Specifically, the valid length (valid_length) recorded in the first active metadata file will be updated from 100 to 120, the total number of rows (total_rows) will be updated from 10 to 12, and the latest update time (latest_tso) will be updated from 123 to 124. At this point, the first active metadata file is updated, that is, the metadata of the first active data file is successfully submitted, and the write operation process for the target data ends.

[0108] Therefore, in this application, before updating the first active metadata file, the first active metadata file is first checked to see if it meets expectations (whether it has changed). This ensures that there are no other concurrent operations in the intermediate process before the update, thus ensuring that no other concurrent operations will cause update errors when updating the metadata recorded in the first active metadata file. As a result, the write performance of the data processing system will not be affected, and the write is correct, thereby achieving write atomicity.

[0109] It's worth noting that during the execution of the "update+where" statement to update the first active metadata file, the "affected_rows" statement can also be used to return a value to inform the user of the update result. For example, if updating the first active metadata file fails, an "ABORT if affected_rows != 1" statement can be executed to return a termination result. This "ABORT if affected_rows != 1" statement means: if affected_rows != 1... If `affected_rows` equals 1 (not equal to 1), it means that the WHERE clause conditions did not find matching records for filename, file ID, valid length, total number of lines, etc., indicating that these records in the first active metadata file have changed. This signifies an unexpected concurrent operation, causing the commit operation to terminate (ABORT, the commit of the first active data file fails). Conversely, if the first active metadata file is successfully updated, `affected_rows` will be 1. In other words, if `affected_rows = 1`, it means that the WHERE clause conditions found matching records for filename, file ID, valid length, total number of lines, etc., and these records in the first active metadata file have not changed, indicating that the commit of the first active log data file was successful.

[0110] Furthermore, after the write operation of the target data is completed, i.e., after the write module 111 commits the target data, the target data can be replayed and committed to the log cache of the master node 11, so that the read module 112 can read the latest committed data. The log cache is located in the memory cache of the master node 11. Specifically, the committed target data can be replayed and stored in the full layer of the log cache. A detailed description of the implementation of replaying and committing the committed target data to the log cache of the master node 11 will be provided in other embodiments. Figure 4a The relevant content in the description is detailed, so it will not be repeated here.

[0111] Based on the above Figure 4b The described write operation process content is approved by this application. Figure 7A flowchart illustrating the primary key index data processing method executed by the primary node 14 is shown. When the primary node 14 fails and the backup node 12 switches to become the primary node, this method applies to the backup node 12. The backup node 12 will be described in detail in other embodiments below, and will not be elaborated here. Furthermore, the aforementioned metadata node 13 is used to store metadata corresponding to each data file in the data storage node 14. This metadata includes usable metadata that ensures the atomicity of data writing (such as file length (valid_length), total number of rows, etc.). Specifically, as shown... Figure 7 As shown, the primary key index data processing method executed by the master node 14 may include the following steps:

[0112] 101. Read the first active metadata file in the metadata node 13, wherein the first active metadata file contains metadata of the first active data file in the data storage node 14;

[0113] 102. Based on the metadata of the first active data file, append the target data to be submitted to the first active data file; wherein, the append write operation log for the target data includes the write location and the write data information;

[0114] 103. Based on the write location, verify the metadata of the first active metadata file record;

[0115] 104. When the verification is successful, update part of the metadata recorded in the first active metadata file according to the written data information.

[0116] The aforementioned write location includes the file identifier of the first active data file and the original data information in the first active data file before the append write operation. The file identifier of the first active data file may include the file name (file_name) and, in addition, the file identifier number (file_id). The original data information in the first active data file may include the valid length of the original data and, in addition, the total number of rows of the original data.

[0117] In addition, still combined Figure 4bThe described write operation process also reveals that unexpected situations can sometimes cause the write operation process to be interrupted. In the event of an interruption, the master node 11 needs to perform repair operations on the metadata node 13 and the data storage node 14 before re-executing the write operation process. In this application, if the write operation process is interrupted before a successful commit due to any reason (such as program crash or other concurrent write operations), and there are residual active log appends in the first active data file of the data storage node 14, the master node 11 needs to perform a repair operation. Among them, the residual append-active log refers to the following: during the write operation process, at least part of the target data has been appended to the first active data file of the data storage node 14, but due to program crash or other concurrent write operations, not all of the target data has been appended to the first active data file or the first active metadata file in the metadata node 13 has not been successfully updated (i.e., the metadata of the first active data file recorded in the metadata node has not been successfully updated), so that the metadata of the first active data file recorded in the first active metadata file (such as file length) is inconsistent with the actual metadata of the first active data file.

[0118] The following details the fix operation process performed by master node 14. The fix operation is triggered when the metadata of the active data file recorded in metadata node 13 is inconsistent with the actual metadata of the currently active data file in data storage node 14.

[0119] See also Figure 4c As shown, during the write operation process for the target data to be submitted, when a repair trigger condition is detected, such as due to concurrent operations or program crashes causing the write operation to be interrupted before successful submission, although the target data is appended to the first active data file named "abc" in data storage node 14, when updating the metadata of the first active data file recorded in the first active metadata file in metadata node 13, it is found that the metadata originally recorded in the first active metadata file has been unexpectedly tampered with and changed. At this point, it can be determined that the repair trigger condition is met, and the master node 11 will execute a repair operation process. The repair operation process is as follows:

[0120] First, a select for update statement is used to lock the first active data file to prevent other concurrent operations from performing repair operations on it. Specifically, locking the first active data file may mean converting it into an immutable log data file.

[0121] Then, based on the metadata recorded in the first active metadata file, the locked first active data file (i.e., the immutable data file) is read. Upon reading, it is found that the file length (valid_length) of the first active data file recorded in the first active metadata file should be 100. However, when reading from data storage node 14, the actual length of the first active data file is found to be 120. This indicates that the last 20 bytes of the first active data file are dirty data previously written but not successfully committed. Since data storage node 14, like OSS, does not support file truncation for append writes, it is impossible to delete these 20 bytes of dirty data by truncating them. The only operation is to create a new second active data file with a filename such as "def" and copy the first 100 bytes of valid data from the locked first active data file to the newly created second active data file. After completing this series of operations, the metadata node 13 is updated to change the filename "abc" recorded in the first active log metadata file to "def" while keeping other record metadata unchanged. This ensures that the metadata recorded in the first active metadata file points to the repaired second active data file. When performing a write operation again, the system will search for the second active data file named "def" based on the metadata recorded in the first active metadata file. Then, it will verify the length of the second active data file. Once the verification is successful, data can be appended to the second active data file.

[0122] It's worth noting that the reason for converting the first active data file into an immutable data file during the repair operation is to archive it as a historical file, so that it can be used only later for troubleshooting or concurrent reads. For example, although the last 20 bytes of the first active data file may contain dirty data, the first 100 bytes are still valid. If, during the repair process, before the first active metadata file in metadata node 13 is successfully updated, some concurrent read operations on backup node 12 or other nodes are still trying to read the old, locked first active data file based on the metadata recorded in that file, then the problem of not being able to read the data due to the old first active data file being deleted will not occur. Furthermore, making the name of the newly created second active data file different from the original old first active data file ensures the correct handling of other concurrent writes. For example, if some concurrent write operations are still operating on the original old first active data file, they will detect the file name change when updating the first active metadata file, causing the metadata update to fail. This will result in the data written to the original old first active data file not being successfully committed and potentially being discarded later. This ensures that data successfully submitted will not be discarded but will be persistently saved. For details on the specific implementation of data persistence, please refer to the compression and merging operations described in other embodiments below.

[0123] Based on combination Figure 4c The above-described data processing method executed by master node 11 may further include the following steps:

[0124] 105. When it is determined that the repair conditions are met before the target data is successfully submitted, the first active data file is converted into an immutable data file, and a second active data file is created in the data storage node 14 for subsequent write operation services, and the original data located before the write point in the immutable data file is copied to the second active data file.

[0125] 106. Update the file name recorded in the first active metadata file to the file name of the second active data file;

[0126] The second active data file has a different file name from the first active data file, and the conditions for satisfying the repair include: an interruption was triggered before the target data was successfully submitted and at least a portion of the target log data had been written to the first active data file.

[0127] In this application, the target data can be uncommitted data (key-value data) stored in the invisible layer (denoted as the delta layer) within the master node 11. The delta layer is located in the log cache of the master node 11, which in turn is located in the memory cache of the master node 11. See [link to relevant documentation]. Figure 4a As shown, the log cache includes two memory data structure layers: an invisible layer (delta layer) and a full layer (full layer). In specific implementations, both the delta layer and the full layer can be map data structures. The delta layer is used to store uncommitted key-value data, and this key-value data is invisible and cannot be read within the delta layer. After receiving a write request, the master node 11 first writes the key-value data carried by the write request into the delta layer via a write thread. When a commit is required, for example, if a batch of key-value data is written into the delta layer, causing the cumulative amount of key-value data in the delta layer to reach a set threshold, or if a user triggers a commit operation, a group commit thread will be executed to traverse all key-value data in the delta layer and generate the target data to be committed (denoted as delta log) based on all the traversed key-value data. Then, by executing the above combined... Figure 4b The described write operation process uploads the target data to data storage node 14 to complete the commit. After the commit is complete, the target data is replayed in the full layer to accelerate subsequent reading. In specific implementation, when replaying the committed target data, a map can be generated for the target data. The map is a key-value data structure. Specifically, the generated map can be a hash map or a red-black tree, etc. Preferably, the map is a hash map in this application. Thus, the full layer can have multiple hash maps, which are linked list hashes (arrays and linked lists) used to record the log data that has been committed. Through this full layer, query performance can be accelerated when querying data later. A detailed description of how the full layer can accelerate query performance will be provided in the read operation process in other embodiments, and will not be elaborated here.

[0128] Therefore, the memory of master node 11 may contain an invisible layer and a complete layer; and the data processing method executed by master node 11 may also include the following steps:

[0129] 107. In response to a write request, write the key-value data carried in the write request to the invisible layer;

[0130] 108. When the submission conditions are met, the target data is generated using all key-value data in the invisible layer; and after the target data is successfully submitted, the target data is re-placed in the complete layer for subsequent reading.

[0131] The submission conditions include, but are not limited to, the following: the cumulative amount of key-value data in the invisible layer reaches a set quantity threshold and / or the storage time reaches a set duration threshold, and the user triggers the submission operation.

[0132] See also Figure 4a As data is continuously written to the full layer, the data size in the full layer will inevitably grow larger and larger. To avoid the full layer from consuming too much memory space due to excessive storage, this application will purge (i.e., eliminate) the data that has been compressed and merged into the storage disk (such as an SSD) of the data storage node 14 that has been persistently stored. Specifically, when it is determined that the key-value data in the first active data file in the data storage node 14 has been merged into the storage disk of the data storage node 14 by the backup node 12 through a compression and merging operation, it is no longer necessary to continue storing the same key-value data as the first active data in the full layer. This will result in a rolling purging operation, which involves rolling the purging of key-value data that has been persistently stored in the data storage node 14 within the full layer. Although the purged key-value data cannot be found in the full layer during reading, it can still be queried in the storage disk of the data storage node 14.

[0133] As can be seen from the descriptions of other embodiments: the first active data file is a data file (active data file) located on the memory layer of data storage node 14, and the storage disk in data storage node 14 also includes multi-level ordered data files. For example, as shown in Figure 5b, the storage disk also includes ordered data files such as SSTd1-1 to SSTd1-4 on the first persistent data layer L1 and ordered data files such as SSTd2-1 to SSTd2-16 on the second persistent data layer L2. The ordered data files are used for persistent storage of data. Specifically, when persistently storing the key-value data in the first active data file, the first active data file is first converted into an immutable data file, and then the immutable data file is compressed and merged to achieve the distribution and merging of the key-value data in the immutable data file into the corresponding four ordered data files such as SSTd1-1 to SSTd1-4 in the adjacent lower layer (first persistent data layer L1) for persistent storage. In other words, the immutable data file is an intermediate state in the process of persistently storing key-value data from the first active data file in the upper layer into several corresponding ordered data files in the adjacent lower layer. The purpose is to prevent write operations from being blocked during the data transfer process. Write operations can be handled by the new second active data file without waiting because the first active log data file is locked. For the specific implementation principle of persistent storage of key-value data in the first active data file, please refer to the relevant content described in other embodiments regarding the compression and merging operation performed on the backup node 12, which will not be elaborated here.

[0134] Therefore, the data processing method executed by master node 11 may also include the following steps:

[0135] 109. When the key-value data in the first active data file of the upper layer is compressed and merged into the corresponding ordered data file of the adjacent lower layer in the data storage node 14 for persistent storage, the key-value data stored in the full layer that is the same as the first active data file is cleared.

[0136] The data storage node 14 contains multiple data files with hierarchical relationships, including the first active data file currently in memory and multi-level ordered data files in the storage disk.

[0137] In practice, the key-value data in the first active data file is compressed and merged into the corresponding ordered data file SSTd in the data storage node 14. This is achieved by the backup node 12 performing a compression and merging operation. Specifically, the compression and merging operation is performed by the compression and merging module 121 in the backup node 12. The reasons for performing the compression and merging operation include: the key-value data recorded in the active data file (such as the first active log data file) may contain multiple insert and delete operations on a single key record. Therefore, it is necessary to merge duplicate records into one record and then merge it with the corresponding ordered data file (SSTd). This facilitates fast searching during subsequent readings, reduces storage space usage, and achieves persistent data storage. In the first active data file, key-value data is appended and unordered, so the time complexity of searching for the existence of a data item is O(n). In contrast, the data in the ordered data file (SSTd) is arranged in key order and is ordered. Therefore, a data item can be searched in the ordered data file using methods such as binary search, with a search time complexity of log(n). Thus, searching for data in the ordered data file is much faster than searching for data in the active data file.

[0138] For a description of the compression and merging operation performed by backup node 12 on active data files in the database storage node, please refer to other embodiments. Figures 5a to 5c The description includes information related to the compression and merging operation process.

[0139] Figures 5a to 5c This diagram illustrates the principle of the compression and merging operation performed by backup node 12 when the compression and merging trigger conditions are met. (See also...) Figure 5a As shown, metadata node 13 contains a first active metadata file and the latest metadata snapshot. The first active metadata file records the metadata of the currently active data file in data storage node 14. The latest metadata snapshot records the metadata of each ordered data file (SSTd) in the persistent storage structure of data storage node 14. The metadata snapshot can be understood as a copy of the state of the metadata merging tree (SSTmeta tree) in metadata node 13 at a certain moment. The metadata merging tree can be understood as... Figure 3bThe metadata node 13 shown in the figure illustrates a multi-level persistent metadata layer (such as the first persistent metadata layer Lm1, the second persistent metadata layer Lm2, etc.). The ordered metadata file (SSTm) on each persistent metadata layer is used to record the metadata of the corresponding ordered data file (SSTd) in the corresponding persistent data layer in the data storage node. For example, the ordered metadata file SSTm1-1 on the first persistent metadata layer Lm1 is used to record the metadata of the ordered data file SSTd1-1 on the first persistent data layer.

[0140] Specifically, the metadata of the ordered data files recorded within the metadata merge tree can include, but is not limited to: file name, file identifier, file level (e.g., the layer number of the persistent data layer), file length (the effective length of the data in the file), total number of lines, latest update time, hash bucket information, etc. Additionally, the latest metadata snapshot may contain immutable metadata files. In this case, it indicates that a previous compression and merge operation was incomplete. The reasons for this incompleteness may include standby node power failure, system crash, or program errors in the compression and merge operation. In this situation, a compression and merge operation will first be performed on the latest snapshot containing the immutable metadata files to remove them and restore the system to its initial state (a metadata snapshot containing only the metadata merge tree and no immutable metadata files). Then, this restored metadata snapshot will be used to implement the final version. Figure 5a The target metadata snapshot is shown for construction. Conversely, if the latest metadata snapshot does not contain an immutable metadata file, this latest metadata snapshot can be used directly to achieve the desired result. Figure 5a The target metadata snapshot is shown.

[0141] See also Figure 5a If the latest metadata snapshot in metadata node 13 contains only the metadata merge tree (i.e., only records metadata of the ordered data file (SSTd)) and does not contain immutable metadata files, then: assuming the length threshold set for the active data file in data storage node 14 is 98, and the file length (effective length) of the first active data file abc in the data storage node recorded in the first active metadata file of metadata node 13 is 100, it indicates that the file length of the first active data file has exceeded the set length threshold, so a compression and merge operation needs to be triggered. Before triggering a compression and merge operation, a selective update statement can be used to implement a global mutex lock to ensure that only one compression and merge operation task is executed at a time.

[0142] For example, `select file_name=abc for update` creates a lock record using the special key value "file_name=abc" in the first active metadata file. This ensures that only one thread is executing the compression and merging operation task globally by preempting the mutex lock of this lock record.

[0143] Then, an update statement can be executed to construct a target metadata snapshot as the target object upon which the compression and merging operation is based. Specifically, the process for constructing the target metadata snapshot is as follows:

[0144] 1) The first active metadata file in metadata node 13 is converted into an immutable metadata file. At the same time, a new second active metadata file is generated. Compared with the previous first active metadata file, the file identifier file_id remains unchanged (still 1), the file name file_name is changed to def, the valid length valid_length and the total number of rows total_rows are both changed to 0, and the latest update time latest_tso remains unchanged (still 123). Subsequent writes can continue to append metadata content to this new second active metadata file.

[0145] 2) Combine the immutable metadata file with the latest metadata snapshot to form a new metadata snapshot as the target snapshot.

[0146] The subsequent compression and merging operation involves performing an equivalent transformation on the target metadata snapshot, eliminating the immutable metadata files contained within it, and restoring it to its original state.

[0147] The implementation principle of the subsequent compression and merging operation, which performs an equivalent transformation on the target metadata snapshot, can be found in Figure 5b below. Figure 5c The relevant content described.

[0148] Before introducing the implementation principle of performing an equivalent transformation on the target metadata snapshot, let's first combine it with... Figure 3a and Figure 3b In addition to the content related to data storage node 14 described in other embodiments, the data storage node 14 will be comprehensively described below. Specifically,

[0149] Such as combination Figure 3a and Figure 3bAs shown, data storage node 14 contains multiple data files with hierarchical relationships. Specifically, data storage node 14 contains multiple storage layers, including a memory layer L0 and at least one persistent data layer. The data files on memory layer L0 include active log data files, such as the current first active log data file, and in some cases, other data files. For example, when the length of the current first active data file reaches a set length threshold, indicating that the accumulated data in the first active data file has reached a set quantity threshold, the first active data file will be converted into an immutable log data file, and a new second active data file will be regenerated to continue providing write operation services. Therefore, memory layer L0 may also contain immutable data files. A persistent data layer contains a fixed number of ordered data files SSTd. Specifically, the number of ordered data files SSTd on a persistent data layer can be strictly controlled according to a pre-set data distribution parameter fanout. The data distribution parameter fanout = N indicates that the data in a data file that needs to be compressed and merged on a data layer needs to be distributed to N data files located in the next lower data layer. N is an integer greater than or equal to 2.

[0150] In one instance, the value N of fanout can be a power of 2, such as 2. 2 =4 or 2 3 = 8, etc. In this scenario, the number of ordered data files SSTd on a persistent data layer can increase exponentially with the increase of the layer based on the data distribution parameter fanout, that is: the number of ordered data files SSTd T on a persistent data layer = the number of ordered data files on the previous persistent data layer * the value of the data distribution parameter fanout (i.e., N), where the number of ordered data files SSTd in the first persistent data layer adjacent to the memory layer (the uppermost persistent data layer in the storage disk of the data storage node) is equal to the value of the data distribution parameter fanout. Alternatively, another expression for the number of ordered data files SSTd T in a persistent data layer is: T = (fanout)^(the layer corresponding to this persistent data layer).

[0151] Figure 5b-1The diagram illustrates an example of a data storage node 14 comprising several storage layers, including a memory layer L0, a first persistent data layer L1, and a second persistent data layer L2. The first persistent data layer L1 is below the memory layer L0, and the second persistent data layer L2 is below the first persistent data layer L1. The memory layer L0 contains a second active data file and an immutable data file. The immutable data file is created by converting the previous first active data file after determining that conversion conditions are met, and a new second active data file is generated during the conversion. Meeting the conversion conditions includes, for example, the file length of the first active data file being greater than or equal to a set length threshold. The first persistent data layer L1 contains four ordered data files, SSTd1-1 to SSTd1-4. The second persistent data layer L2 contains 16 ordered data files, SSTd2-1 to SSTd2-16. The data distribution parameter `fount` is set to 4. When the backup node 12 triggers a compression and merging operation, it scans the data storage node 14 based on the aforementioned target metadata snapshot to determine at least one subcompaction task to be executed. For example, key-value data in the immutable data file in memory layer L0 needs to be distributed to the four ordered data files SSTd1-1 to SSTd1-4 in the first persistent data layer L1, which have a corresponding relationship with it, for persistent storage. Thus, the immutable data file in memory layer L0 and the four ordered data files SSTd1-1 to SSTd1-4 in the first persistent data layer L1 form a sub-compression and merging task. At the same time, it is found that the key-value data in the first ordered data file SSTd1-1 in the first persistent data layer L1 exceeds the threshold, and its key-value data needs to be distributed to the first four ordered data files (SSTd2-1 to SSTd2-4) in the second persistent data layer L2. Thus, the ordered data file SSTd1-1 in the first persistent data layer L1 and the first four ordered data files (SSTd2-1 to SSTd2-4) in the second persistent data layer L2 form a sub-merge and compression task. As can be seen, each sub-merge compression task involves one data file in the upper layer and four (e.g., four) corresponding data files in the adjacent lower layer. Furthermore, the execution of each sub-merge compression task is independent and forms a completely separate unit of work. Therefore, once at least one sub-merge compression task is identified, it can be executed in parallel.

[0152] In another instance, the value N of the data scattering parameter fanout can also be any other positive integer other than a power of 2, such as 3, 5, or 7. The value N of fanout indicates that data in a data file at a higher level needs to be scattered into N data files at the next lower level for storage. In this scenario, the value N of the data scattering parameter fanout for different levels can be the same, different, or some levels can be the same while the rest are different, and so on.

[0153] Figure 5b-3 Examples of the same fanout across different layers are shown. See also... Figure 5b-3 Assuming that the fanout value N for each layer is 3, and that data storage node 14 has one immutable data file on memory layer L0, three ordered data files (SSTd1-1 to SSTd1-3) on the first persistent data layer L1, and nine ordered data files (SSTd2-1 to SSTd2-9) on the second persistent data layer L2, when backup node 12 triggers the compression and merging operation, it will scan data storage node 14 based on the aforementioned target metadata snapshot to determine at least one sub-compaction task that needs to be executed. For example, if the key-value data in the immutable data file in memory layer L0 needs to be distributed, compressed, and merged downwards, then this immutable data file and the three ordered data files SSTd1-1 to SSTd1-3 on the next layer, the first persistent data layer L1, form a sub-compression and merge task. At the same time, it is found that the amount of key-value data in the first ordered data file SSTd1-1 in the first persistent data layer L1 exceeds the threshold, and its key-value data needs to be distributed and compressed downwards. Then SSTd1-1 and its corresponding SSTd2-1 to SSTd2-3 on the second persistent data layer L2 form a compression and merge task.

[0154] Figure 5b-4 This shows an example where different layers have different fanout values. See also... Figure 5b-4The fanout value N for memory layer L0 of data storage node 14 is 3, indicating that data in a data file on memory layer L0 needs to be distributed to three corresponding data files (SSTd) on the next layer, the first persistent data layer L1. Therefore, the first persistent data layer L1 should contain 3 data files, such as the ordered data files SSTd1-1 to SSTd1-3. The fanout value N for the first persistent data layer L1 is 2, indicating that data in a data file on the first persistent data layer L1 needs to be distributed to two corresponding data files on the next layer, the second persistent data layer L2. Therefore, the second persistent data layer L2 should contain 6 data files (= number of data files on the first persistent data layer L1 * corresponding fanout value N = 3 * 2), such as SSTd2-1 to SSTd2-6. When backup node 12 triggers the compression and merging operation, it scans data storage node 14 based on the aforementioned target metadata snapshot to determine at least one sub-compaction task to be executed. For example, if the key-value data in the immutable data file in memory layer L0 needs to be distributed, compressed, and merged downwards, then this immutable data file and the three ordered data files SSTd1-1 to SSTd1-3 on the first persistent data layer L1 below form a sub-compression and merge task. At the same time, it is found that the amount of key-value data in the first ordered data file SSTd1-1 in the first persistent data layer L1 exceeds the threshold, and its key-value data needs to be distributed and compressed downwards. Then SSTd1-1 and SSTd2-1 and SSTd2-2 on the second persistent data layer L2 form a compression and merge task.

[0155] The following is combined Figure 5c Let me describe in detail the execution process of one of the sub-compression and merging tasks.

[0156] like Figure 5c As shown, assuming that at least one sub-compression and merging task is identified, including sub-compression and merging task A, sub-compression and merging task B, etc., each sub-compression and merging task includes one upper-level data file and corresponding fanout data files in the adjacent lower-level layers, that is, each sub-compression and merging task includes fanout+1 data files. Taking the execution of sub-compression and merging task A as an example, the backup node 12 specifically performs the following steps:

[0157] Step 21: Load Data Files. Load all the fanout+1 data files involved in sub-compression and merging task A into local memory (memory cache). In practice, data file loading can be implemented based on the set caching mechanism. For example, if the data files in data storage node 14 are cached in local memory beforehand, the fanout+1 data files involved in sub-compression and merging task A can be directly read from local memory. If the data files in data storage node 14 are cached in local disk (localSSD cache), the fanout+1 data files involved in sub-compression and merging task A can be loaded into local memory through the local disk-related thread. If the files cannot be loaded from either local memory or local disk, the five data files involved in sub-compression and merging task A will be loaded into local memory from data storage node 14, such as OSS, through a network thread. Because sub-compression and merging task A involves loading fanout+1 data files, there will be a synchronous wait (syns) to wait for all fanout+1 data files involved in sub-compression and merging task A to be loaded into local memory before proceeding to the next step Step22.

[0158] Step 22: Merge Operation. This merge operation generates an iterator for each of the fanout+1 data files involved in sub-compression and merging task A. This iterator is a pointer to an ordered sequence. The data in these fanout+1 data files can be reordered sequentially using the iterator before merging. Specifically, during data merging, data from one upper-level data file is distributed to the corresponding fanout data files in the lower levels. Therefore, the sorted data in the upper-level data file needs to be processed one by one. First, the partition is used to determine which of the fanout data files the data belongs to, and then the data is merged with the corresponding lower-level data file. The merging process can be implemented based on the hash values ​​of the data in the upper-level data files and the hash bucket information of the lower-level data files.

[0159] To facilitate understanding of Step 22, the following example will be provided:

[0160] 1. Scenario where the value of the data dispersion parameter fanout, N, is a power of 2. Specifically, for example, fanout = 4 (= 2...). 2 (For example, in the scenario)

[0161] Combination Figure 5b-1 and Figure 5b-2Assume that the sub-compression and merging task A involves 5 (fanout+1) data files, including: an immutable data file on the memory layer L0 (layer 0) of data storage node 14 and four ordered data files SSTd1-1 to SSTd1-4 on the first persistent data layer L1 (layer 1). The immutable data file contains multiple sorted key-value pairs: key-value pair data01 (Key1:Value1), key-value pair data02 (Key2:Value2), key-value pair data03 (Key3:Value3), etc. According to the order of the key-value pairs in the immutable data file, we can first perform a hash calculation on the primary key key1 in key-value pair data01 to obtain the corresponding hash value HashV01. Then, based on the last two digits of the hash value HashV01 and the hash bucket information of the four ordered data files SSTd1-1 to SSTd1-4, the target ordered data file can be determined from the four ordered data files SSTd1-1 to SSTd1-4. Among them, the last two digits of the hash value of the primary key of all key-value data stored in each of the four ordered data files SSTd1-1 to SSTd1-4 are the same, and the hash bucket information of each ordered data file may contain a hash bucket number, which can indicate the last two digits of the hash value of the primary key of the key-value data in the corresponding ordered data file. For example, if the last two digits of the hash value HashV01 corresponding to the key-value data data01 are 00, and the hash bucket information of the four ordered data files SSTd1-1, SSTd1-2, SSTd1-3, and SSTd1-4 contains hash bucket numbers of 00, 01, 10, and 11 respectively, then it means that SSTd1-1 is used to store the key-value data whose last two digits of the hash value are 00. Thus, SSTd1-1 can be identified as the target ordered data file, and the key-value data data01 will be inserted into SSTd1-1. Similarly, other key-value data in the immutable data file can be distributed and stored into the corresponding ordered data files of the four ordered data files SSTd1-1 to SSTd1-4.

[0162] Alternatively, suppose the five data files involved in sub-compression and merging task A include: the ordered data file SSTd1-1 on the first persistent data layer of data storage node 14, and the four ordered data files SSTd2-1 to SSTd2-4 on the second persistent data layer. The data in the upper-level ordered data file SSTd1-1 needs to be distributed to the corresponding lower-level ordered data files SSTd2-1 to SSTd2-4. The key-value data in each ordered data file is ordered (i.e., sorted by key). The ordered key-value data in the ordered data file SSTd1-1 includes key-value data data11, key-value data dat12, key-value data data13, etc. Based on the sorting of key-value data in the ordered data file SSTd1-1, we can first perform a hash calculation on the primary key key11 in the key-value data data11 to obtain the corresponding hash value HashV11. Then, based on the last four digits of this hash value HashV11 and the hash bucket information of the four ordered data files SSTd2-1 to SSTd2-4, we can determine the target ordered data file from these four ordered data files. Specifically, in each of the four ordered data files SSTd2-1 to SSTd2-4, the last four digits of the hash value of the primary key of all key-value pairs stored in each ordered data file are the same, and the hash bucket information in each ordered data file may contain a hash bucket number, which indicates the last four digits of the hash value of the primary key of the key-value data in the corresponding ordered data file. For example, if the last four digits of the hash value HashV11 corresponding to the key-value data data11 are 1000, and the hash bucket information of the four ordered data files SSTd2-1, SSTd2-2, SSTd2-3, and SSTd2-4 contains hash bucket numbers of 0000, 0100, 1000, and 1100 respectively, then it means that SSTd2-3 is used to store the key-value data whose last four digits of the hash value are 1000. Thus, SSTd2-3 can be identified as the target ordered data file, and the key-value data data11 will be inserted into the ordered data file SSTd2-3. Similarly, other key-value data in the ordered data file SSTd1-1 can be distributed and stored into the corresponding ordered data files of the four ordered data files SSTd2-1 to SSTd2-4.

[0163] It should be noted here that: because the last two bits of the hash value of the primary key of all key-value data in the ordered data file SSTd1-1 are the same (all are 00), in this case, simply using only the last two bits of the hash value of the primary key of the key-value data to distribute the key-value data in the ordered data file SSTd1-1 into the four ordered data files SSTd2-1 to SSTd2-4, as in Example 1 above, will not work. It is necessary to further combine the last third and fourth bits of the hash value of the primary key of the key-value data to achieve the desired result. The reason for this distributed storage is that the key-value data in the four ordered data files SSTd2-1 to SSTd2-4 at the lower level all originate from the ordered data file SSTd1-1 at the upper level. Therefore, the last two digits of the hash value corresponding to the key-value data in each of these four ordered data files will be the same as the last two digits of the hash value corresponding to the key-value data in the upper-level SSTd1-1. Thus, relying solely on the last two digits of the hash value is insufficient for differentiation. However, the last third and fourth digits of the hash value will have different combinations. Therefore, when distributing the key-value data in the ordered data file SSTd1-1, if the last two digits of the hash value of the primary key of all key-value data are the same, the last third and fourth digits of the hash value can be used to determine which lower-level ordered data file the key-value data should be distributed to for persistent storage.

[0164] In summary, this application uses consistent hashing (rehash) to distribute data from a data file in the upper layer to several corresponding data files in the lower layer for persistent storage. Furthermore, this consistent hashing method ensures that when a data file in the lower layer needs to be rehashed, the ownership relationship of the data files in the upper layer will not be disrupted. In other words, data from a data file in the upper layer cannot be infinitely distributed to other data files in the lower layer; it can only be distributed to a few specific data files in the lower layer. This is also the principle behind this application's ability to execute multiple sub-compression and merging tasks in parallel.

[0165] 2. In scenarios where the value N of the data scattering parameter fanout is not a power of 2, when compressing and merging data from an upper-level data file down to a lower level, this can be achieved by performing division and modulo operations on the hash value of the data. Specifically, the hash value of the data can be divided by the number of upper-level data files, and then the modulo operation can be performed on the corresponding fanout value of the upper level. The division is an integer division, and only the integer part of the result is retained; the modulo operation is the remainder operation.

[0166] Example 1, combined with Figure 5b-3The example shown illustrates different layers with the same fanout (all 3). Assume that sub-compression merging task A involves the following files: an immutable data file on memory layer L0 (layer 0) of data storage node 14, and three ordered data files SSTd1-1 to SSTd1-3 on the first persistent data layer L1 (layer 1). Key-value data data01 (Key1:Value1) is a data point in the immutable data file. The primary key key1 in key-value data data01 is hashed to obtain a hash value HashV01. This hash value HashV01 is divided by 1 (the number of files on memory layer L0; dividing by 1 here is equivalent to no operation) and then modulo 3 (i.e., taking the remainder of the fanout corresponding to memory layer L0 (whose value N is 3)) to obtain the modulo value m (remainder); where the modulo value m may be 0, 1, or 2. Next, based on the modulus m, one of the three ordered data files (SSTd1-1 to SSTd1-3) is selected as the target ordered data file, and the key-value data data01 is inserted into this target ordered data file for storage. For example, if the hash bucket numbers of the three ordered data files SSTd1-1 to SSTd1-3 are numbered sequentially starting from 0 and incrementing by one, then the ordered data file whose hash bucket number is the same as the modulus m among SSTd1-1 to SSTd1-3 is the target ordered data file. For instance, if the modulus m is 1, then SSTd1-2 (whose hash bucket number is 1) is the target ordered data file, and the key-value data data01 can be inserted into SSTd1-2 for storage. Similarly, other key-value data from immutable data files can be distributed and inserted into the corresponding ordered data files in the first persistent data layer L1. Similarly, if the files involved in sub-compression and merging task A include SSTd1-1 on the first persistent data layer L1 (layer 1) of data storage node 14 and SSTd2-1, SSTd2-2 and SSTd2-3 on the second persistent data layer L2 (layer 2), it is also possible to distribute the data in SSTd1-1 to the corresponding ordered data files in the three ordered data files SSTd2-1, SSTd2-2 and SSTd2-3 for storage. This will not be described in detail here.

[0167] Example 2, combined Figure 5b-4The example shown illustrates different fanout values ​​for different layers. If the files involved in sub-compression merging task A include: an immutable data file on memory layer L0 (layer 0) of data storage node 14 and three ordered data files SSTd1-1 to SSTd1-3 on the first persistent data layer L1 (layer 1), and the fanout value N corresponding to memory layer L0 is 3, then the specific implementation of distributing the key-value data in an immutable data file to the corresponding ordered data files in the three ordered data files SSTd1-1 to SSTd1-3 for storage can be found in the relevant example given in Example 1 above, and will not be elaborated here.

[0168] Or, continue to see Figure 5b-4 If the files involved in sub-compression and merging task A include: one ordered data file SSTd1-1 on the first persistent data layer L1 in data storage node 14, and two ordered data files SSTd2-1 and SSTd2-2 on the second persistent data layer L2, and the key-value data data11 (Key11:Value11) is a piece of data in SSTd1-1, the primary key key11 in the key-value data data11 is hashed to obtain the hash value HashV11. Then, the hash value HashV11 is first divided by 3 (the number of ordered data files on the first persistent data layer L1) and then modulo 2 (i.e., the remainder is taken with respect to the fanout (its value N is 2) corresponding to the first persistent data layer L1) to obtain the corresponding modulo value k (remainder), where the modulo value k may be 0 or 1. Then, based on the modulo value k, one of SSTd2-1 and SSTd2-2 is selected as the target ordered data file, and the key-value data data01 is inserted into the target ordered data file for storage. For example, if the hash bucket numbers of the two ordered data files SSTd2-1 and SSTd2-2 are numbered sequentially starting from 0 and incrementing by 1, then the ordered data file in SSTd2-1 and SSTd2-2 with the same hash bucket number as the module d will be the target ordered data file. For instance, if the modulus k is 1, then SSTd2-2 (with hash bucket number 1) will be the target ordered data file, and the key-value data data11 can be inserted into SSTd2-2 for storage. Similarly, other key-value data from SSTd1-1 can be distributed and inserted into the corresponding ordered data files in SSTd2-1 and SSTd2-2 for storage.

[0169] Step 23: Upload Task. Continuing from Example 1 given in Step 22, after merging the data from the upper-level immutable data file involved in the sub-compression and merging task A into the four lower-level ordered data files SSTd1-1 to SSTd1-4 in Step 22, four new ordered data files will be formed, denoted as SSTd1-1', SSTd1-2', SSTd1-3', and SSTd1-4' respectively. Then, upload tasks are executed for each of these four newly formed ordered data files. Upload tasks may include: uploading the corresponding new ordered data file to data storage node 14 (e.g., OSS) and writing its metadata to metadata node 13; additionally, they may include writing the corresponding new ordered data file to the local SSD disk.

[0170] For example, taking the upload task for a new ordered data file SSTd1-1' as an example, when uploading the new ordered data file SSTd1-1' to the data storage node 14, such as OSS, the upload method of adding a new file can be adopted. This achieves the addition of the ordered data file SSTd1-1' to the first persistent data layer L1 of the data storage node 14, while the original old ordered data file SSTd1-1 can be retained for a period of time for delayed deletion. The reason for this is that, considering that before the upload task is completed and the metadata node 13 is updated, the original old ordered data file SSTd1-1 may be read in parallel based on the old metadata recorded in the metadata node 13, SSTd1-1 will not be deleted immediately. After the ordered data file SSTd1-1' is uploaded to the data storage node 14, as shown in Figure 5b and... Figure 5cAs shown, metadata updates will be performed on metadata node 13 to write the metadata of the uploaded ordered data file SSTd1-1' to metadata node 13. For example, assuming that the ordered metadata file SSTm1-1 in the metadata merge tree (SST meta tree) of the target metadata snapshot is used to record the metadata of the original ordered data file SSTd1-1, then during metadata updates, the metadata of the ordered data file SSTd1-1 recorded in the ordered metadata file SSTm1-1 will be updated to the metadata of the new ordered data file SSTd1-1', so that the ordered metadata file SSTm1-1 points to the new ordered data file SSTd1-1'; wherein, the updated metadata content includes, but is not limited to: file name, file length, and check information (such as CRC), etc. Furthermore, when performing an upload task for a new ordered data file SSTd1-1', there is a synchronization wait (sync) to wait for the ordered data file SSTd1-1' to be uploaded to the data storage node 14, such as OSS, and written to the local disk SSD, and for its metadata to be written to the metadata node 13. At this point, the upload task of the ordered data file SSTd1-1' is completed.

[0171] Furthermore, for the four new ordered data files SSTd1-1' to SSTd1-4', there will be a synchronization wait. This synchronization wait is to wait for all four new ordered data files to be uploaded. Specifically, it is to wait for all four new ordered data files to be uploaded to the data storage node 14 (such as OSS) and written to the local disk SSD, and for the metadata to be written to the metadata node 13.

[0172] Once all four new ordered data files have been uploaded, sub-compression and merging task A is complete.

[0173] For other sub-compression and merging tasks, such as sub-compression and merging task B, the operations performed for sub-compression and merging task A can be referenced, while also executing the operations described in Steps 21 to 23 above. After all sub-compression and merging tasks have been completed, as shown in Figure 5b and... Figure 5c As shown, update the metadata node 13 again to eliminate the immutable metadata file in the target metadata snapshot, thereby completing the update of the entire target metadata snapshot. This restores the updated target metadata snapshot to its initial state (i.e., it does not contain immutable metadata files and only contains the metadata merging tree) so that it can be compressed and merged in the next operation.

[0174] It should be noted here that: see Step 22 above for further details. Figure 5b-1 and Figure 5b-2 The given examples illustrate this: if sub-compression and merging task A involves an immutable data file on memory layer L0 of data storage node 14 and four ordered data files SSTd1-1 to SSTd1-4 on the first persistent data layer; and if sub-compression and merging task B involves the ordered data file SSTd1-1 on the first persistent data layer of data storage node 14 and the four ordered data files SSTd2-1 to SSTd2-4 on the second persistent data layer, then when sub-compression and merging tasks A and B are executed in parallel, the ordered data file SSTd1-1 on the first persistent data layer must both push key-value data from the immutable data file into it and push its own key-value data down to the next layer. To address this parallel compression and merging scenario, this application employs a special approach: when executing sub-compression and merging task A, the ordered data file SSTd1-1 is treated as an empty file, and data can only be inserted into it. This prevents task conflicts between sub-compression and merging tasks A and B.

[0175] For ease of description, the above combination is used in other embodiments described below. Figures 5a to 5c When summarizing the function of backup node 12, the "sub-compression and merging task" mentioned above is described as a "sub-task", and the "one data file" at the upper level involved in the sub-compression and merging task is described as the "first data file", and the corresponding "fanout data files" at the lower level is described as the "second data file".

[0176] Therefore, based on the above combination Figures 5a to 5c The content described in this application is approved. Figure 8 A schematic diagram of the data processing method that can be executed by backup node 12 is provided. Figure 8 As shown, the data processing method that can be executed by backup node 12 includes the following steps:

[0177] 201. When the compression and merging trigger condition is met, the compression and merging task to be executed is determined based on the metadata of each data file in the data storage node 14 recorded by the metadata node 13; wherein, the compression and merging task includes at least one subtask that can be executed in parallel, and the subtask involves a first data file at the upper layer and N second data files at the adjacent lower layer in the data storage node 14.

[0178] 202. When executing the subtask, consistent hashing is used to distribute the key-value data in the first data file involved in the subtask to the corresponding N second data files for persistent storage;

[0179] 203. After the distribution is completed, update the metadata of the N second data files recorded in the metadata node 13;

[0180] Wherein, N is the value of the data distribution parameter of the consistent hash, which is used to indicate that data in one data file of the upper layer is distributed to N data files of the lower layer, and N is an integer greater than or equal to 2.

[0181] In practice, the value N of the data dispersion parameter (fanout) can be a power of 2, such as 2... 1 =2 or 2 2 =4 or 2 3 =8 or 2 4 =16, etc., so that when consistent hashing (rehashing) is needed later, it is convenient to directly divide and describe the binary bits of the hash value (see other embodiments for details). Figure 5b-1 and Figure 5b-2 (Example described).

[0182] Of course, in other embodiments, N can also be any integer other than a power of 2. However, in this case, data distribution can be achieved by performing integer division and modulo operations on the hash values ​​of the key-value data in the upper-level data file. For details on this implementation, please refer to the scenarios in the aforementioned other embodiments where the value of fanout N is not a power of 2. Figure 5b-3 and Figure 5b-4 Example content described.

[0183] In this application, the value of fanout N is preferably a power of 2, thereby enabling direct use of bitwise operations to avoid the complexity of integer division and modulo operations.

[0184] In addition, the conditions for triggering compression and merging can include, but are not limited to: the file length of the current first active data file in data storage node 14 reaches the set length threshold, a compression and merging operation actively triggered by the user is detected, the set compression and merging cycle is reached, etc.

[0185] Furthermore, in conjunction with the above... Figure 5a The relevant content described above, specifically "determining the compression and merging task to be performed based on the metadata of each data file in data storage node 14 recorded by metadata node 13," may include:

[0186] 2011. Convert the currently active first meta-information file in the meta-information node 13 into a layer-immutable meta-information file;

[0187] 2012. Combine the immutable metadata file with the latest metadata snapshot in the metadata node 13 to obtain the target metadata snapshot;

[0188] 2013. Based on the target metadata snapshot, determine the compression and merging task that needs to be executed.

[0189] In specific implementation, for example, a snapshot of the target metadata can be scanned. Based on metadata such as file length recorded in each metadata file in the target metadata snapshot, the file identifiers (e.g., file names) of the data files (possibly one or more) in the data storage node that need to be compressed and merged at the lower level can be determined. After determining the data files, the file identifiers of N data files in the lower level corresponding to the data file that needs to be compressed and merged can be determined based on the hierarchical correspondence between the data files in the data storage node. Thus, based on the file identifier of one data file in the upper level and the file descriptions of several corresponding data files in the lower level, a corresponding sub-task (i.e., the sub-compression and merging task described in other embodiments) can be determined. Furthermore, based on the N+1 file identifiers corresponding to the sub-task, the N+1 data files involved in the sub-task can be determined. Therefore, in other embodiments, in conjunction with the "loading data files" related content in Figure 5b, it can be implemented by querying based on the N+1 file identifiers corresponding to the sub-compression and merging task. Furthermore, the attribution correspondence between the aforementioned data files may not be limited to being based on the hash bucket information of each data file recorded in the target metadata snapshot, or the relevant file attribution correspondence table may be pre-defined. By default, immutable data files on memory layer L1 in data storage node 14 may have an attribution correspondence with all data files on their adjacent lower layer (first persistent data layer L1).

[0190] Furthermore, before performing the above steps 2011–2013, the following steps may also be performed first:

[0191] 2010a. Check whether the latest metadata snapshot contains an immutable metadata file;

[0192] 2010b. If not, then steps 2011 to 2013 are triggered; if they exist, then based on the latest metadata snapshot, the compression and merging task related to the latest metadata snapshot continues to be executed.

[0193] Furthermore, the second data file mentioned in section 201 above refers to the data file within the storage disk (e.g., a hard disk) of data storage node 14. Also, in the scenario where the fanout value N is a power of 2, the last M*log2(N) bits of the hash value of the primary key of all key-value data stored in each data file within the storage disk of data storage node 14 are the same; M is a positive integer greater than or equal to 1, representing the level to which the corresponding data file belongs. In data storage node 14, the level of memory layer L0 can be numbered as 0, and correspondingly, the levels of multiple storage layers in the hard disk (such as the first persistent data layer L1 and the second persistent data layer L1 described in other embodiments, each layer having multiple data files) can be sequentially incremented starting from 1, meaning the level to which the data file in the storage disk belongs is a positive integer greater than or equal to 1. Moreover, the metadata of each data file within the storage includes hash bucket information, which indicates the last M*log2(N) bits of the hash value of the primary key of all key-value data in the data file. Since memory layer L0 is the highest layer, the data in the data files above it is compressed down to the lower layers, and there is no need to compress data into the data files above it. Therefore, there is no requirement for the last bit of the hash value in the data files above memory layer L0.

[0194] For example, assuming fanout = N = 4, then there are 4 data files (ordered data files SSTd) on the first persistent data layer L1 in data storage node 14, and the last two digits of the hash value corresponding to the key-value data in these 4 data files are (1*log2(2 2 The four data files are 00, 01, 10, and 11, and each of these four data files has four subordinate data files on the second persistent data layer L2 (i.e., there are 4*4=16 data files on the second persistent data layer L2). For example, taking the data file on the first persistent data layer L1 whose hash value ends in 01, the last four bits of the hash value of the key-value data in its four subordinate data files are (2*log2(2...). 2 The four data files (0001, 0101, 1001, and 1101) have the same last two bits in their hash values ​​(all 01, the same as the last two bits of the hash value of the upper-level data file to which they belong). See also the example described here. Figure 5b-2 Understand the relevant content described.

[0195] For example, suppose fanout = N = 8 (= 2 3 If there are 8 data files (ordered data files SSTd) on the first persistent data layer L1 of data storage node 14, and the last three digits of the hash value corresponding to the key-value data in these 8 data files are (1*log2(2)). 3The eight data files are 000, 001, 010, 011, 100, 101, 110, and 111. Each of these eight data files has eight subordinate data files on the second persistent data layer L2 (i.e., there are 8*8=64 data files on the second persistent data layer L2). For example, taking the data file on the first persistent data layer L1 whose hash value ends in 010, the last six digits of the hash value of the key-value data in its eight subordinate data files are (2*log2(2...). 3 The numbers 000010, 001010, 010010, 011010, 100010, 110010, and 111010 are respectively. The last three digits of the hash values ​​of the eight subordinate data files are the same (all are 010, which is the same as the last three digits of the hash values ​​of the upper-level data files to which they belong).

[0196] In the two examples above, in the storage disk, the hash value corresponding to the lower-level data file is lower than (M-1)*log2(2). 3 ) bits and the corresponding hash value of the upper-level data file to which it belongs, low M*log2(2 3 () The same position.

[0197] If the first key-value data is one of all key-value data in the first data file involved in the subtask, then:

[0198] The phrase "using consistent hashing to store the first key-value data into the corresponding N second data files" in the aforementioned 2012 may include:

[0199] 20121. Perform a hash calculation on the primary key of the first key-value data to obtain a hash value;

[0200] 20122. Based on the last (M-1)*log2(N)+1 to M*log2(N) bits of the hash value, determine the target second data file from the N second data files, and insert the first key-value data into the target second data file based on the hash value.

[0201] Continuing with the above assumptions, fanout = N = 8 (= 2) 3In the example described, there are 8 data files on the first persistent data layer L1. If the last three bits of the hash value of all key-value data in the third data file (which is the first data file) are 010, then when compressing and merging the first key-value data from one of the upper-layer data files into the corresponding data file on the lower layer (the second persistent data layer L2), the last six bits of the hash value corresponding to the first key-value data will be observed. If the last six bits of the hash value corresponding to the first key-value data are 100010, then the first key-value data will be inserted into the fifth data file among the 8 data files corresponding to the third data file in the lower layer (that is, the data file in the lower layer whose last six bits of the hash value are 100010). In fact, it is based on comparing the last 4th to 6th bits of the hash value to determine whether the first key-value data needs to be inserted into the fifth data file. In the example, M = 2 in the logarithmic function (log function) given in step 20122 above.

[0202] As described above, standby node 12 performs compaction offload on primary node 11. Compaction offload means offloading the compaction operation to standby node 12, that is, the compaction operation is performed by standby node 12, which reduces the load (such as CPU pressure) on primary node 11. This explains... Figure 3a The reason why the compression and merging module 121 on the backup node 12 is displayed in bright light and the compression and merging module on the master node 11 is displayed in gray is as follows.

[0203] There is an above combination Figures 5a to 5c As described, the data partitioning scheme of this application, which adopts a consistent hashing rehash scheme with a fixed data distribution parameter (fanout), has the following advantages:

[0204] 1) The number of files at each level in the LSM-tree is strictly determined, and the primary key exists only in a fixed position at each level. By optimizing read and search, the upper limit of the amplification can be strictly limited.

[0205] 2) It strictly limits the size of the data to be merged in the compaction operation, so that it can be split into several strictly independent subtasks, which facilitates parallelism (adapting to the high latency and high throughput characteristics of object storage under high concurrency) and allows for breakpoint continuation.

[0206] 3) Facilitates efficient parallel traversal and querying of data, strictly limits the number of merged iterators, and also facilitates parallel task orchestration to maximize cache utilization.

[0207] 4) Meta information is very efficient and simple. Compared with LSM-tree segmented by numerical range (the more nodes there are, the lower the search efficiency), the search complexity of consistent hash rehash sharding is O(1). Meta information is a full multi-way tree represented by an array. The search complexity of parent and child SST nodes in the meta information is O(1).

[0208] The above content mainly combines Figure 4a and Figure 4b The write functionality of master node 11 is introduced, and its combination with... Figures 5a to 5c The compression and merging function of the backup node 12 is introduced. Furthermore, in this application, the primary node 11 and the backup node 12 also have other functions. For example, the primary node 11 implements a data retrieval (query) function through its internal read module 112, and the backup node 12 implements an alignment function with the memory data state of the primary node 11 through its internal alignment module 122. The data retrieval function of the primary node 11 and the alignment function of the backup node 12 will be described in detail below.

[0209] Figure 4d-A The data retrieval process of master node 12 is illustrated. As also described in other embodiments of this application, data storage node 14 contains multiple data files for storing data, and these multiple data files have a hierarchical relationship. The number of data files in each layer and the hierarchical correspondence between the data files in each layer are all related to a pre-set data distribution parameter, fanout. Figure 4d-AAssuming fanout = 2, the data storage node 14 includes the currently active log data file on memory layer L0, two ordered data files SSTd1-1 and SSTd1-2 on the first persistent data layer L1, and four ordered data files SSTd2-11, SSTd2-12, SSTd2-21, and SSTd2-22 on the second persistent data layer L2. The active data file on memory layer L0 has a corresponding relationship with the two ordered data files SSTd1-1 and SSTd1-2 in the adjacent lower layer of the first persistent data layer. Similarly, the ordered data file SSTd1-1 on the first persistent data layer L1 has a corresponding relationship with the two ordered data files SSTd2-11 and SSTd2-12 on the adjacent lower layer of the second persistent data layer L2, and the ordered data file SSTd1-2 has a corresponding relationship with the two ordered data files SSTd2-21 and SSTd2-22 on the second persistent data layer L2. The attribution correspondence can be understood as the compression and merging relationship between data files at different levels. For example, when the size of the key-value data in the currently active log data file on memory layer L0 reaches a set threshold, according to its attribution correspondence, the key-value data in the currently active log data file is distributed, compressed, and merged into the lower-level ordered data files SSTd1-1 and SSTd1-2. See also... Figure 4d-A As shown, when master node 11 receives a query request, it can remotely query data (values) matching the primary key key in data storage node 14 based on the primary key key carried in the query request. In specific implementation, data storage node 14 supports the following two query methods:

[0210] 1) Point-by-point lookup method. Using the point-by-point lookup method, based on the known primary key, the data files in each layer of the data storage node 14 are traversed from top to bottom. For example, when performing a read query on the data storage node 14 based on the known primary key, the search is first performed in the currently active data file in its memory layer. If the data is not found in the currently active log data file, the search is then performed in the two ordered data files SSTd1-1 and SSTd1-2 on the lower first persistent data layer L1. If the data is not found in the two ordered data files SSTd1-1 and SSTd1-2, the search is further performed in the four data files SSTd2-11, SSTd2-12, SSTd2-21, and SSTd2-22 on the lower second persistent data layer L2. Since the data in the upper-level data files is always newer than the data in the lower-level data files, the search stops immediately after the first matching data is found, and the found data is returned. If it is known that no matching data can be found at the last level, it is assumed that no matching data for this primary key can be found in the storage engine, meaning that the primary key has not been stored in the storage engine.

[0211] 2) Full Traversal Method. The full traversal method uses consistent hashing and traverses the data files of each layer in data storage node 14 from bottom to top based on the known primary key. Because in data storage node 14, data written later is in the upper-level data file, and data written earlier is compressed and merged into the lower-level data file, the data in the upper-level data file is always newer than the data in the lower-level data file. Based on this, during the full traversal and searching layer by layer upwards, if a matching data is found in the upper level, it means that the data found in the lower level is old data, so the old data will be discarded. This process is repeated from bottom to top to achieve the traversal of all data files in data storage node 14.

[0212] Specifically, as exemplified, see see Figure 4d-AIn the example given, when traversing the data files of each layer in the data storage node 14 from bottom to top using the full traversal method, the four ordered data files SSTd2-11, SSTd2-12, SSTd2-21, and SSTd2-22 on the last layer, the second persistent data layer, are traversed first. After the second persistent data layer is traversed, the two ordered data files SSTd1-1 and SSTd1-2 on the first persistent data layer L2 are traversed. Finally, the currently active data file on the memory layer L0 is traversed. Each time a data file is traversed, for example, traversing the ordered data file SSTd2-11 on the second persistent data layer L2, based on the attribution and correspondence between data files described in the other embodiments above, it can be determined that the key-value hash range of the ordered data file SSTd2-11 must fall entirely within the key-value hash range of the adjacent upper-level ordered data file SSTd1-1. Therefore, whenever a matching data is found in the ordered data file SSTd2-11, the search is then performed in the upper-level ordered data file SSTd1-1. If a matching data is also found in the upper-level ordered data file SSTd1-1, it indicates that the data in the ordered data file SSTd2-11 is overwritten by the upper-level ordered data file SSTd1-1. Therefore, the data found in the lower-level ordered data file SSTd2-11 will not be returned but will be discarded. If no matching data is found in the upper-level ordered data file SSTd1-1, it means that the data found in the lower-level ordered data file SSTd2-11 has not been overwritten by new data, so it can be returned to the traversal results. Similarly, when traversing the ordered data file SSTd1-1, the search continues in its adjacent upper-level active log data file. Therefore, the full traversal query method is a bottom-up data query process. It is based on consistent hashing to ensure that the data in each level uniquely appears only in the upper-level ordered data file SST, thus enabling fast filtering (Bloom filter) during the traversal query.

[0213] However, considering that the data in the active data file on memory layer L0 of data storage node 14 is appended and unordered, the time complexity of searching for the existence of data in the active data file on memory layer L0 will be O(n). Since the ordered data file SST on the storage disk has been sorted by key, searching for data in the ordered data file SST can be done using binary search, with a time complexity of log(n). Clearly, searching for data in the data files contained in memory of data storage node 14 has a relatively high time complexity. Therefore, during read queries, a local in-memory data structure needs to be created for the data in memory of data storage node 14 to accelerate query performance. For more information on this issue, see [link to relevant documentation]. Figure 4a This application's solution constructs a log cache in the memory of the master node 11, comprising an invisible layer (delta layer) and a full layer. This log cache serves as a data retrieval acceleration module for data in the memory of the data storage node 14. For details regarding the specific implementation principles of data writing in the invisible layer (delta layer) and the full layer (full layer), please refer to the relevant content in the write operation flow described for the master node 11 in other embodiments; these details will not be elaborated here. Based on this, when the master node 11 receives a query request, it can first search locally (such as in the delta layer and full layer in memory). If the query is not found locally, it will then search in the data storage node 14.

[0214] This is for Figure 4a It should be noted that master node 11 is enabled by... Figure 4a The read thread shown in the figure implements this. Figure 4d-A The data retrieval process is illustrated. When the read thread is activated in response to a query request, it first checks the local log cache (specifically the delta and full layers) to find relevant data. If not found, it then checks the data file cache (SST cache). The SST cache is a multi-level cache, containing the master node's local memory, local storage disks (such as local SSDs), and data storage nodes (such as OSS), with increasing capacity to ensure read performance. Figure 4aThe initialization thread shown is an operation started during initialization, used to replay the log portion of data storage node 14. The log portion includes the active log file and may also contain immutable log data files. Replay involves reading all data from the beginning of the log portion and placing it in the log cache, which can be placed in the full layer, allowing the read thread to access the data in the log portion. For example, during the data state alignment process, standby node 12 can enable this initialization thread to replay the corresponding log portion in data storage node 14 according to the replay point, synchronizing it to the full layer storage space within it, thus catching up with the state of the primary node 12. This allows standby node 12 to quickly become the primary node and provide the corresponding services if the primary node 12 fails, without needing to spend time performing a replay operation. The commit point of the active log file can be used as the replay point. The specific implementation of the data state alignment operation process for backup node 12 will be described in detail below in conjunction with Figure 6.

[0215] There is an above combination Figure 4a and Figure 4d-A The data processing methods performed by master node 11 may also include the following steps, as described above:

[0216] 110. In response to a query request, retrieve data matching the query request from the local machine and the data storage node according to a preset query strategy.

[0217] Specifically, step 110 above is implemented as follows:

[0218] 1101. Receive a query request, wherein the query request carries a primary key;

[0219] 1102. Based on the primary key carried in the query request, query the value that matches the primary key from the local machine and the data storage node according to the preset query strategy.

[0220] The above query strategy includes: if no value matching the primary key is found locally, then a search is performed on the data storage node. The local storage may include the master node's memory, specifically, it may include the complete layer within the master node's memory; furthermore, the local storage may also include the master node's local storage disk (such as a local SSD), because: combined with... Figure 5cAs described, the local storage disk of the master node, such as a local SSD, may also contain data stored in data storage node 14. The search method from the data storage node includes a full traversal method; the full traversal method involves traversing the multiple data files with hierarchical relationships within the data storage node from bottom to top. If a matching value is found in a lower-level data file but not in the corresponding upper-level data file, the value found in the lower-level data file is returned; the data range of the lower-level data file is within the data range of the upper-level data file. The data range can refer to the hash range of the keys of the key-value data that are allowed to be stored in the corresponding data file. Furthermore, the search method from the data storage node may also include other methods, such as a point-and-click method. For detailed descriptions of the full traversal method and the point-and-click method, please refer to other embodiments. Figure 4d-A The relevant content is introduced.

[0221] Next, let's combine... Figure 4d-B To summarize and supplement the explanation of the data retrieval performed by master node 12, the specific points are as follows:

[0222] 1) The key-value index of this application is based on consistent hashing (rehash). The data range of a data file in the lower layer exists only in a data file in the upper layer. Therefore, parallel scanning can be performed, and the data file in the upper layer will be reused as much as possible during the parallel scanning task orchestration to complete a parallel fast traversal capability.

[0223] For example, see Figure 4d-BAssuming a top-down full traversal approach is used to query data, the data merging tree (SST tree) in data storage node 14 is composed of data files (ordered data files SST) at the first persistent layer L1 and the second persistent layer L2. When multiple scanning threads are enabled to scan the data merging tree (SST tree) simultaneously, since the data range of the four data files 21, 25, 29, and 33 at the bottom level each exists only in one data file 11 at the top level, parallel scanning tasks can be arranged for these four data files at the bottom level. Moreover, each of these arranged parallel scanning tasks reuses one data file 11 at the adjacent top level. Therefore, during the parallel scanning of the four data files 21, 25, 29, and 33 at the bottom level, if matching data is found at the top level, the matching data found at the bottom level will be filtered out. For example, if a matching data is found in data file 21, it will then search in the upper-level data file 11. If a matching data is also found in data file 11, it will then filter to find the matching data in the lower-level data file 21.

[0224] Furthermore, since the data within the data file of the Data Merge Tree (SST tree) is ordered, binary search can be used when traversing and searching the tree, effectively reducing the search time complexity. In practice, a binary search with a hint can be used to update the lower bound of the binary search, reducing the reading of invalid data.

[0225] When using binary search with hints, an upper and lower bound are typically determined first. Then, the median value is checked. If the median value is smaller than the search data, the lower bound is updated to this median value; conversely, if the median value is larger than the upper bound, the upper bound is updated to this median value. It's clear that with binary search with hints, the search data is traversed in ascending order each time. Therefore, the data searched in the next iteration might be larger than the data searched in the previous iteration. This allows for maintaining a dynamically recommended lower bound, avoiding a full-range binary search and reducing the reading of some invalid data.

[0226] 2) Because the data stored in in-memory data structures is often unordered, some preprocessing is performed on the log file in memory during queries. Specifically, this includes... Figure 4d-BThe system builds a Bloom filter on the log file and sorts the key-value data in the log file by key (sorted PKs, also known as sorting by primary key), thereby reducing the filtering pressure on the memory layer. The Bloom filter can quickly determine whether a corresponding primary key exists. The log file in memory contains the corresponding active data file and may also contain immutable data files.

[0227] Figures 6a to 6c The diagram illustrates the principle of data state alignment performed by the standby node 12. The main purpose of aligning the standby node 12 with the memory data state of the master node is to ensure that the standby node 12 can quickly switch over and take over read and write traffic to provide services as the master node in the event of a failure of the master node 11.

[0228] In this application, the backup node 12 has all the modules and functions of the master node 11. For example, the backup node 12 also has log caching, write modules, and read modules including delta and full layers, but it does not have read / write traffic. However, in practical applications, the master node 11 may become ineffective due to network outages, crashes, or other anomalies. In this case, the backup node 12 needs to be able to quickly switch over and take over read / write traffic to provide services as the master node. In order for the backup node 12 to switch over quickly, the backup node 12 needs to implement hot standby. To achieve hot standby, the backup node 12 needs to keep up with the key-value index progress of the master node 11 in real time. This can be understood as the backup node 12's key-value index needing to align the memory data structure with the latest key-value index state of the master node 11 represented by the metadata recorded in the metadata node 13. That is, the memory data state of the backup node 12 needs to be able to quickly align with the memory data state of the master node 11. Therefore, the backup node 12 needs to have data state alignment capabilities. In this application, the backup node 12 has an alignment module 122 (such as... Figure 3a (As shown) is used to achieve data state alignment capability.

[0229] Alignment module 122, also known as catch-up module, is implemented using the Catchup algorithm. Catchup means to catch up or catch up, and it is an algorithm similar to data synchronization that can keep track of the key-value index progress of master node 11 in real time to achieve hot backup.

[0230] Specifically, when implementing data alignment capability (Catchup capability) on backup node 12, the following points need to be implemented:

[0231] (1) It needs to be able to align to the latest data with minimal I / O (input / output) and CPU (processor) costs, regardless of the current key-value index's state. I / O cost refers to the resource consumption required for I / O operations (including CPU time, memory bandwidth, disk space, etc.). CPU cost refers to the processing resources and time required for the current operation.

[0232] (2) Its key-value index should be able to know whether it may be outdated in order to reduce the cost of additional verification.

[0233] (3) Its key-value index maintains the current write position of data (i.e., the active data file (activelog) that the key-value index considers to be lagging behind), and uses this as a checksum for subsequent writes.

[0234] It should be further explained here that the logical data description of the key-value index in this application is as follows: the key-value index of any data state can be described by a snapshot. A snapshot may include metadata of the currently active data file (active log) in the data storage node, metadata of each ordered data file (SST), and may also include metadata of the immutable data file (immutable log). Metadata includes things like file ID and file length (valid length). Only one immutable data file (immutable log) with a non-zero length is allowed in a snapshot. This restriction is to reduce processing logic and the number of log files (facilitating fast replay after restart). In other embodiments, during the read operation process of the master node 11 and the compression and merging operation process performed by the backup node 12, the read of the key-value index is equivalent to the metadata of the active data file (active log) plus a snapshot that does not contain metadata of the active data file (active log).

[0235] Based on the above points, the data state alignment operation process for backup node 12 is as follows:

[0236] 1) Determine the target state of data storage node 14

[0237] The target state is related to the state of the compression and merging operation performed by the backup node 12, and mainly includes the following two types:

[0238] ① The target state when the compression and merging operation is completed: active log + SST tree. That is, in this state, the data storage node 14 currently only contains the corresponding active log data file (active log) and the data merge count (SST tree). The SST tree contains multiple ordered data files SST with hierarchical relationships.

[0239] ② The target state when the compression and merging operation is not yet completed: active log + immutable log + SST tree. That is, in this state, the data storage node 14 currently has the corresponding active log data file, immutable data file, and data merging tree. The immutable data file is transformed from the previous active data file.

[0240] 2) Catchup Algorithm

[0241] In this application, during the data state alignment operation with the master node 11 using the Catchup algorithm, the backup node 12 uses the write position (also known as the data commit point) corresponding to the active data file (active log) in the data storage node 14 as the replay point. The write position is related to the file length of the active data file. For example, if the current file length of the active data file in the data storage node 14 is 100, then when the master node 12 appends the target data to be committed to the active data file, the corresponding write position is 100, thus appending the target data to the end of the active data file starting from this write position. Therefore, the "replay point" can also be understood as the file length of the active log data file (active log) in the data storage node 12.

[0242] Data in the active data files within data storage node 14 is appended to by the primary node 11. Simultaneously, the backup node 12 will trigger compaction operations at appropriate times to transform the currently active data files in data storage node 14 into immutable log files. For example, when target data to be committed is appended to the currently active data file in data storage node 14, increasing its length from 100 to 120 (exceeding the set file length threshold), the backup node 12 will trigger a compaction operation to transform the current active data file into an immutable data file, and simultaneously generate a new log file as the new active data file.

[0243] Based on the above, the standby node may see the following discrepancies between its own memory data status and the actual memory data status of the master node 11:

[0244] Scenario 1: The replay site is on the active log file.

[0245] See also Figure 6aAs shown, in this case, the primary node 11 has successfully committed the target data (delta log) to be submitted as described in the other embodiments above. That is, the target data has been appended to the current first active data file in the data storage node 14 and the metadata node 13 has also been successfully updated. At this time, the backup node 12 knows from the metadata recorded in the metadata node 13 (such as the metadata recorded in the current first active metadata file) that the write position has been delayed but is still in the first active data file (active log). At this time, the backup node 12 needs to catch up with the first active data file (active log) and replay the newly appended data content in the first active data file to achieve consistency with the memory data state of the primary node 11.

[0246] For example, after successfully submitting the target data, the master node 11 will replay the target data in its full-layer memory for storage. To keep up with the master node 11's memory data state, the backup node 12 will read the current first active data file from the data storage node 14, replay the target data located after the replay point in the first active data file, and synchronously store it in its full-layer storage space, thus catching up with the master node 12's state. The replay point is the write position of the target data in the first active data file, which can also be understood as the file length of the first active data file before the target data is written.

[0247] Scenario 2: The replay site is on an immutable data file (immutable log).

[0248] In this situation, because the primary node 11 performs operations quickly, it has already triggered a compression and merge operation, causing the current active data file (active log) in the data storage node to be transformed into an immutable data file (immutable log). Correspondingly, the standby node, by reading the metadata node 13, discovers that the replay point is now actually on the immutable data file. Therefore, to catch up with the memory data state of the primary node 11, it needs to replay the appended portion of the immutable data file (immutable log). After catching up with the immutable data file, the new active data file (active log) is then replayed, thus achieving the goal of catching up with the memory data state of the primary node.

[0249] For example, see Figure 6bAs shown, after the master node 11 successfully submits the target data, the file length of the current first active data file in the data storage node 14 exceeds the set length threshold, thereby triggering the standby node 12 to perform a compression and merge operation. This compression and merge operation is not yet completed. During the compression and merge operation performed by the standby node 12, the original first active data file is converted into an immutable log data file. At the same time, a second active data file is created in the data storage node 14 so that the master node 11 can continue to append data to the second active data file. Subsequently, when the data in the delta layer newly stored in its memory meets the submission requirements, the master node 11 will append the data in the delta layer to the current new second active data file in the data storage node 13 until the submission is completed, and then merge the data in the delta layer into the full layer in its memory. Therefore, in order to keep up with the memory data status of the master node 11, the standby node 12 will read the immutable data file in the data storage node 14 to replay the target data located after the replay point in the immutable data file and synchronize it into the full layer storage space therein. At the same time, it will also read the currently active second data file in the data storage node 14 to replay the data in the second active data file and synchronize it into its memory such as the delta layer.

[0250] Case 3: The data file containing the replay site is unknown.

[0251] In this scenario, standby node 12 is unaware of how far behind the master node 11 its memory data state lags. Therefore, it cannot replay data because it doesn't know where the replay point points. To catch up with the master node 11's memory data state, standby node 12 performs a full recovery operation. See [link to documentation]. Figure 6c As shown, the full recovery operation includes:

[0252] First, it will clear all its own memory data state. Specifically, the standby node 12 will cache the metadata of the data merging tree (SST tree) in the data storage node 14, that is, it will cache the metadata merging tree (SST tree meta) in the metadata node 13. In addition, the standby node 12 also has a log cache in its memory, which contains two log data, namely the log data of the delta layer and the full layer. The standby node 12 will first clear all the data, including the SST tree metadata cached in memory and the two log data in the log cache.

[0253] Then, it reads the metadata node 13 to load the latest STT tree metadata into its own memory; and it also reads the data storage node 14 to replay all the log files (including active log files and possibly immutable log files) currently in memory in the data storage node 14 to synchronize the data in the log files to its own memory. Specifically, it can synchronize the data to the full layer of the log cache in its own memory.

[0254] Therefore, the backup node 12 can catch up with the memory data state of the master node 11.

[0255] Based on the above, the data processing method executed by backup node 12 also includes the following steps:

[0256] 204. Based on the interaction with the metadata node, determine the replay point; wherein the replay point can indicate the write position corresponding to when the master node writes the target data to be submitted in its own memory to the data storage node;

[0257] 205. If the replay point is on the first active data file in the data storage node, then the key-value data located after the replay point in the first active data file is replayed to be synchronized to the local memory.

[0258] 206. If the replay point is on an immutable data file in the data storage node, then the key-value data located after the replay point in the immutable data file and the key-value data in the current second active data file in the data storage node are replayed to synchronize to local memory; wherein, the second active data file is converted from the previous first active data file in the data storage node;

[0259] 207. If the replay point is unknown, after clearing all data in the local memory, the metadata of each data file in the disk of the data storage node that is currently recorded in the metadata node is loaded into the local memory, and the data in the current log file in the data storage node is replayed to synchronize to the local memory; wherein, the log file contains the current first active data file, or the log file includes the current second active data file and an immutable data file converted from the previous first active data file.

[0260] In this application, the backup node 12 can quickly replay aligned data through interaction with the metadata node 13 to keep up with the memory data status of the master node 11 in real time. This makes it easy for the backup node 11 to efficiently take over traffic and switch to the master.

[0261] Furthermore, master node 11 can also perform other operations, such as a truncate operation. This truncate operation is mainly used to clear the contents of the active log file when the entire key-value index is deleted.

[0262] Figure 4e A schematic diagram of the truncate operation process is shown. (See also...) Figure 4e In metadata node 13, the currently active first metadata file points to the currently active first data file in data storage node 14. In other words, the currently active first metadata file records the metadata of the first active data file. When all data in the entire storage engine needs to be actively cleared, the master node 11, during the truncate operation, will delete all data in the data merge tree (SST tree) of data storage node 14 and simultaneously reposition the currently active first metadata file in metadata node 11 to point to the new active data file in data storage node 14. Specifically, firstly, the old data in the memory of the data storage node is archived or made unusable (e.g., unreadable). Specifically, the first active data file is locked. For example, a selective update statement (select for update) can be used to convert the first active data file into an immutable data file for archiving. At the same time, a new empty active data file is generated in data storage node 13 for subsequent writing. Assuming that the name of the generated new empty active data file is def, the metadata recorded in the current first active metadata file in metadata node 11 is updated according to the metadata of the new empty active data file. This updates the file name to def, the file length (valid length valid_length), the total number of lines, the latest update time, etc., to 0. This makes the current first active metadata file in metadata node 11 point to the new empty active data file in data storage node 14.

[0263] Based on the above description, the data storage node 14, metadata node 13, master node 11, and backup node 12 included in the data storage system provided in one embodiment of this application each have the following functions:

[0264] Data storage node 14 includes multiple data files with a hierarchical relationship; the multiple data files include a first active data file and multiple ordered data files;

[0265] Meta-information node 13 is used to record the metadata of each data file in the plurality of data files; wherein, the currently active first metadata file in the metadata node is used to record the metadata of the first active data file;

[0266] Master node 11, whose memory contains an invisible layer and a complete layer; the master node is used to implement the data processing method steps that the master node described in other embodiments of this application can execute.

[0267] Backup node 12 is used to implement the data processing method steps that the backup node described in other embodiments of this application can perform.

[0268] The data storage system provided in this embodiment can be a key-value storage engine, specifically, for example, a primary key index storage engine in a columnar storage engine. For detailed descriptions of the functional implementation of the primary node 11, backup node 12, data storage node 14, and metadata node 13, please refer to the relevant content in other embodiments.

[0269] This application also provides a primary key index data processing apparatus, which is deployed in... Figure 3a The data processing system is illustrated on the master node 11. Specifically, a portion of the components can be deployed in the write module 111 within the master node 11, and another portion can be deployed in the read module 112. The data processing system also includes a backup node, a metadata node, and a data storage node. The metadata node stores metadata corresponding to each data file in the data storage node. The metadata includes available metadata that ensures the atomicity of data writing. Furthermore, when the master node fails, the backup node switches to become the master node.

[0270] like Figure 9 As shown, the data processing device includes:

[0271] Reading module 31 is used to read the first active metadata file in the metadata node, wherein the first active metadata file contains the metadata of the first active data file in the data storage node;

[0272] The writing module 32 is used to append the target data to be submitted to the active data file based on the metadata of the first active data file; wherein, the append write operation log for the target data includes: write position and write data information;

[0273] Verification module 33 is used to verify the metadata of the first active metadata file record based on the write position;

[0274] The update module 34 is used to update part of the metadata recorded in the first active metadata file according to the written data information when the verification passes.

[0275] Optionally, the write location includes: the file identifier of the first active data file and the original data information in the first active data file before the append write operation; the write data information includes: the write time, the number of rows and bytes of the written data.

[0276] Optionally, after the first active metadata file is successfully updated, the target data is confirmed to have been successfully submitted.

[0277] The device further includes: a conversion module, configured to, when it is determined that the repair conditions are met before the target data is successfully submitted, convert the first active data file into an immutable data file, create a second active data file in the data storage node to continue providing write services, and copy the original data located before the write point in the immutable data file to the second active data file; update the file name recorded in the first active metadata file to the file name of the second active data file; wherein the second active data file has a different file name from the first active data file, and the met repair conditions include: triggering an interruption before the target data is successfully submitted and at least a portion of the target data has been written to the first active data file.

[0278] Optionally, the master node's memory includes an invisible layer and a complete layer; and the device further includes a response module and a generation module. The response module is configured to, in response to a write request, write the key-value data carried in the write request to the invisible layer. The generation module is configured to, when the submission conditions are met, generate the target data using all key-value data in the invisible layer. Furthermore, the write module is configured to, after the target data is successfully submitted, write the target data to the complete layer for retrieval.

[0279] Optionally, the device further includes: a clearing module, configured to clear key-value data stored in the complete layer that is identical to the key-value data in the first active data file when the key-value data in the first active data file of the upper layer is compressed and merged into the corresponding ordered data file of the adjacent lower layer in the data storage node for persistent storage; wherein the data storage node includes multiple data files with upper and lower layer relationships, the multiple data files including the current first active data file in memory and multi-level ordered data files in the storage disk.

[0280] Optionally, the above-mentioned response module is further configured to respond to a query request by obtaining the primary key carried in the query request; and searching for a value matching the primary key from the local storage and the data storage node according to a preset query strategy. The query strategy includes: if no value matching the primary key is found locally, then searching from the data storage node. The local storage includes the complete layer; the search method from the data storage node includes a full traversal method; the full traversal method is as follows: during the process of traversing the multiple data files with hierarchical relationships in the data storage node from bottom to top, if a matching value is found in a lower-level data file but not in the corresponding upper-level data file, then the value found in the lower-level data file is returned; the data range of the lower-level data file is within the data range of the upper-level data file.

[0281] For details regarding the specific implementation of the functions of each module in the device provided in this embodiment, please refer to the relevant content in other embodiments of this application. Furthermore, in addition to the functional steps described above, each module in the device provided in this embodiment can also implement other functional steps. For details regarding these other functional steps, please refer to the relevant content in other embodiments of this application.

[0282] This application also provides another data processing apparatus, which is deployed in... Figure 3a Specifically, on the backup node 11 shown, a portion can be deployed in the compression and merging module 121 of the backup node 12, and another portion can be deployed in the alignment module 122. For example... Figure 10 As shown, the data processing device includes: a determination module 41, an execution module 42, and an update module 43. The determination module 41, when the compression and merging trigger condition is met, determines the compression and merging task to be executed based on the metadata of each data file in the data storage node recorded by the metadata node. The compression and merging task includes at least one subtask that can be executed in parallel. The subtask involves a first data file at the upper level and N adjacent second data files at the lower level in the data storage node. The execution module 42, when executing the subtask, uses consistent hashing to distribute all key-value data in the first data file involved in the subtask to the corresponding N second data files for persistent storage. The update module 43, after the distribution is completed, updates the metadata of the N second data files recorded in the metadata node. N is the value of the consistent hashing data distribution parameter, which indicates that data in a data file at the upper level should be distributed to N data files at the lower level, and N is an integer greater than or equal to 2.

[0283] Optionally, N is a power of 2. Furthermore, the second data file is a data file within a storage disk in the data storage node, where the hash values ​​of the primary keys of all key-value pairs stored in each data file within the storage disk have the same last M*log2(N) bits; M is a positive integer greater than or equal to 1, representing the level to which the corresponding data file belongs; and the metadata of each data file within the storage disk includes hash bucket information, which indicates the last M*log2(N) bits of the hash value of the primary keys of all key-value pairs in the data file.

[0284] And, if the first key-value data is one of all key-value data in the first data file involved in the subtask, then: when the execution module 42 is used to store the first key-value data into the corresponding N second data files using consistent hashing, it can specifically be used to: perform a hash calculation on the primary key of the first key-value pair data to obtain a hash value; determine the target second data file from the N second data files based on the last (M-1)*log2(N)+1 to M*log2(N) bits of the hash value; and insert the first key-value data into the target second data file based on the hash value.

[0285] Optionally, the determining module 41 described above, when determining the merging task to be executed based on the metadata of each data file in the data storage node recorded in the metadata node, may specifically be used to: convert the current first active metadata file in the metadata node into an immutable metadata file; combine the immutable metadata file with the latest metadata snapshot in the metadata node to obtain a target metadata snapshot; and determine the compression and merging task to be executed based on the target metadata snapshot.

[0286] Optionally, the device further includes: a check trigger module, configured to: check whether an immutable metadata file exists in the latest metadata snapshot; if not, trigger the execution of combining the immutable metadata file with the latest metadata snapshot; if so, continue to execute the compression and merging task related to the latest metadata snapshot based on the latest metadata snapshot.

[0287] Optionally, the aforementioned determining module 41 can also be used to determine the replay point based on the interaction with the meta-information node; wherein the replay point can indicate the write position corresponding to when the master node writes the target data to be submitted in its own memory to the data storage node. The device may further include: a replay module, configured to: if the replay point is on the current first active data file in the data storage node, replay the key-value data in the first active data file located after the replay point to synchronize it to local memory; if the replay point is on an immutable data file in the data storage node, replay the key-value data in the immutable data file located after the replay point and the key-value data in the current second active data file in the data storage node to synchronize it to local memory; wherein, the second active data file is converted from the previous first active data file in the data storage node; if the replay point is unknown, after clearing all data in local memory, load the latest metadata of each data file in the disk of the data storage node currently recorded in the metadata node into local memory, and replay the data in the current log file in the data storage node to synchronize it to local memory; wherein, the log file contains the current first active data file, or the log file includes the current second active data file and an immutable data file converted from the previous first active data file.

[0288] For details regarding the specific implementation of the functions of each module in the device provided in this embodiment, please refer to the relevant content in other embodiments of this application. Furthermore, in addition to the functional steps described above, each module in the device provided in this embodiment can also implement other functional steps. For details regarding these other functional steps, please refer to the relevant content in other embodiments of this application.

[0289] Figure 11 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Figure 11 As shown, the electronic device includes a memory 51 and a processor 52. The memory 51 can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Specifically,

[0290] The aforementioned memory 51 is used to store programs;

[0291] The processor 52, coupled to the memory 51, is used to execute the program stored in the memory for the steps or functions in the methods provided in the embodiments of this application.

[0292] Furthermore, such as Figure 11 The electronic device also includes other components such as communication component 53 and power supply component 54. Figure 11 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 11 The components shown. The electronic device may be a device that includes the data processing system (KV storage engine) provided in other embodiments of this application.

[0293] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a computer, can implement the method steps provided in the above embodiments.

[0294] This application also provides a computer program product, including a computer program that, when executed by a processor, enables the processor to implement the method steps or functions provided in the above embodiments.

[0295] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0296] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for processing primary key index data, characterized in that, The master node is applied to the data processing system, which also includes backup nodes, metadata nodes, and data storage nodes. The metadata nodes are used to store metadata corresponding to each data file in the data storage nodes. The metadata includes available metadata that can guarantee the atomicity of data writing. And when the primary node fails, the backup node switches to become the primary node; The method includes: Read the first active metadata file in the metadata node, where the first active metadata file contains metadata of the first active data file in the data storage node; Based on the metadata of the first active data file, the target data to be submitted is appended to the first active data file. The append write operation log for the target data includes: write location and write data information. Based on the write location, verify the metadata recorded in the first active metadata file; When the verification passes, update part of the metadata recorded in the first active metadata file according to the written data information.

2. The method according to claim 1, characterized in that, The write site includes: the file identifier of the first active data file and the original data information in the first active data file before the append write operation; The written data information includes: write time, number of rows and bytes of written data.

3. The method according to claim 1 or 2, characterized in that, After the first active metadata file is successfully updated, the target data is confirmed to have been successfully submitted. Furthermore, the method further includes: When it is determined that the repair conditions are met before the target data is successfully submitted, the first active data file is converted into an immutable data file, and a second active data file is created in the data storage node to continue to provide write services, and the original data located before the write point in the immutable data file is copied to the second active data file; Update the file name recorded in the first active metadata file to the file name of the second active data file; The second active data file has a different file name from the first active data file, and the conditions for satisfying the repair include: an interruption was triggered before the target data was successfully submitted and at least a portion of the target data has been written to the first active data file.

4. The method according to claim 1 or 2, characterized in that, The master node's memory contains an invisible layer and a complete layer; and the method further includes: In response to a write request, the key-value data carried in the write request is written to the invisible layer; When the submission conditions are met, the target data is generated using all key-value data in the invisible layer. After the target data is successfully submitted, it is written to the complete layer for reading.

5. The method according to claim 4, characterized in that, Also includes: When the key-value data in the first active data file of the upper layer is compressed and merged into the corresponding ordered data file of the adjacent lower layer in the data storage node for persistent storage, the key-value data stored in the complete layer that is the same as the first active data file is cleared. The data storage node contains multiple data files with hierarchical relationships, including the first active data file currently in memory and multi-level ordered data files in the storage disk.

6. The method according to claim 5, characterized in that, Also includes: In response to a query request, obtain the primary key carried in the query request; According to the preset query strategy, the system searches for a value that matches the primary key in both the local machine and the data storage node; The query strategy includes: if no value matching the primary key can be found locally, then a search is performed from the data storage node; The local area includes the complete layer; The search method from the data storage node includes a full traversal method; the full traversal method is as follows: during the process of traversing the multiple data files with hierarchical relationships in the data storage node from bottom to top, if a matching value is found in a lower-level data file and no matching value is found in the corresponding upper-level data file, then the value found in the lower-level data file is returned; the data range of the lower-level data file is within the data range of the upper-level data file.

7. A data processing method, characterized in that, A backup node is applied in a data processing system, which further includes metadata nodes, data storage nodes, and a master node; the method includes: When the compression and merging trigger condition is met, the compression and merging task to be executed is determined based on the metadata of each data file in the data storage node recorded by the metadata node; the compression and merging task includes at least one subtask that can be executed in parallel, and the subtask involves a first data file at the upper layer and N second data files at the adjacent lower layer in the data storage node; When executing the subtask, consistent hashing is used to distribute all key-value data in the first data file involved in the subtask to the corresponding N second data files for persistent storage; After the distribution is completed, update the metadata of the N second data files recorded in the metadata node; Wherein, N is the value of the data distribution parameter of the consistent hash, which is used to indicate that data in one data file of the upper layer is distributed to N data files of the lower layer, and N is an integer greater than or equal to 2.

8. The method according to claim 7, characterized in that, N is a power of 2; and the second data file is a data file in the storage disk of the data storage node, wherein the hash value of the primary key of all key-value data stored in each data file in the storage disk has the same last M*log2(N) bits; M is a positive integer greater than or equal to 1, which represents the level to which the corresponding data file belongs; Furthermore, the metadata of each data file in the storage disk includes hash bucket information, which can indicate the last M*log2(N) bits of the hash value of the primary key of all key-value data in the data file; And, if the first key-value data is one of all key-value data in the first data file involved in the subtask, then: Consistent hashing is used to store the first key-value data into the corresponding N second data files, including: Perform a hash calculation on the primary key of the first key-value pair data to obtain a hash value; Based on the last (M-1)*log2(N)+1 to M*log2(N) bits of the hash value, the target second data file is determined from the N second data files; Based on the hash value, the first key-value data is inserted into the target second data file.

9. The method according to claim 7, characterized in that, Based on the metadata of each data file in the data storage node recorded in the metadata node, determine the merging tasks to be performed, including: Convert the current first active metadata file in the metadata node into an immutable metadata file; The immutable metadata file is combined with the latest metadata snapshot in the metadata node to obtain the target metadata snapshot. Based on the target metadata snapshot, the compression and merging task that needs to be executed is determined.

10. The method according to claim 9, characterized in that, Also includes: Check whether the latest metadata snapshot contains an immutable metadata file; If it does not exist, then the process of combining the immutable metadata file with the latest metadata snapshot is triggered. If it exists, then based on the latest metadata snapshot, continue to execute the compression and merging task related to the latest metadata snapshot.

11. The method according to any one of claims 8 to 10, characterized in that, Also includes: Based on the interaction with the metadata node, the replay point is determined; wherein the replay point can indicate the write position corresponding to when the master node writes the target data to be submitted in its own memory to the data storage node; If the replay point is on the first active data file in the data storage node, then the key-value data located after the replay point in the first active data file is replayed to be synchronized to the local memory. If the replay point is on an immutable data file in the data storage node, then the key-value data in the immutable data file after the replay point and the key-value data in the current second active data file in the data storage node are replayed to synchronize to local memory; wherein, the second active data file is transformed from the previous first active data file in the data storage node; If the replay point is unknown, after clearing all data in the local memory, the metadata of each data file in the disk of the data storage node that is currently recorded in the metadata node is loaded into the local memory, and the data in the current log file in the data storage node is replayed to synchronize to the local memory; wherein, the log file contains the current first active data file, or the log file includes the current second active data file and an immutable data file transformed from the previous first active data file.

12. A data processing system, characterized in that, include: Data storage nodes include multiple data files with hierarchical relationships; The plurality of data files includes a first active data file and a plurality of ordered data files; Meta-information nodes are used to record the metadata of each data file in the plurality of data files; wherein, the currently active first metadata file in the metadata node is used to record the metadata of the first active data file; A master node, whose memory contains an invisible layer and a complete layer; the master node is used to implement the data processing method according to any one of claims 1 to 6 above. Backup nodes are used to implement the data processing method according to any one of claims 7 to 11.

13. An electronic device, characterized in that, include: Memory and processor; among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps of the data processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program; when the computer program is executed by a computer, it can implement the steps of the data processing method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program that, when executed by a processor, causes the processor to perform the steps of the data processing method according to any one of claims 1 to 11.

Images