A heterogeneous engine distributed database architecture construction method of non-volatile storage

Abstract

The application is a kind of non-volatile storage heterogeneous engine distributed database architecture construction method and device, including the following steps: constructing a root node server; constructing a structured operation server for parsing structured query language; constructing a distributed heterogeneous storage engine server, which further includes: constructing at least one external storage engine node server, including: constructing at least one external storage engine node; constructing at least one non-volatile storage engine node server, including: constructing at least one non-volatile storage engine node; the root node server saves the meta information and data shard meta information of the external storage engine node and the non-volatile storage engine node; the root node server collects the performance statistical data of the external storage engine node server and the non-volatile storage engine node server; the root node server formulates a scheduling strategy for distributed data shards according to the performance statistical data and data shard meta information.

Description

Technical Field

[0001] This invention relates to the field of computer science and technology, and particularly to the field of distributed heterogeneous databases. Background Technology

[0002] With the rise of cloud platforms, distributed database systems are widely used in production and daily life. Many distributed database systems use LSM-Tree as the underlying data storage engine, storing relational data records in a key-value format in an LSM-Tree structure based on external storage (such as solid-state drives, SSDs).

[0003] However, distributed database systems exhibit diverse workloads, with different storage engine nodes often experiencing access hotspots and exhibiting varying access characteristics. In LSM-Tree-based storage engines, a large number of write operations frequently cause compaction, leading to significant fluctuations in system throughput and latency. Conversely, a large number of random read operations may trigger external storage access, resulting in performance degradation. On the other hand, next-generation non-volatile memory (NVM) is a replacement or supplement to existing DRAM (Dynamic Random Access Memory) main memory technology. Compared to DRAM, NVM can significantly increase memory capacity, achieving performance similar to DRAM, and providing persistent storage. Compared to SSDs, NVM offers significantly improved read and write performance.

[0004] Existing solutions primarily aim to completely replace existing LSM-Tree-based storage engines and build homogeneous distributed database systems based on the new engine. However, NVM is more expensive than SSDs. Therefore, how to better leverage NVM to support distributed database systems has become a common concern for both industry and academia. Summary of the Invention

[0005] To address the aforementioned problems, this invention discloses a method for constructing a heterogeneous engine distributed database architecture with non-volatile storage, comprising the following steps:

[0006] Steps to build a root node server;

[0007] The steps to build a structured operations server are used to parse structured query language;

[0008] The steps for building a distributed heterogeneous storage engine server include:

[0009] The steps to build at least one external storage engine node server include:

[0010] The step of constructing at least one external storage engine node;

[0011] The steps to build at least one non-volatile storage engine node server include:

[0012] The step of constructing at least one non-volatile storage engine node;

[0013] The root node server stores the metadata and data fragment metadata of the external storage engine node and the non-volatile storage engine node;

[0014] The root node server collects performance statistics of the external storage engine node server and the non-volatile storage engine node server.

[0015] The root node server formulates a scheduling strategy for distributed data sharding based on the performance statistics and data sharding metadata.

[0016] In one embodiment of the method described above, the step of constructing at least one external storage engine node further includes:

[0017] Step 311: Construct dynamic random access memory to store multiple data shards;

[0018] Step 312: Construct external storage to store multiple data shards;

[0019] Step 313: Build the external storage engine.

[0020] In one embodiment of the method described above, the external storage engine uses an LSM-Tree data storage engine to manage data storage.

[0021] In one embodiment of the method described above, the external storage device includes: a hard disk drive and / or a solid-state drive.

[0022] In one embodiment of the method described above, the step of the at least one non-volatile storage engine node further includes:

[0023] Step 321: Construct non-volatile main memory to store multiple data shards;

[0024] Step 322: Construct dynamic random access memory;

[0025] Step 323: Build a non-volatile storage engine.

[0026] In one embodiment of the method described above, the non-volatile main memory includes one or more of the following: phase change memory, impedance random access memory, carbon nanotube random access memory, and magnetic random access memory.

[0027] In one embodiment of the method described above, the step of constructing the non-volatile storage engine is performed by constructing an interface identical to that of the external storage engine.

[0028] In one embodiment of the method described above, the step of constructing the same interface between the non-volatile storage engine and the external storage engine further includes:

[0029] Build a key-value access interface;

[0030] Build a data backup interface;

[0031] Build a data sharding and splitting interface;

[0032] Build a data sharding migration interface.

[0033] In one embodiment of the method described above, the step of constructing the non-volatile memory engine further includes:

[0034] Step 331: Store all key-value data and logs in non-volatile main memory;

[0035] Step 332: Construct a dynamic random access memory index;

[0036] Step 333: Construct key-value pairs based on contiguous placement techniques.

[0037] In one embodiment of the method described above, the step of constructing a dynamic random access memory index further includes:

[0038] Step 3321: Construct the index structure required for searching key-value pairs;

[0039] Step 3322: Set the index structure in dynamic random access memory;

[0040] Step 3323: When recovering from a crash or shutdown of the dynamic random access memory, the index structure is reconstructed based on the key-value data in the non-volatile main memory.

[0041] In one embodiment of the method described above, the step of constructing key values ​​based on contiguous placement technology further includes:

[0042] Step 3331: The non-volatile storage engine uses a multi-version concurrency control mechanism to generate multiple versions of key-value data records for a single key;

[0043] Step 3332: When allocating non-volatile storage space for the multiple versions of key-value data records, allocate space that allows the multiple versions of key-value data records to be stored contiguously.

[0044] In one embodiment of the method described above, the root node server further includes storing the metadata of the external storage engine node and the non-volatile storage engine node:

[0045] The type of storage engine node.

[0046] In one embodiment of the method of the present invention, the step of the root node server formulating a scheduling strategy for distributed data sharding based on the performance statistics and data sharding metadata further includes:

[0047] Step 61: Establish performance models based on external storage engines and non-volatile storage engines;

[0048] Step 62: Perform performance comparison scheduling strategy and performance incremental scheduling strategy based on the performance model.

[0049] In one embodiment of the method described above, the step based on the performance comparison scheduling strategy further includes:

[0050] Step 6211: Calculate the performance improvement for each data shard when migrating it to a node of another type of storage engine;

[0051] Step 6212: Sort the samples from highest to lowest performance improvement;

[0052] Step 6213: Based on the space capacity of the storage engine node, migrate the data shards with significant performance improvements to another type of storage engine.

[0053] In one embodiment of the method described above, the step based on the performance incremental scheduling strategy further includes:

[0054] Step 6221: For each data shard, calculate the performance increment for migrating it to each node of another storage engine type;

[0055] Step 6222: Using a greedy algorithm, iteratively calculate the data shards to determine the data shard with the largest performance increment and the target storage engine node for migration, thereby obtaining a scheduling scheme for each data shard.

[0056] In one embodiment of the method of the present invention, the step of the root node server formulating a scheduling strategy for distributed data sharding based on the performance statistics and data sharding metadata further includes:

[0057] Step 6231: Multiple copies can be generated from the data shards;

[0058] Step 6232: Maintain consistency among multiple replicas using the Raft protocol;

[0059] Step 6233: Store multiple copies of the data shards on multiple storage engine nodes of the same type.

[0060] This invention also discloses a non-volatile storage heterogeneous engine distributed database architecture construction apparatus for implementing the steps of any of the above methods, including:

[0061] Root node server building module: Used to build the root node server;

[0062] Structured Operations Server Building Block: Used to build structured operations servers;

[0063] Distributed Heterogeneous Storage Engine Server Building Module: Used to build distributed heterogeneous storage engine servers, which includes:

[0064] External Storage Engine Node Server Building Module: Used to build at least one external storage engine node server, including:

[0065] External storage engine node building module: used to build at least one external storage engine node;

[0066] Non-volatile storage engine node server building block: Used to build at least one non-volatile storage engine node server, including:

[0067] Non-volatile storage engine node building module: Used to build at least one non-volatile storage engine node;

[0068] The root node server stores the metadata and data fragment metadata of the external storage engine node and the non-volatile storage engine node;

[0069] The root node server collects performance statistics of the external storage engine node server and the non-volatile storage engine node server.

[0070] The root node server formulates a scheduling strategy for distributed data sharding based on the performance statistics and data sharding metadata.

[0071] The present invention also discloses a non-volatile storage heterogeneous engine distributed database architecture, which is constructed using the above-described apparatus.

[0072] The present invention also discloses a non-volatile storage heterogeneous engine distributed database, which is built using the above-described architecture.

[0073] The present invention also provides a non-volatile memory engine, which is constructed using the steps of the above-described non-volatile memory engine method.

[0074] The present invention also discloses a storage medium for storing a computer control program, the computer control program being used to execute the steps of any of the above methods.

[0075] This invention utilizes an NVM server and an existing SSD server to form a cluster. The NVM-optimized storage engine runs on the NVM server, while the original LSM-Tree storage engine runs on the SSD server, forming a heterogeneous distributed database system to reduce costs and efficiently support various types and characteristics of application workloads. Attached Figure Description

[0076] Figure 1 This is a schematic diagram illustrating the steps of a method for constructing a heterogeneous engine distributed database architecture based on non-volatile storage in one embodiment of the present invention.

[0077] Figure 2 This is a block diagram illustrating the construction of a non-volatile storage engine node in one embodiment of the present invention.

[0078] Figure 3 This is a schematic diagram illustrating, in one embodiment of the invention, the key value and other versions of the key value being continuously stored in non-volatile main memory.

[0079] Figure 4 This is a schematic diagram illustrating data fragments and multiple copies of data fragments stored in non-volatile main memory in one embodiment of the invention.

[0080] Figure 5 This is a schematic diagram illustrating data fragments and multiple copies of data fragments stored in external storage in one embodiment of the invention.

[0081] Figure 6 This invention relates to an apparatus for constructing a heterogeneous engine-based distributed database architecture based on non-volatile storage, as described in one embodiment of the invention.

[0082] Figure 7 This is a heterogeneous engine distributed database architecture based on non-volatile storage in one embodiment of the present invention.

[0083] In the attached figures, the following labels are used:

[0084] 1: Root node server

[0085] 2: Structured Operation Server

[0086] 3: Distributed heterogeneous storage engine server

[0087] 4: External storage engine node server

[0088] 5: Non-volatile storage engine node server

[0089] 6: External storage engine node

[0090] 7: Non-volatile storage engine node

[0091] 10: A non-volatile storage heterogeneous engine distributed database architecture construction device

[0092] 11: Root Node Server Building Module

[0093] 12: Structured Operation Server Building Module

[0094] 13: Distributed Heterogeneous Storage Engine Server Building Module

[0095] 14: External Storage Engine Node Server Building Module

[0096] 15: External Storage Engine Node Building Module

[0097] 16: Non-volatile storage engine node server building block

[0098] 17: Non-volatile storage engine node building blocks

[0099] 20: A heterogeneous engine distributed database architecture with non-volatile storage

[0100] 61, 71: Dynamic Random Access Memory

[0101] 62: External Storage Engine

[0102] 63: External storage

[0103] 631, 731: Data storage space

[0104] 6313, 7313: Data Sharding

[0105] 6314, 7314: Data sharded replicas

[0106] 711: Based on Dynamic Random Access Memory Index

[0107] 7111: Hash Index

[0108] 7112: Skip list

[0109] 72: Non-volatile memory engine

[0110] 73: Non-volatile main memory

[0111] 7311: Key Value

[0112] 7312: Other versions of key-value pairs

[0113] 732: Log

[0114] 733: Lightweight Non-volatile Memory Space Allocator Detailed Implementation

[0115] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that references to "an embodiment," "embodiment," "example embodiment," etc., in the specification refer to the described embodiment including specific features, structures, or characteristics, but not necessarily including these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge scope of those skilled in the art.

[0116] The specification and subsequent claims use certain terms to refer to specific modules, components, or parts. Those skilled in the art will understand that users or manufacturers may use different names or terms to refer to the same module, component, or part. This specification and subsequent claims do not distinguish modules, components, or parts by differences in name, but rather by differences in function. The terms "comprising" and "including" used throughout the specification and subsequent claims are open-ended and should be interpreted as "including but not limited to." Furthermore, the term "connection" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connections via other means.

[0117] Furthermore, in the following description and claims, numerous terms will be referenced, which should be defined as having the following meanings. The singular forms “a” and “the” include plural referents, unless the context clearly specifies otherwise. “Optional” or “optionally” indicates that an event or situation subsequently described may or may not occur, and the description includes both the scenario where the event occurs and the scenario where the event does not occur.

[0118] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating the steps of a method for constructing a heterogeneous engine distributed database architecture based on non-volatile storage in one embodiment of the present invention.

[0119] This invention discloses a method for constructing a heterogeneous engine distributed database architecture with non-volatile storage, comprising the following steps:

[0120] Step S1: Build root node server 1;

[0121] Step S2: Construct a structured operations server 2 to parse structured query language;

[0122] Step S3: Construct distributed heterogeneous storage engine server 3, which includes:

[0123] Construct at least one external storage engine node server 4, including:

[0124] Build at least one external storage engine node 6;

[0125] Construct at least one non-volatile storage engine node server 5, including:

[0126] Construct at least one non-volatile storage engine node 7;

[0127] Step S4: The root node server 1 stores the metadata and data fragment metadata of the external storage engine node 6 and the non-volatile storage engine node 7;

[0128] Step S5: The root node server 1 collects performance statistics data of the external storage engine node server 4 and the non-volatile storage engine node server 5.

[0129] Step S6: Root node server 1 formulates a scheduling strategy for distributed data sharding based on the performance statistics and data sharding metadata.

[0130] In one embodiment of the method described above, the step of constructing at least one external storage engine node further includes:

[0131] Step 311: Construct dynamic random access memory 61 to store multiple data fragments;

[0132] Step 312: Construct external storage 63 to store multiple data fragments 6313;

[0133] Step 313: Build the external storage engine 62.

[0134] In one embodiment of the method described above, the external storage engine 62 uses an LSM-Tree data storage engine to manage data storage.

[0135] In one embodiment of the method described above, the external storage device includes: a hard disk drive and / or a solid-state drive.

[0136] In one embodiment of the method described above, the step of the at least one non-volatile storage engine node 7 further includes:

[0137] Step 321: Construct non-volatile main memory 73 to store multiple data shards 7313;

[0138] Step 322: Construct dynamic random access memory 71;

[0139] Step 323: Build a non-volatile storage engine 72.

[0140] In one embodiment of the method described above, the non-volatile main memory 73 includes one or more of the following: phase change memory, impedance random access memory, carbon nanotube random access memory, and magnetic random access memory.

[0141] Please refer to Figure 2 , Figure 2 This is a block diagram illustrating the construction of a non-volatile storage engine node in one embodiment of the present invention.

[0142] In one embodiment of the method described above, the step of constructing the non-volatile storage engine 72 is performed by constructing the same interface as the external storage engine 62.

[0143] In one embodiment of the method described above, the step of constructing the same interface between the non-volatile memory engine 72 and the external memory engine 62 further includes:

[0144] Build a key-value access interface;

[0145] Build a data backup interface;

[0146] Build a data sharding and splitting interface;

[0147] Build a data sharding migration interface.

[0148] In one embodiment of the method described above, the step of constructing the non-volatile memory engine 72 further includes:

[0149] Step 331: Store all key-value data and logs 732 in non-volatile main memory 73;

[0150] Step 332: Construct a dynamic random access memory index 711;

[0151] Step 333: Construct key-value pairs based on contiguous placement techniques.

[0152] In one embodiment of the method described above, the step of constructing the dynamic random access memory index 711 further includes:

[0153] Step 3321: Construct the index structure required for searching key-value pairs, including the hash index 7111 and the skip list 7112;

[0154] Step 3322: Set the index structure in the dynamic random access memory 71;

[0155] Step 3323: When recovering from a crash or shutdown of the dynamic random access memory 71, the index structure is reconstructed based on the key-value data in the non-volatile main memory 73.

[0156] Specifically, key-value (KV) data, i.e., data shards 7313 and logs 732, are stored in the data storage space 731 of the non-volatile main memory 73. This avoids the I / O overhead of the SSD and leverages the performance advantages of the NVM main memory. Simultaneously, the non-volatile storage engine 72 places the hash index and skip list index structure into the DRAM main memory to reduce the performance overhead of NVM writes and persistence. The non-volatile storage engine 72 employs contiguous placement of key-value pairs to reduce NVM random access to the versioned linked list. Furthermore, the non-volatile main memory includes a lightweight non-volatile memory space allocator 733 to reduce the performance overhead of NVM allocation and deallocation.

[0157] Please refer to Figure 3 , Figure 3 This is a schematic diagram illustrating, in one embodiment of the invention, the key value and other versions of the key value being continuously stored in non-volatile main memory.

[0158] In one embodiment of the method described above, the step of constructing key values ​​based on contiguous placement technology further includes:

[0159] Step 3331: The non-volatile storage engine 72 uses a multi-version concurrency control mechanism to generate multiple versions of key-value data records for a single key;

[0160] Step 3332: When allocating non-volatile storage space for the multiple versions of key-value data records, allocate space that allows the multiple versions of key-value data records to be stored contiguously.

[0161] like Figure 3 As shown, key value 7311 and three other versions of key value 7312 are consecutively placed in the data storage space 731 of non-volatile main memory 73. Those skilled in the art should understand that the above three other versions of key value are only used to explain the present invention, and the present invention is not limited thereto.

[0162] In one embodiment of the method described above, the root node server 1 further includes storing the metadata of the external storage engine node 6 and the non-volatile storage engine node 7, including:

[0163] The type of storage engine node.

[0164] In one embodiment of the method of the present invention, the step of the root node server 1 formulating a scheduling strategy for distributed data sharding based on the performance statistics and data sharding metadata further includes:

[0165] Step 61: Establish performance models based on external storage engine 62 and non-volatile storage engine 72;

[0166] Step 62: Perform performance comparison scheduling strategy and performance incremental scheduling strategy based on the performance model.

[0167] Specifically, root node server 1 collects operational statistics, establishes performance models based on external storage engine 62 and non-volatile storage engine 72, and then predicts performance based on the performance models. For example, the input parameters of the performance model include information such as the read / write ratio of data shards, key size, and parallelism, and the output of the performance model is the throughput of the data shards.

[0168] In one embodiment of the method described above, the step based on the performance comparison scheduling strategy further includes:

[0169] Step 6211: Calculate the performance improvement for each data shard when migrating it to a node of another type of storage engine;

[0170] Step 6212: Sort the samples from highest to lowest performance improvement;

[0171] Step 6213: Based on the space capacity of the storage engine node, migrate the data shards with significant performance improvements to another type of storage engine.

[0172] In one embodiment of the method described above, the step based on the performance incremental scheduling strategy further includes:

[0173] Step 6221: For each data shard, calculate the performance increment for migrating it to each node of another storage engine type;

[0174] Step 6222: Using a greedy algorithm, iteratively calculate the data shards to determine the data shard with the largest performance increment and the target storage engine node for migration, thereby obtaining a scheduling scheme for each data shard.

[0175] In one embodiment of the method of the present invention, the step of the root node server 1 formulating a scheduling strategy for distributed data sharding based on the performance statistics and data sharding metadata further includes:

[0176] Step 6231: Multiple copies can be generated from the data shards;

[0177] Step 6232: Maintain consistency among multiple replicas of the data shard using the Raft protocol;

[0178] Step 6233: Store multiple copies of the data shards on multiple storage engine nodes of the same type.

[0179] Specifically, storage engine node groups are formed through the Raft protocol. A group can contain, for example, three storage engine nodes of the same type, and the number of storage engine nodes contained in a group is not limited to this.

[0180] Please see Figure 4 and Figure 5 , Figure 4 This is a schematic diagram illustrating data fragments and multiple copies of data fragments stored in non-volatile main memory in one embodiment of the invention. Figure 5 This is a schematic diagram illustrating data fragmentation and multiple copies of data fragments stored in external storage in one embodiment of the invention.

[0181] Specifically, since root node server 1 maintains the metadata of all storage engine nodes and data shards, the storage engine node type, i.e., the NVM tag, is added to the metadata of the storage engine nodes to distinguish different types of storage engines, ensuring that data shards and their replicas reside on homogeneous storage engine nodes. For example... Figure 4 and Figure 5 As shown, this is a 4-replica storage, including data shards and three replicas of the data shards. Data shard 7313 and three data shard replicas 7314 are stored in the data storage space 731 of the non-volatile main memory 73 of multiple non-volatile storage engine nodes 7. Data shard 6313 and multiple data shard replicas 6314 are stored in the data storage space 631 of the external storage 63 of multiple external storage engine nodes 6 and / or dynamic random access memory 61. It should be understood that the number of data shards and the number of data shard replicas described above are only for explaining the present invention, and the present invention is not limited thereto.

[0182] Please refer to Figure 6 , Figure 6 This invention relates to an apparatus for constructing a heterogeneous engine-based distributed database architecture based on non-volatile storage, as described in one embodiment of the invention.

[0183] The present invention also discloses a non-volatile storage heterogeneous engine distributed database architecture construction device 10, used to implement the steps of the method described in any of the above claims, including:

[0184] Root node server building module 11: Used to build root node server 1;

[0185] Structured Operations Server Building Module 12: Used to build Structured Operations Server 2;

[0186] Distributed Heterogeneous Storage Engine Server Building Module 13: Used to build distributed heterogeneous storage engine server 3, which includes:

[0187] External Storage Engine Node Server Building Module 14: Used to build at least one external storage engine node server 4, including:

[0188] External storage engine node building module 15: used to build at least one external storage engine node 6;

[0189] Non-volatile storage engine node server building block 16: Used to build at least one non-volatile storage engine node server 5, including:

[0190] Non-volatile storage engine node building module 17: used to build at least one non-volatile storage engine node 7;

[0191] The root node server 1 stores the metadata and data fragment metadata of the external storage engine node 6 and the non-volatile storage engine node 7.

[0192] The root node server 1 collects performance statistics of the external storage engine node server 4 and the non-volatile storage engine node server 5.

[0193] Root node server 1 formulates a scheduling strategy for distributed data sharding based on the performance statistics and data sharding metadata.

[0194] Please refer to Figure 7 , Figure 7 This is a heterogeneous engine distributed database architecture based on non-volatile storage in one embodiment of the present invention.

[0195] The present invention also discloses a non-volatile storage heterogeneous engine distributed database architecture 20, which is constructed using the above-described apparatus.

[0196] like Figure 7As shown, the heterogeneous engine distributed database architecture for non-volatile storage disclosed in this invention includes a root node server 1, a structured operation server 2, and a distributed heterogeneous storage engine server 3. The distributed heterogeneous storage engine server 3 supports distributed storage and is backed up via the Raft protocol. It further includes at least one external storage engine node server 4 and at least one non-volatile storage engine node server 5. Each external storage engine node server 4 includes at least one external storage engine node 6, and each external storage engine node 6 further includes dynamic random access memory 61, an external storage engine 62, and external storage 63. Each non-volatile storage engine node server 5 includes at least one non-volatile storage engine node 7, and each at least one non-volatile storage engine node 7 further includes dynamic random access memory 71, a non-volatile storage engine 72, and non-volatile main memory 73. The structured operation server 2 is responsible for parsing the upper-level structured query language (SQL), converting the query statement into key-value operations, and sending them to the distributed heterogeneous storage engine server 3 for specific key-value operations. Root node server 1 stores the metadata of the storage engine of distributed heterogeneous storage engine server 3. After receiving a request from structured operation server 2, it assigns the task to distributed heterogeneous storage engine server 3.

[0197] Each storage engine node reports its performance statistics to root server 1, which implements various data sharding scheduling strategies. For heterogeneous engines, root server 1, based on performance modeling, implements scheduling strategies based on performance comparisons of heterogeneous storage engine types and scheduling strategies based on performance increments of heterogeneous storage engine nodes.

[0198] The present invention also discloses a non-volatile storage heterogeneous engine distributed database, which is built using the above-described architecture.

[0199] Experiments have shown that a distributed database using a heterogeneous engine architecture with non-volatile storage can outperform SSD-based LSM-Tree engine databases by up to 2.6 times under various load conditions. In addition to an existing all-SSD server distributed database, introducing a small number of NVM servers to build a non-volatile storage heterogeneous engine distributed database significantly improves overall performance. In mixed load tests, performance improvement reaches 2.5 times after data sharding, scheduling, and migration.

[0200] The present invention also provides a non-volatile memory engine 72, which is constructed using the steps of the method described above for a non-volatile memory engine 72.

[0201] The present invention also discloses a storage medium for storing a computer control program, the computer control program being used to execute the steps of any of the above methods.

[0202] The aforementioned computer program may be placed in random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0203] In summary, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can devise various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the protection scope of the patent application of the present invention.

Claims

1. A method for constructing a heterogeneous engine distributed database architecture with non-volatile storage, characterized in that, Includes the following steps: Steps to build a root node server; The steps to build a structured operations server for parsing structured query language; The steps for building a distributed heterogeneous storage engine server include: The steps to build at least one external storage engine node server include: The step of constructing at least one external storage engine node; The steps to build at least one non-volatile storage engine node server include: The step of constructing at least one non-volatile storage engine node, wherein the step of constructing the non-volatile storage engine node further includes: Construct non-volatile main memory to store multiple data shards; Construct dynamic random access memory; The step of constructing a non-volatile memory engine further includes: Store all key-value data and logs in non-volatile main memory; Construct a memory index based on dynamic random access; Construct key-value pairs based on contiguous placement techniques; The root node server stores the metadata and data fragment metadata of the external storage engine node and the non-volatile storage engine node; The root node server collects performance statistics of the external storage engine node server and the non-volatile storage engine node server. The root node server formulates a scheduling strategy for distributed data sharding based on the performance statistics and data sharding metadata, including: Establish performance models based on external storage engines and non-volatile storage engines; Based on the performance model, a performance-based scheduling strategy and a performance-based incremental scheduling strategy are implemented. The data is sharded to generate multiple copies; The multiple copies are stored in multiple storage engine nodes of the same type, wherein the type of the storage engine nodes of the same type is the external storage engine node or the non-volatile storage engine node. The metadata of the external storage engine node and the non-volatile storage engine node includes the type of the storage engine node.

2. The method as described in claim 1, characterized in that, The step of constructing at least one external storage engine node also includes: Step 311: Construct dynamic random access memory to store multiple data shards; Step 312: Construct external storage to store multiple data shards; Step 313: Build the external storage engine.

3. The method as described in claim 2, characterized in that, The external storage engine uses the LSM-Tree data storage engine to manage data storage.

4. The method as described in claim 2, characterized in that, The external storage includes: hard disk drives and / or solid-state drives.

5. The method as described in claim 1, characterized in that, The non-volatile main memory includes one or more of the following: phase change memory, impedance random access memory, carbon nanotube random access memory, and magnetic random access memory.

6. The method as described in claim 1, characterized in that, The step of constructing the non-volatile storage engine is performed by constructing the same interface as the external storage engine.

7. The method as described in claim 6, characterized in that, The step of constructing the same interface between the non-volatile storage engine and the external storage engine further includes: Build a key-value access interface; Build a data backup interface; Build a data sharding and splitting interface; Build a data sharding migration interface.

8. The method as described in claim 1, characterized in that, The step of constructing a dynamic random access memory index further includes: Step 3321: Construct the index structure required for searching key-value pairs; Step 3322: Set the index structure in dynamic random access memory; Step 3323: When recovering from a crash or shutdown of the dynamic random access memory, the index structure is reconstructed based on the key-value data in the non-volatile main memory.

9. The method as described in claim 8, characterized in that, The step of constructing key-value pairs based on contiguous placement technology further includes: Step 3331: The non-volatile storage engine uses a multi-version concurrency control mechanism to generate multiple versions of key-value data records for a single key; Step 3332: When allocating non-volatile storage space for the multiple versions of key-value data records, allocate space that allows the multiple versions of key-value data records to be stored contiguously.

10. The method as described in claim 1, characterized in that, The steps of the performance comparison-based scheduling strategy further include: Step 6211: Calculate the performance improvement for each data shard when migrating it to a node of another type of storage engine; Step 6212: Sort the samples from highest to lowest performance improvement; Step 6213: Based on the space capacity of the storage engine node, migrate the data shards with significant performance improvements to another type of storage engine.

11. The method as described in claim 1 or 10, characterized in that, The steps based on the performance incremental scheduling strategy further include: Step 6221: For each data shard, calculate the performance increment for migrating it to each node of another storage engine type; Step 6222: Using a greedy algorithm, iteratively calculate the data shards to determine the data shard with the largest performance increment and the target storage engine node for migration, thereby obtaining a scheduling scheme for each data shard.

12. The method as described in claim 1, characterized in that, The step of the root node server formulating a scheduling strategy for distributed data sharding based on the performance statistics and data sharding metadata further includes: Maintain consistency among multiple replicas using the Raft protocol.

13. A non-volatile storage heterogeneous engine distributed database architecture construction apparatus, used to implement the steps of the method as described in any one of claims 1 to 12, characterized in that, include: Root node server building module: Used to build the root node server; Structured Operations Server Building Block: Used to build structured operations servers; Distributed Heterogeneous Storage Engine Server Building Module: Used to build distributed heterogeneous storage engine servers, which includes: External Storage Engine Node Server Building Module: Used to build at least one external storage engine node server, including: External storage engine node building module: used to build at least one external storage engine node; Non-volatile storage engine node server building block: Used to build at least one non-volatile storage engine node server, including: Non-volatile storage engine node building module: Used to build at least one non-volatile storage engine node; The root node server stores the metadata and data fragment metadata of the external storage engine node and the non-volatile storage engine node; The root node server collects performance statistics of the external storage engine node server and the non-volatile storage engine node server. The root node server formulates a scheduling strategy for distributed data sharding based on the performance statistics and data sharding metadata.

14. A heterogeneous engine distributed database architecture with non-volatile storage, characterized in that, It is constructed using the apparatus as described in claim 13.

15. A heterogeneous engine distributed database with non-volatile storage, characterized in that, It is constructed using the architecture described in claim 14.

16. A non-volatile memory engine, characterized in that, It is constructed using the steps of the method as described in any one of claims 7 to 9.

17. A storage medium for storing a computer control program, characterized in that, The computer control program is used to perform the steps of the method as described in any one of claims 1 to 12.

Images