Graph data scheduling method for heterogeneous database cluster

Abstract

The invention relates to a graph data scheduling method for a heterogeneous database cluster, and belongs to the technical field of data replication, distribution or synchronization in a distributed database. The method comprises the following steps: firstly, constructing a database cluster consisting of disk type and memory type storage nodes, and determining how to construct a chained copy structure through the storage nodes according to the delay sensitivity of graph data when the graph data is stored; when the graph data is modified, logs of graph data modification content are generated and sequentially transmitted to the next node along the chain type copying structure; when a copy of the graph data needs to be added, selecting storage nodes which are not on the chain structure to store the added copy according to the characteristics of the graph data, and selecting a position for accessing the chain structure according to the load capacity; and finally, when an access request is processed, selecting a proper storage node to respond to the access according to the characteristics of the graph data. According to the method, the flexibility and efficiency of storing and accessing the graph data in the heterogeneous database cluster are improved.

Description

Technical Field

[0001] The present invention relates to a graph data scheduling method for a heterogeneous database cluster, and belongs to the technical field of data replication, distribution or synchronization (G06F16 / 27) in a distributed database. Background Art

[0002] With the widespread application of network technology, the cross-regional development of many enterprises has ushered in unprecedented growth. In the information age, key business data distributed everywhere has become the core asset of enterprises. With the continuous growth of storage capacity and the sharp increase in user access, the flexibility of traditional centralized database server management can no longer meet the needs of modern applications.

[0003] To solve this problem, database clusters came into being. A cluster is a group of independent computer systems that run the same set of applications in a collaborative way, providing a unified system image for clients and applications. A database cluster brings together multiple database servers and has functions such as distributed operation, data backup, and data disaster recovery, thereby providing reliable services for users and various applications. In order to better adapt to the diverse workloads of clients, storage nodes of different architecture types need to be used in database clusters. Due to the characteristics of different architecture types, specific types of storage nodes can better adapt to specific types of workloads. However, as the number of requests increases, storage and access often cause delays, which affects the system's response speed and user experience.

[0004] Therefore, there is an urgent need for a graph data scheduling method for heterogeneous database clusters to enable storage nodes of different architecture types to cooperate efficiently and improve the efficiency of storage and access. Summary of the invention

[0005] The technical problem to be solved by the present invention is: how to improve the efficiency of storing and accessing graph data in a heterogeneous database cluster.

[0006] The technical solution proposed by the present invention to solve the above technical problems is: a graph data scheduling method for a heterogeneous database cluster, comprising the following steps: Step 1: Build a database cluster consisting of storage nodes; The storage nodes include two types of architectures: disk-type storage nodes and memory-type storage nodes; Step 2: When a storage request for graph data is received, the graph data is copied to generate a copy of the graph data; If the delay sensitivity of the graph data is high, select at least one memory storage node and at least one disk storage node, and use the memory storage node with the lowest load as the first head node to build a first chain replication structure, and write the copy of the graph data into the first head node; If the delay sensitivity of the graph data is low, at least two disk-type storage nodes are selected, and the disk-type storage node with the lowest load is used as the second head node to build a second chain replication structure, and the copy of the graph data is written into the second head node; Step 2.1: The first head node or the second head node finally obtained in step 2 is used as the standard head node, and the first chain replication structure or the second chain replication structure finally obtained is used as the standard chain replication structure; Step 2.2: Copy the graph data copy in the standard head node to the next storage node in sequence along the standard chain replication structure until the copy is stopped at the last storage node; Step 3: when modifying the graph data, generating a log containing the graph data modification content, wherein the log adopts an intermediate format, and the intermediate format is different from the format of the copy of the graph data in the storage node; The log is copied into two copies, and the intermediate format of one of the copied logs is converted into the same format as the copy of the graph data in the standard head node, and the log after the format conversion is written into the standard head node; the other copied log is transferred to the next storage node along the standard chain replication structure and copied into two more copies, and one of the two re-copied logs is converted into the same format as the copy of the graph data in the next storage node, and the re-copied log after the format conversion is written into the next storage node, and the remaining re-copied log continues to be transferred along the standard chain replication structure, and so on, until the log reaches the last storage node and stops copying and transferring; Step 4: When the number of copies of the graph data needs to be increased, if the delay sensitivity of the graph data is high, a memory-type storage node that does not belong to any of the standard chain replication structures is selected as the first node to be added; Copying a copy of the graph data in any memory storage node on the standard chain replication structure to the first node to be added; Step 4.1: After the replication is completed, if the load of the first node to be added is lower than or equal to the load of the standard head node, the first node to be added is connected before the standard head node; If the load of the first node to be added is higher than the load of the standard head node, the first node to be added is connected to any position after the standard head node in the standard chain replication structure; Step 4.2: If the delay sensitivity of the graph data is low, select any disk-type storage node that does not belong to the standard chain replication structure as the second node to be added; Copying the graph data copy in any disk-type storage node on the standard chain replication structure to the second node to be added; Step 4.3: After the replication is completed, if the load of the second node to be added is lower than or equal to the load of the standard head node, the second node to be added is connected before the standard head node; If the load of the second node to be added is higher than the load of the standard head node, the second node to be added is connected to any position after the standard head node in the standard chain replication structure; Step 5: When a request to access graph data is received; Step 5.1: If the latency sensitivity of the graph data to be accessed is high, select the storage node with the lowest load among the memory-type storage nodes storing the graph data copies for access; If the loads of multiple memory-type storage nodes are all the lowest, then among the memory-type storage nodes with the lowest loads, the memory-type storage node with the lowest network delay to the client that issued the access request is selected for access; Step 5.2: If the delay sensitivity of the graph data is low, select a storage node with the lowest load among the disk-type storage nodes storing copies of the graph data for access; If the loads of multiple disk-type storage nodes are all the lowest values, then among the disk-type storage nodes with the lowest loads, the disk-type storage node with the lowest network delay to the client that issues the access request is selected for access.

[0007] The beneficial effects of the present invention are as follows: since a graph database cluster composed of heterogeneous storage nodes is constructed, it can adapt to the diverse workloads of clients; then, according to the degree of delay sensitivity of the graph data, storage nodes of a specific type of architecture are selected to build a chain replication structure, and then, while accurately adapting to the specific type of workload, the consistency of graph data storage is ensured, and then, combined with the intermediate format log delivery method, the consistency of graph data modification content is ensured; in addition, the access logic of the storage node when the graph data replica is added, and the subsequent access logic, are based on the delay-sensitive characteristics of the graph data, which effectively guarantees the load balancing of the graph data cluster, achieves performance optimization, and thus improves the efficiency of storage and access of graph data in heterogeneous database clusters. DETAILED DESCRIPTION Example

[0008] The graph data scheduling method for a heterogeneous database cluster of this embodiment includes the following steps: Step 1: Build a database cluster consisting of storage nodes; Storage nodes include two types of architectures: disk-based storage nodes and memory-based storage nodes; In this embodiment, the disk-type storage node usually uses a high-capacity mechanical hard disk or solid-state hard disk as the storage medium, which is suitable for graph data with low latency sensitivity, and can also be used to store data with large capacity, such as historical records, log files, etc. The disk-type storage node has a relatively low cost, but the read and write performance is relatively weak.

[0009] Memory-based storage nodes store data in memory and are suitable for graph data that is highly latency-sensitive. They can also be used to store data with smaller storage capacity, such as cache, real-time transaction data, etc. Memory-based storage nodes are relatively expensive, but have extremely high read and write performance.

[0010] Step 2: When a storage request for graph data is received, the graph data is copied to generate a copy of the graph data; In this embodiment, copying the graph data to generate a copy of the graph data can provide storage redundancy.

[0011] If the graph data is highly sensitive to latency, select at least one memory-type storage node and at least one disk-type storage node, and use the memory-type storage node with the lowest load as the first head node to build a first chain replication structure, and write the graph data copy into the first head node at the same time; If the delay sensitivity of the graph data is low, select at least two disk-type storage nodes, and use the disk-type storage node with the lowest load as the second head node to build a second chain replication structure, and write the copy of the graph data into the second head node; In this embodiment, a specific storage node is selected for storage according to the characteristics of the graph data, that is, the level of delay sensitivity. A specific type of storage node can better adapt to a specific type of workload; and the chain replication structure can well ensure the consistency of the graph data copies in the storage node; Step 2.1: The first head node or the second head node finally obtained in step 2 is used as the standard head node, and the first chain replication structure or the second chain replication structure finally obtained is used as the standard chain replication structure; Step 2.2: Copy the graph data copy in the standard head node to the next storage node in sequence along the standard chain replication structure until it stops copying to the last storage node.

[0012] Step 3: When modifying graph data, a log containing the modified content of the graph data is generated. The log adopts an intermediate format, which is different from the format of the copy of the graph data in the storage node; When the client modifies the graph data, the modified content of the graph data needs to be synchronized to each graph data replica in the database cluster. This embodiment is achieved through log transmission of the chain replication protocol. Considering that the storage nodes are heterogeneous, and the heterogeneous storage nodes have different graph data replica formats, different graph data replica formats are not universal in storage nodes of different architectures, so the log is converted into an intermediate format that can be understood by each storage node, such as a lightweight data exchange format (JSON) or a binary serialization format (MessagePack), which can avoid data replica format conversion between different storage nodes. The log is copied into two copies, and the intermediate format of one of the copied logs is converted into the same format as the copy of the graph data in the standard head node, and the log after the conversion is written to the standard head node; the other copied log is transferred to the next storage node along the standard chain replication structure and copied into two more copies, and one of the two re-copied logs is converted into the same format as the copy of the graph data in the next storage node, and the re-copied log after the conversion is written to the next storage node, and the remaining re-copied log continues to be transferred along the standard chain replication structure, and so on, until the log reaches the last storage node and stops copying and transferring.

[0013] Step 4: When the number of graph data replicas needs to be increased, if the graph data is highly sensitive to latency, select any memory-type storage node that does not belong to the standard chain replication structure as the first node to be added; Copy the graph data replica in any memory storage node on the standard chain replication structure to the first node to be added; Step 4.1: After the replication is completed, if the load of the first node to be added is lower than or equal to the load of the standard head node, the first node to be added is connected before the standard head node; If the load of the first node to be added is higher than the load of the standard head node, the first node to be added is connected to any position after the standard head node in the standard chain replication structure; Step 4.2: If the graph data has low latency sensitivity, select any disk-type storage node that is not on the standard chain replication structure as the second node to be added; Copy the graph data replica in any disk-type storage node on the standard chain replication structure to the second node to be added; Step 4.3: After the replication is completed, if the load of the second node to be added is lower than or equal to the load of the standard head node, the second node to be added is connected before the standard head node; If the load of the second node to be added is higher than the load of the standard head node, the second node to be added is connected to any position after the standard head node in the standard chain replication structure.

[0014] The storage node access logic of this embodiment ensures the load balancing of the database cluster and provides continuous storage persistence and stability.

[0015] Step 5: When a request to access graph data is received; Step 5.1: If the latency sensitivity of the graph data to be accessed is high, select the storage node with the lowest load among the memory-type storage nodes storing the graph data copies for access; If the loads of multiple memory-type storage nodes are all the lowest, then among the memory-type storage nodes with the lowest loads, the memory-type storage node with the lowest network delay to the client that issued the access request is selected for access; Step 5.2: If the delay sensitivity of the graph data is low, select the storage node with the lowest load among the disk-type storage nodes storing the graph data replicas for access; If the loads of multiple disk-type storage nodes are all the lowest values, then among the disk-type storage nodes with the lowest loads, the disk-type storage node with the lowest network delay to the client that issues the access request is selected for access.

[0016] This embodiment uses this access logic to help optimize the performance of the database cluster and thus improve access efficiency.

[0017] The above description is only a preferred embodiment of the present invention, but the present invention is not limited thereto. All equivalent replacements or equivalent changes of the concepts and technical solutions according to the present invention should be included in the protection scope of the present invention.

Claims

1. A graph data scheduling method for a heterogeneous database cluster, characterized in that The steps include: Step 1: Build a database cluster consisting of storage nodes; The storage nodes include two types of architectures: disk-type storage nodes and memory-type storage nodes; Step 2: when a storage request for graph data is received, the graph data is copied to generate a copy of the graph data; If the delay sensitivity of the graph data is high, select at least one memory storage node and at least one disk storage node, and use the memory storage node with the lowest load as the first head node to build a first chain replication structure, and write the copy of the graph data into the first head node; If the delay sensitivity of the graph data is low, at least two disk-type storage nodes are selected, and the disk-type storage node with the lowest load is used as the second head node to build a second chain replication structure, and the copy of the graph data is written into the second head node; Step 2.1: The first head node or the second head node finally obtained in step 2 is used as the standard head node, and the first chain replication structure or the second chain replication structure finally obtained is used as the standard chain replication structure; Step 2.2: Copy the graph data copy in the standard head node to the next storage node in sequence along the standard chain replication structure until the copy is stopped at the last storage node; Step 3: when modifying the graph data, generating a log containing the graph data modification content, wherein the log adopts an intermediate format, and the intermediate format is different from the format of the graph data copy in the storage node; Copy the log into two copies, convert the intermediate format of one of the two copies of the log into the same format as the copy of the graph data in the standard head node, and write the converted log into the standard head node; The other of the two replicated copies of the log is transferred to the next storage node along the standard chain replication structure and replicated into two more copies, one of the two replicated copies of the log is converted into the same format as the copy of the graph data in the next storage node, the replicated log after the conversion is written into the next storage node, and the remaining other replicated log continues to be transferred along the standard chain replication structure, and so on, until the log reaches the last storage node and stops being replicated and transferred; Step 4: When the number of copies of the graph data needs to be increased, if the delay sensitivity of the graph data is high, a memory-type storage node that does not belong to any of the standard chain replication structures is selected as the first node to be added; Copying a copy of the graph data in any memory storage node on the standard chain replication structure to the first node to be added; Step 4.1: After the replication is completed, if the load of the first node to be added is lower than or equal to the load of the standard head node, the first node to be added is connected before the standard head node; If the load of the first node to be added is higher than the load of the standard head node, the first node to be added is connected to any position after the standard head node in the standard chain replication structure; Step 4.2: If the delay sensitivity of the graph data is low, select any disk storage node that does not belong to the standard chain replication structure as the second node to be added; Copying the graph data copy in any disk-type storage node on the standard chain replication structure to the second node to be added; Step 4.3: After the replication is completed, if the load of the second node to be added is lower than or equal to the load of the standard head node, the second node to be added is connected before the standard head node; If the load of the second node to be added is higher than the load of the standard head node, the second node to be added is connected to any position after the standard head node in the standard chain replication structure; Step 5: When a request to access graph data is received; Step 5.1: If the latency sensitivity of the graph data to be accessed is high, select the storage node with the lowest load among the memory-type storage nodes storing the graph data copies for access; If the loads of multiple memory-type storage nodes are all the lowest, then among the memory-type storage nodes with the lowest loads, the memory-type storage node with the lowest network delay to the client that issued the access request is selected for access; Step 5.2: If the delay sensitivity of the graph data is low, select a storage node with the lowest load among the disk-type storage nodes storing copies of the graph data for access; If the loads of multiple disk-type storage nodes are all the lowest values, then among the disk-type storage nodes with the lowest loads, the disk-type storage node with the lowest network delay to the client that issues the access request is selected for access.