A backup method and apparatus based on a distributed graph database

Abstract

This invention relates to the field of databases, and in particular to a backup method and apparatus based on a distributed graph database. It mainly includes: generating a corresponding transaction accessible table according to a backup command, and broadcasting the transaction accessible table to all graph nodes that need to be backed up; each graph node generating a top-level backup task based on the transaction accessible table, and then deriving backup tasks for each level from the top-level backup task; each graph node asynchronously executing the bottom-level backup task, obtaining user data for backup through the bottom-level backup task, and backing up the user data on its own graph node. This invention can improve the rational utilization of computer resources, achieve the effect of fully parallel backup, and greatly improve the efficiency and performance of backup. It also reduces network overhead during recovery, significantly improving recovery efficiency.

Description

Technical Field

[0001] This invention relates to the field of databases, and in particular to a backup method and apparatus based on a distributed graph database. Background Technology

[0002] Graph databases are a type of non-relational database that primarily uses graph theory to store information about relationships between entities. Because they can store the unique relational value modeled in graphs, the market demand for graph databases is increasing, leading to the emergence of distributed graph databases. Data security is particularly important in distributed architectures, making data backup devices for distributed graph databases an indispensable component.

[0003] With the development of database systems, there are now many types of backups. However, based on backup characteristics such as the stored dataset, server status, and backup package objects, they can be roughly divided into the following types: full backup, incremental backup, differential backup, physical backup, logical backup, cold backup, and hot backup. During backup, it is necessary to verify the backup path and extract, process, and write metadata and user node edge data from the graph database. Each step requires multiple communications or data exports. Due to the complexity of the graph relationships stored in graph databases, the intricate relationships between nodes and edges, and the fact that data in each graph database node is related to other nodes, storing all data backups in a unified manner or using export methods would result in significant network and computer resource overhead.

[0004] Therefore, how to overcome the shortcomings of existing technologies and solve the problem of high resource consumption in existing graph database backup methods is a problem to be solved in this technical field. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention solves the problem of high resource consumption in the existing graph database backup method.

[0006] The embodiments of the present invention adopt the following technical solutions:

[0007] In a first aspect, the present invention provides a backup method based on a distributed graph database, specifically comprising: generating a corresponding transaction accessible table according to a backup command, and broadcasting the transaction accessible table to all graph nodes that need to be backed up; each graph node generating a top-level backup task on the graph node according to the transaction accessible table, and deriving backup tasks at each level layer by layer through the top-level backup task; each graph node asynchronously executing the bottom-level backup task, obtaining user data for backup through the bottom-level backup task, and backing up the user data on the graph node.

[0008] Preferably, the step of generating the corresponding transaction access table according to the backup command specifically includes: when performing a full backup, counting the transaction IDs related to the backup command, obtaining all visible transaction numbers based on the statistical results, and writing all visible transaction numbers into the transaction access table; when performing an incremental backup, obtaining all visible transaction numbers according to the backup command, obtaining the previous visible transaction number based on the transaction access table from the previous backup, comparing all visible transaction numbers with the previous visible transaction number to obtain the current visible transaction number, and writing the current visible transaction number into the transaction access table.

[0009] Preferably, each graph node generates a top-level backup task on that graph node based on the transaction-accessible table, specifically including: the graph node establishing a backup task for each graph, and using the set of backup tasks for all graphs as the top-level backup task; establishing a query engine in the top-level backup task to query the execution status of the backup task for each graph.

[0010] Preferably, the step of deriving backup tasks at each level from the top-level backup task specifically includes: obtaining the graph corresponding to the user data in the top-level backup task, and treating the backup task of each graph as a second-level backup task; obtaining the labels and relationships of the graph in each second-level backup task, and treating the backup task of the labels and relationships in each graph as a third-level backup task; obtaining the log sequence number of the data corresponding to the labels or relationships in each third-level backup task, and treating the backup task of each batch of log sequence numbers as a fourth-level backup task; organizing the backup tasks at each level according to a tree structure, and establishing a query engine in each level of backup task so that the upper-level backup task can query the status of the lower-level backup task.

[0011] Preferably, the step of obtaining user data for backup through the lowest-level backup task and backing up the user data on this library node specifically includes: obtaining each log sequence number in the lowest-level backup task, obtaining the corresponding user data according to the log sequence number; uniformly encoding the obtained user data according to a specified format, and writing the encoded binary stream into the corresponding backup file.

[0012] Preferably, the step of obtaining the corresponding user data according to the log sequence number further includes: each graph library node obtaining the point data and edge data in the user data stored on this graph library node according to the log sequence number; when the edge data that needs to query the endpoint data in the obtained edge data reaches a specified number, the unique identifier of the endpoint data to be queried is sent in batch to other graph library nodes to obtain the corresponding endpoint data.

[0013] Preferably, the step of uniformly encoding the obtained user data according to a specified format specifically includes: generating a corresponding operation code for each user data according to the backup method and the recovery operation corresponding to each user data, and using the operation code as the header field of the user data; using the key-value pair of each user data as the data field of the user data; and combining the header field and data field of each user data to form the encoding of the user data.

[0014] Preferably, the method further includes: generating corresponding metadata information according to the backup command, broadcasting the metadata information to all image library nodes that need to be backed up; each image library node generating a metadata backup task on the image library node according to the metadata information, and backing up the metadata through the metadata backup task.

[0015] In a second aspect, the present invention provides a backup device based on a distributed graph database, specifically comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by the at least one processor, and the instructions, after being executed by the processor, are used to complete the backup method based on a distributed graph database in the first aspect.

[0016] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the method described in the first aspect.

[0017] Fourthly, a chip is provided, comprising: a processor and an interface for calling and running a computer program stored in memory, performing the method as provided in the first aspect.

[0018] Fifthly, a computer program product containing instructions is provided that, when executed on a computer or processor, causes the computer or processor to perform the method provided in the first aspect.

[0019] Compared with existing technologies, the advantages of this invention are as follows: It divides the backup task of the entire graph database into multiple levels and distributes the backup task to each distributed graph database node. On each graph database node, the specific data backup process is executed asynchronously and in parallel, distributing each backup data packet across all graph database nodes. This method improves the rational utilization of computer resources, achieving a fully parallel backup effect and significantly improving backup efficiency and performance. It also reduces network overhead during recovery, greatly improving recovery efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0021] Figure 1 A flowchart of a backup method based on a distributed graph database is provided for an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram illustrating the backup initiation process in the method provided in this embodiment of the invention;

[0023] Figure 3 This is a schematic diagram illustrating the backup termination process in the method provided in this embodiment of the invention;

[0024] Figure 4 This is a schematic diagram of the task hierarchy in the method provided in the embodiments of the present invention;

[0025] Figure 5 A flowchart illustrating another backup method based on a distributed graph database provided in this embodiment of the invention;

[0026] Figure 6 A flowchart illustrating another backup method based on a distributed graph database provided in this embodiment of the invention;

[0027] Figure 7 This is a schematic diagram of the field structure for data encoding in the method provided in this embodiment of the invention;

[0028] Figure 8 This is a schematic diagram of the data storage structure in a specific scenario of the method provided in the embodiments of the present invention;

[0029] Figure 9 This invention provides a method for representing the transaction accessibility intent in a specific scenario.

[0030] Figure 10 This is a schematic diagram of the backup thread model in the method provided in the embodiments of the present invention;

[0031] Figure 11 This is a schematic diagram of the backup structure in a specific scenario of the method provided in the embodiments of the present invention;

[0032] Figure 12 This is a data illustration of a backup file in a specific scenario provided by the method of this invention.

[0033] Figure 13 This is a timing diagram of the backup process in the method provided in the embodiments of the present invention;

[0034] Figure 14 A schematic diagram of a backup device structure based on a distributed graph database provided in an embodiment of the present invention;

[0035] The reference numerals in the attached figures are as follows:

[0036] 11: Processor; 12: Memory. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0038] This invention is an architecture of a specific functional system. Therefore, the specific embodiments mainly describe the functional logic relationship of each structural module, and do not limit the specific software and hardware implementation methods.

[0039] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0040] The following is an explanation of some terms used in the embodiments of the present invention:

[0041] (1) Graph Store (GSTORE): A native graph distributed storage engine, referred to as "graph store" below.

[0042] (2) Graph Query Engine (Cypher Server): A graph query engine based on open cypher syntax rules, hereinafter referred to as the "query engine".

[0043] (3) Management node (Graph Cluster Manager, abbreviated as GCM): The management node of the native graph storage GSTORE engine.

[0044] (4) Distributed Library Node: A GSTORE node deployed in a distributed library. Multiple GSTORE nodes, one or more GSTORE nodes and one or more GCM nodes form a distributed cluster, which will be referred to as "library node" below.

[0045] (5) SEGMENT: A data sharding device inside GSTORE.

[0046] (6) VLOG: A data storage device inside GSTORE that fully stores metadata, user data or system data. A piece of data can be uniquely located by the log sequence number (LSN).

[0047] The main types of graph database backup methods include:

[0048] (1) Full backup: A complete backup of all data or applications at a specific point in time. This includes system metadata, user metadata, system data, and user data.

[0049] (2) Incremental backup: A backup is performed on data that has changed since the last full or incremental backup, including data that has changed due to addition, modification, deletion, etc.

[0050] (3) Differential backup: Performs a backup of data that has changed since the last full backup. It can be seen that the incremental backup function actually covers the differential backup function.

[0051] (4) Physical backup: Database file-level backup, the backup content is the database file on the operating system.

[0052] (5) Logical backup: Backup data in a certain logical form and store it in a backup set.

[0053] (6) Cold backup: also known as offline backup, refers to a complete backup of the database when the database is shut down and cannot be updated. During the backup process, users cannot access the system or the database normally.

[0054] (7) Hot Backup: Backup, also known as online backup or online backup, is a data backup performed while the system is running normally. During the backup process, users can access the system or database normally.

[0055] Example 1:

[0056] This embodiment provides a backup method based on a distributed graph database, which decomposes the backup task of the entire graph database into multiple levels of backup tasks on each graph database node, and backs up the data in the graph database to the current distributed graph database node in a unified data format.

[0057] like Figure 1 As shown in the figure, the specific steps of the backup method based on a distributed graph database provided in this embodiment of the invention are as follows.

[0058] Step 101: Generate the corresponding transaction access table according to the backup command, and broadcast the transaction access table to all graph nodes that need to be backed up.

[0059] When performing a backup, configuration attributes (config) such as the range of transaction IDs to be backed up, the backup start time, and the backup range can be specified through the backup command. In practice, the query engine can use the graph backup statement entered in the command line as the backup command to notify GCM to initiate a full or incremental backup.

[0060] In the method provided in this embodiment, the backup task is deployed separately on each map library node, which facilitates the individual recovery of each node. In a distributed map library, the GCM can know the data access status on each map library node, but each map library node cannot obtain the data access status on other map library nodes. Therefore, the GCM needs to generate a transaction accessibility table based on the data access status, and each map library node can determine the scope of data transactions that need to be backed up based on the transaction accessibility table. Furthermore, in order for each map library node to obtain the attributes of the backup task and the transaction accessibility table, the GCM needs to send the corresponding data to each map library node via broadcast.

[0061] Step 102: Each graph node generates a top-level backup task on the graph node based on the transaction-accessible table, and then generates backup tasks for each level through the top-level backup task.

[0062] After GCM notifies each GSTORE graph node of the config configuration and transaction accessibility table to initiate distributed backup, each GSTORE graph node will perform preparatory work, including verification, generation of metadata (graph, label, relationship) for this backup record, and calculation of graph objects for this backup.

[0063] In a typical backup process, only the graph library is backed up, and the regular tables in the database are not. This embodiment provides a backup method that supports two levels of data backup: graph-level backup and graph library-level backup. Graph-level backup backs up all user data and relationships between user data in each graph, i.e., backing up the points, edges, and metadata of a single graph. Graph library-level backup refers to backing up the entire graph library in the cluster, including regular tables such as tag tables and relationship tables within the graph library. This includes initiating a full backup, querying backup progress, initiating incremental backups, terminating ongoing backups, and deleting existing backups. Accordingly, in the method provided in this embodiment, each graph library node executes concurrently in a multi-level task queue. The multi-level tasks organize the graph library information stored on each node, as well as the hierarchical relationships of the tag tables and relationship tables of all graphs under each graph library, thus supporting both graph library-level and graph-level backups.

[0064] In practice, the graph-level backup task can be used as the top-level backup task, and then backup tasks at the graph level, tag relationship level, and data level can be derived sequentially downwards. The graph node creates a backup task for each graph, and the set of all graph backup tasks is used as the top-level backup task. A query engine is built into the top-level backup task to query the execution status of each graph's backup task. The top-level backup task is a collection of all graph backup tasks, mainly used to create graph backup tasks as subtasks and determine whether the subtasks are completed. In actual implementation, the completion status of the upload task can be determined based on the completion status of the lower-level tasks: when all lower-level tasks are completed, the upper-level task is considered complete; when any lower-level task is incomplete, the upper-level task is considered incomplete.

[0065] Step 103: Each gallery node asynchronously executes the lowest-level backup task, obtains the user data for backup through the lowest-level backup task, and backs up the user data on this gallery node.

[0066] The graph node retrieves the transaction ID from the transaction-accessible table, calculates the user data corresponding to the transaction based on the transaction ID, and obtains the log sequence number corresponding to each user data. User data includes vertex data and edge data. In practice, the LSNs of vertex or edge data visible in the transaction-accessible table can be assembled into backup tasks (hereinafter referred to as ltasks) that actually need to be executed and distributed to the task queue. One task can store 100,000 vertex / edge LSNs, and each LSN in an ltask can only correspond to the operation event log of a specific vertex or edge data. In practice, the ltasks that each graph node needs to execute can be organized into a task queue for easy acquisition and management. In practice, ltasks in the task queue can be executed in the order they were added to the queue, or they can be executed concurrently using multi-threading. Appropriate concurrency methods can be selected or set as needed to ensure that the execution order does not affect recovery and subsequent queries.

[0067] The lowest-level backup task is a subtask that backs up a batch of data for each tag or relationship. After each library node executes the lowest-level backup task, it carries the LSN of a batch of data. It then obtains specific data tuples through the LSNs, generates a backup record from these tuples using a unified encoding, and finally writes the backup to disk, thus completing the user data backup. On the other hand, each GSTORE also needs to back up the metadata under the current library / chart asynchronously. Since the amount of metadata is relatively small, task hierarchy is not required; each library node directly generates a metadata backup task for backup. GCM generates corresponding metadata information based on the backup command and broadcasts the metadata information to all library nodes that need backup. Each library node generates a metadata backup task based on the metadata information and backs up the metadata through the metadata backup task. After the metadata backup is completed, the metadata is also written to the node's backup.meta file using a unified encoding. In addition, the backup.meta file also records the transaction access table for this backup.

[0068] After steps 101-103 provided in this embodiment, each gallery node can asynchronously execute the backup task. The backup initiation process is as follows: Figure 2 As shown, the backup termination process is as follows: Figure 3 As shown, each gallery node can complete the backup of user data and metadata on each gallery node in parallel. Data interaction between nodes does not block other work, saving backup time and resources.

[0069] The method provided in this embodiment can support full backup and incremental backup. Since different backup methods require different data to be backed up, when GCM generates the transaction accessible table, the corresponding transaction accessible table can be generated according to the backup type configured in the backup command.

[0070] (1) When performing a full backup, count the transaction IDs related to the backup command, obtain all visible transaction numbers based on the statistical results, and write all visible transaction numbers into the transaction access table.

[0071] During a full backup, all data needs to be backed up. GCM will assemble the configuration for this backup, configure the metadata information of this backup package, and assign the current timestamp as the start time. In addition, GCM will perform transaction overview statistics based on the transaction ID of the current backup command and generate a transaction access table, which records the currently active transactions, visible transaction numbers, and invisible transaction numbers.

[0072] (2) When performing an incremental backup, obtain all visible transaction numbers according to the backup command, obtain the visible transaction number of the previous backup according to the transaction access table of the previous backup, compare all visible transaction numbers with the visible transaction number of the previous backup to obtain the visible transaction number of the current backup, and write the visible transaction number of the current backup into the transaction access table.

[0073] Incremental backups are based on the previous backup package A at any given time. The metadata and transaction access tables from the previous backup are retrieved from backup package A and compared with the transaction access tables at the time of the current incremental backup. This comparison yields the corresponding transaction access tables for the changed incremental data, which are then used to back up the incremental data. In practice, further incremental backups can be performed based on full or incremental backup packages made at any given time, outputting a new incremental backup package.

[0074] As can be seen, the backup method provided in this embodiment can simultaneously support backup in the form of logical backup and backup in the form of backup statements, and can meet the backup requirements of full backup, incremental backup and hot backup.

[0075] In this embodiment, as Figure 4 As shown, backup tasks at each level can be derived layer by layer from the top-level backup task.

[0076] (1) The top-level backup task is the top-level task for each gallery backup, and the backup status of its subtasks can be queried.

[0077] (2) Obtain the graph corresponding to the user data in the top-level backup task, and treat the backup task of each graph as a second-level backup task. A second-level backup task represents the backup task of each graph in a graph library, and its sub-task backup status can be queried. Figure 4 The backup tasks include two graphs in the graph library that contain the top-level backup task.

[0078] (3) Obtain the log sequence number of the data corresponding to the tag or relation in each three-level backup task, and treat each batch of log sequence number backup tasks as a four-level backup task. The third-level backup task is to obtain the LSN of the data corresponding to the tag or relation in this task, package a batch of LSNs, and then generate the packaged LSN data into a fourth-level task. Figure 4 It includes backup tasks that contain the labels and relationships of each graph in a two-level backup task.

[0079] (4) Obtain the data corresponding to the tags or relationships in each three-level backup task, and treat the backup task of the user data corresponding to each tag or relationship as a four-level backup task. The fourth-level backup task is a subtask of the LSN backup of a batch of data corresponding to a tag or relationship. It is the actual data backup process, which carries the LSN of a batch of data and is responsible for obtaining user data through the LSN, generating a backup record according to a unified encoding, and controlling the backup to be written to disk. Figure 4 It includes backup tasks for the data corresponding to each tag and relationship in the three-layer backup task.

[0080] In practice, tasks at each level are structured in a tree-like relationship. Therefore, when deriving tasks from lower levels, each level of backup tasks needs to be organized according to this tree structure, and a query engine needs to be established within each level of backup tasks so that higher-level backup tasks can query the status of lower-level backup tasks. Specifically, during the backup process, the query engine can initiate backup status queries, and each level of task can query the execution status of its own subtasks. For example, the backup status will show a RUNNING indicator during runtime and a FINISHED indicator upon completion. If the query engine initiates a backup termination command, GCM will notify each GSTORE to immediately stop the backup and mark the backup information as ABORT.

[0081] like Figure 5 As shown, user data for backup can be obtained through the lowest-level backup task and backed up in the following manner.

[0082] Step 201: Obtain the sequence number of each log in the lowest-level backup task, and retrieve the corresponding user data based on the log sequence number.

[0083] The transaction accessibility table stores the transaction range of this backup, as well as which invisible transactions are within this range and which visible transactions are outside this range. This allows the LSNs of all point and edge data that need to be backed up to be obtained based on the visible transactions. During backup, each graph node can scan the graph for point and edge data based on the LSNs in the transaction accessibility table to obtain the point and edge data that need to be backed up.

[0084] Step 202: Encode the obtained user data in a unified format according to the specified format, and write the encoded binary stream into the corresponding backup file.

[0085] To facilitate data recovery and reduce the space occupied by backup files, in this embodiment, user data is uniformly encoded and written into the backup file in binary form. Specifically, the backup file is stored in the backup directory specified in the configuration file of each gallery node. The path to the backup directory can be specified through configuration modification, or a mapping between the point edge data filename and the backup filename can be provided before writing.

[0086] Furthermore, after each gallery node completes backup through backup files, when data recovery is needed, the backup data can be read according to the unified encoding and restored to the gallery; alternatively, the backup files can be uploaded to GCM for unified restoration when idle, or transferred to other gallery nodes or other storage devices for multi-node distributed restoration.

[0087] After steps 201-202 provided in this embodiment, data backup on each gallery node can be completed.

[0088] In practice, each node in the distributed graph database does not store the start and end vertex keys for each edge data. When the underlying task backup performs edge data backup, it is still necessary to obtain the end point key of the edge data on the remote node through data interaction between nodes. For example Figure 6 As shown, the corresponding user data can be obtained based on the log sequence number in the following manner.

[0089] Step 301: Each graph node retrieves the point data and edge data from the user data stored on this graph node based on the log sequence number.

[0090] Each graph node asynchronously and in parallel starts the execution thread of the lowest-level backup task, retrieves ltask tasks from the task queue, obtains point data or edge data from VLOG based on the LSN in the ltask task, and then encodes this data in a unified way.

[0091] In practice, the encoding can be divided into two parts: header and body. Based on the backup method and the recovery operation corresponding to each piece of user data, a corresponding operation code is generated for each piece of user data, and this operation code serves as the header field of that piece of user data. The key-value pairs of each piece of user data serve as the data field of that piece of user data. The header field and data field of each piece of user data are combined to form the encoding of that piece of user data. For example... Figure 7As shown, the header in the data encoding includes LEN, XOP, and XID; where LEN indicates the length of this backup data entry; XOP indicates the opcode for this data entry, which is INSERT for a full backup and INSERT, UPDATE, or DELETE for incremental backups; XID is a unique identifier for this backup data entry. The BODY in the data encoding stores the actual data for each backup data entry, which can be represented using key-value pairs.

[0092] Step 302: When the number of edge data that need to be queried for endpoint data in the obtained edge data reaches a specified number, send the unique identifier of the endpoint data to be queried in batches to other graph nodes to obtain the corresponding endpoint data.

[0093] In a distributed graph library scenario, when GSTORE saves edges, if the starting and ending vertices of an edge are not on the same node, the endpoint data is located on the remote node, and the starting data is located on the current node. When backing up edge data, if the starting vertices of the edges to be backed up are not on the same node, the edge data is backed up based on the starting data. Each node obtains the starting data key on its own node and retrieves the endpoint data key located on the remote node through data interaction. A complete edge is then formed based on the starting and ending data keys. In specific implementation, the information sent to the remote node is a unique identifier for the endpoint data, such as a cluster-unique endpoint data rowid. To reduce communication overhead between different graph library nodes during backup, when a graph library node needs to query a certain number of edge data points for endpoint data in memory, it will send endpoint data key information to other nodes in batches based on the edge information and complete the backup record.

[0094] After steps 301-302 provided in this embodiment, the encoding and completion of the backup data can be completed.

[0095] The backup method based on a distributed graph database provided in this embodiment supports data backup without database downtime. It also supports initiating backup operations via data definition language and data manipulation language during the backup process, ensuring the integrity of data prior to the backup start time. The method provided in this embodiment runs independently on each distributed graph database node from the start time, and can perform incremental backups based on full or incremental backup packages made at any time, outputting a further incremental backup package. By fully utilizing the computer resources of each node, it enables independent backup operations for each node of the distributed graph database, reducing unnecessary resource contention.

[0096] Example 2:

[0097] Based on the backup method for distributed graph databases provided in Embodiment 1, it can be implemented in certain specific embodiments using the specific implementation methods described in this embodiment. It is understood that the specific implementation methods provided in this embodiment are only used to illustrate the specific implementation process of the method in Embodiment 1 in certain specific scenarios and are not intended to limit the scope of protection.

[0098] In this embodiment of the distributed graph library, the data storage structure is as follows: Figure 8 As shown in the example, the `person` tag has three dots with keys `person1`, `person2`, and `person3`, and the `car` tag has three dots with keys `car1`, `car2`, and `car3`. The relationship between `person` and `car` is ownership; `person1` owns `car3`, `person2` owns `car1`, and `person3` owns `car2`. `person1`, `person2`, `car1`, and `car3` are located on the `GSTORE1` node, while `person3` and `car2` are located on `GSTORE2`.

[0099] This data backup is a full backup at the gallery level, requiring the backup of all the aforementioned data. In practice, users can initiate a specific backup task using the backup execution statement template: `BACKUP ALL DATABASES FULLTO "backupName" [COMMENT `descriptions`]`. Here, `backupName` is the backup name string, a required field, which can also be used as the backup file name; `descriptions` is a description of this backup, an optional field.

[0100] During this backup, the user first initiates the backup execution through the query engine, specifically with the statement: `BACKUP ALLDATABASES FULL TO "backup1" "Full Backup"`. The query engine sends this statement to GCM to notify GCM to initiate the corresponding backup command.

[0101] GCM generates a backup configuration based on the backup command, including the backup ID, backup name, backup start time, and transaction access table for this backup. In this scenario, the backup transaction ID is 1000, and the active transaction IDs are 300, 890, and 900. The generated transaction access table is as follows: Figure 9 As shown. Figure 9 As shown, the transaction IDs of the data accessible in this backup range from 0 to 1000, excluding transaction IDs 300, 890, and 900. After obtaining the transaction access table, GCM will generate the metadata for this backup based on the transaction IDs.

[0102] GCM broadcasts the config settings along with the transaction accessibility table for this backup to the GSTORE nodes it manages, and starts a timer task to monitor the backup status of each GSTORE. The GCM timer task will periodically obtain the backup status of the GSTORE nodes so that GCM can update the backup information in a timely manner. This information includes backup status, backup progress, backup end time, backup error code, etc.

[0103] After receiving the backup configuration and transaction accessibility table, each GSTORE node generates local backup metadata, followed by an asynchronous metadata backup task and a top-level asynchronous user data backup task. The metadata backup task carries the metadata to be backed up and its responsibility is to perform a full backup of the metadata. At this point, GSTORE returns the broadcast result and begins running the metadata backup task and the top-level user data backup task. During the backup process, GCM immediately returns the recovery information broadcast by GSTORE to the client so that users can view the backup status of each GSTORE.

[0104] At this time, the metadata backup task and the user data backup task in the GSTORE node are executed concurrently. The metadata backup task writes the metadata of the backup target to the backup file; while the ltask user data top-level task is derived into a 4-layer full backup task.

[0105] The specific implementation of the ltask task is as follows, and its backup thread model is as follows: Figure 10 As shown.

[0106] (1) The ltask top-level backup task carries all graph objects and all information of this backup. When it is executed, it will generate second-level backup tasks and is responsible for monitoring the execution status of the second-level backup tasks. If the second-level backup task of any graph node fails, the top-level task will modify the backup status and notify other subtask threads to stop the backup immediately; once GCM obtains the backup failure status, it will notify other graph nodes to stop the backup immediately.

[0107] (2) The ltask second-level backup task is responsible for extending the third-level backup task. It carries an object of a graph that is being backed up. When it is executed, it will obtain all the labels or relationships under the graph based on the graph object and derive the third-level backup task one by one according to the labels or relationships.

[0108] (3) The ltask third-level backup task is responsible for scanning the tag table or relationship table of this layer backup task, packaging its data LSNs in batches, and generating the fourth-level backup task.

[0109] (4) The ltask fourth-level backup task is responsible for obtaining specific data based on the LSN in the batch, generating backup record data according to the unified encoding, and writing the backup record data batch to disk. When performing a full backup, the unified encoding XOP is INSERT.

[0110] During the backup process, users can use the SHOW BACKUPS command to query the status of the backup. The backup results will be as follows: Figure 11 As shown. If the backup task has been completed, the "status" column will display "FINISHED"; if the backup task has not yet been completed, the "status" column will display "RUNNING"; when the backup fails for some reason, the "code" column will display the error code, and you can look up the error code to find out what caused the failure. When the backup task is completed correctly, the data in its backup file is as follows: Figure 12 As shown.

[0111] The complete timing and data interaction process of the above backup method is as follows: Figure 13 As shown. The user inputs an asynchronous backup statement to the query engine. The query engine sends a backup command to the GCM, which broadcasts the transactionally accessible table to the GSTORE. The GSTORE then completes the backup of metadata and user data based on the transactionally accessible table. The GSTORE reports the backup task execution status to the GCM, and the GCM updates the backup information according to the GSTROE. The user inputs a backup information query statement to the query engine. The query engine sends an information query command to the GCM, which confirms the current backup information and sends it back to the query engine. The query engine updates the backup information and sends it back to the user.

[0112] Example 3:

[0113] Based on the backup method for distributed graph databases provided in Embodiment 1, it can be implemented in certain specific embodiments using the specific implementation methods described in this embodiment. It is understood that the specific implementation methods provided in this embodiment are only used to illustrate the specific implementation process of the method in Embodiment 1 in certain specific scenarios and are not intended to limit the scope of protection.

[0114] In the scenario of Example 2, the user initiated a full backup (backup1). In this example, an incremental backup is performed based on the backup results from Example 2, building upon the full backup (backup1) from Example 2.

[0115] The user initiates an incremental backup task using the following statement: `BACKUP ALL DATABASES INCREMENT TO backup2 BASE ON backup1`. This statement indicates that an incremental backup is performed based on the full backup package `backup1`, with the backup named `backup2` and the transaction ID of this incremental backup being 1800.

[0116] Since this backup is incremental, GCM will read the transaction ID (1000) and the active transaction ID from the previous backup when generating the transaction access table. It will then generate the transaction access table based on the current active transactions and the active transactions from the previous backup. After comparison, the transaction range in the transaction access table will be [1000, 1800]. Active transactions within this range need to be excluded, and the table will include completed transaction numbers from the active transactions of the previous backup. After obtaining the transaction access table, GSTORE will scan and back up the data based on this newly generated incremental transaction access table.

[0117] On the other hand, in incremental backup, when backing up metadata, the metadata file `backup.mate` from the previous backup is first retrieved based on the previous backup task name `backup1`. By comparing the metadata information with the existing system metadata information, the incremental changes in metadata since the last backup can be determined. Subsequently, the metadata backup task generates a backup record for the metadata, whose XOP is one of UPDATE, DELETE, or INSERT, representing that the metadata record has been modified, deleted, or added. The user data ltask, based on the transactions from the previous backup, performs a data scan based on the incrementally transaction-accessible tables, then generates a multi-level ltask task, and completes the backup task according to the backup method in Example 2.

[0118] Example 4:

[0119] Based on the backup methods based on distributed graph databases provided in Embodiments 1 to 3 above, the present invention also provides a backup device based on a distributed graph database that can be used to implement the above methods, such as... Figure 14 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. The backup device based on a distributed graph database in this embodiment includes one or more processors 11 and a memory 12. Figure 14 Take a processor 11 as an example.

[0120] Processor 11 and memory 12 can be connected via a bus or other means. Figure 14 Taking the example of a connection between China and Israel via a bus.

[0121] The memory 12, as a non-volatile computer-readable storage medium for a backup method based on a distributed graph database, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the backup methods based on distributed graph databases in Embodiments 1 to 3. The processor 11 executes various functional applications and data processing of the backup device based on the distributed graph database by running the non-volatile software programs, instructions, and modules stored in the memory 12, thereby implementing the backup methods based on distributed graph databases in Embodiments 1 to 3.

[0122] Memory 12 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 12 may optionally include memory remotely located relative to processor 11, which can be connected to processor 11 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0123] The program instructions / modules are stored in memory 12. When executed by one or more processors 11, they perform the backup methods based on distributed graph databases described in Embodiments 1 to 3 above. For example, they perform the methods described above. Figure 1 , Figure 5 and Figure 6 The steps shown.

[0124] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 14 One of the processors 11 can enable the aforementioned one or more processors to execute the backup method based on the distributed graph database in Embodiments 1 to 3, for example, to perform the above-described... Figure 1 , Figure 5 and Figure 6 The steps shown can also be implemented. Figure 8 , Figure 10 The various modules and units mentioned above.

[0125] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0126] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A backup method based on a distributed graph database, characterized in that, include: Generate the corresponding transaction access table based on the backup command, and broadcast the transaction access table to all graph nodes that need to be backed up; Each graph node generates a top-level backup task based on the transaction-accessible table. Backup tasks at each level are then derived from this top-level backup task. This includes: obtaining the graph corresponding to the user data in the top-level backup task, treating each graph's backup task as a second-level backup task; obtaining the graph's labels and relationships in each second-level backup task, treating each graph's label and relationship backup task as a third-level backup task; obtaining the log sequence number of the data corresponding to the labels or relationships in each third-level backup task, treating each batch of log sequence number backup tasks as a fourth-level backup task; organizing the backup tasks at each level according to a tree structure, and establishing a query engine in each backup task level so that upper-level backup tasks can query the status of lower-level backup tasks. Each gallery node asynchronously executes the lowest-level backup task, obtains the user data for backup through the lowest-level backup task, and backs up the user data on this gallery node.

2. The backup method based on a distributed graph database according to claim 1, characterized in that, The step of generating the corresponding transaction-accessible table based on the backup command specifically includes: When performing a full backup, the transaction IDs related to the backup command are counted, and all visible transaction numbers are obtained based on the count results and written into the transaction access table. When performing an incremental backup, all visible transaction numbers are obtained according to the backup command. The visible transaction numbers from the previous backup are obtained from the transaction access table. All visible transaction numbers are compared with the visible transaction numbers from the previous backup to obtain the visible transaction numbers for the current backup. The visible transaction numbers for the current backup are then written into the transaction access table.

3. The backup method based on a distributed graph database according to claim 1, characterized in that, Each gallery node generates a top-level backup task on that gallery node based on the transaction-accessible table, specifically including: The library node creates a backup task for each graph, and the set of backup tasks for all graphs is used as the top-level backup task. A query engine is built in the top-level backup task to query the execution status of the backup task for each graph.

4. The backup method based on a distributed graph database according to claim 1, characterized in that, The process of obtaining user data for backup through the lowest-level backup task and backing up the user data on this gallery node specifically includes: Get the sequence number of each log in the lowest-level backup task, and retrieve the corresponding user data based on the log sequence number; The obtained user data is uniformly encoded according to the specified format, and the encoded binary stream is written to the corresponding backup file.

5. The backup method based on a distributed graph database according to claim 4, characterized in that, The step of obtaining the corresponding user data based on the log sequence number also includes: Each graph node retrieves the point and edge data from the user data stored on this graph node based on the log sequence number; When the number of edge data points that require querying endpoint data reaches a specified number, the unique identifiers of the endpoint data to be queried are sent in batches to other graph nodes to obtain the corresponding endpoint data.

6. The backup method based on a distributed graph database according to claim 4, characterized in that, The process of uniformly encoding the obtained user data according to a specified format specifically includes: Based on the backup method and the recovery operation corresponding to each piece of user data, generate a corresponding operation code for each piece of user data, and use the operation code as the header field of that piece of user data; Use the key-value pair of each user data record as the data field of that user data record; The header field and data field of each user data entry are combined to form the encoding of that user data entry.

7. The backup method based on a distributed graph database according to claim 1, characterized in that, The method further includes: Generate corresponding metadata information based on the backup command, and broadcast the metadata information to all gallery nodes that need to be backed up; Each gallery node generates a metadata backup task based on the metadata information, and the metadata is backed up through the metadata backup task.

8. A backup device based on a distributed graph database, characterized in that: The method includes at least one processor and a memory, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to complete the backup method based on a distributed graph database as described in any one of claims 1-7.

9. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the backup method based on a distributed graph database as described in any one of claims 1-7.

Images