Backup method and device based on distributed graph database

Abstract

The invention relates to the field of databases, in particular to a backup method and device based on a distributed graph database. The method mainly comprises the following steps: generating a corresponding transaction accessible table according to a backup command, and broadcasting the transaction accessible table to all gallery nodes needing to be backed up; generating a top-layer backup task on the image library node by each image library node according to the transaction accessible table, and deriving a backup task of each level layer by layer through the top-layer backup task; each gallery node asynchronously executes the backup task of the lowest layer, user data used for backup are obtained through the backup task of the lowest layer, and the user data on the gallery node are backed up. According to the method, reasonable utilization of computer resources can be improved, the effect of complete parallel backup is achieved, and the backup efficiency and performance are improved to a great extent. And the network overhead can be reduced during recovery, so that the recovery efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of databases, and in particular to a backup method and device based on a distributed graph database. Background Art

[0002] Graph database is a type of non-relational database, which mainly uses graph theory to store the relationship information between entities. Because it can store the unique relationship value of graph model, the market demand for graph database is increasing, and distributed graph database comes into being. Data security is particularly important under distributed architecture, and data backup device of distributed graph database has also become an indispensable part.

[0003] With the development of database systems, there are many types of backups. However, according to the backup characteristics such as storage data sets, server status and backup package objects, they can be roughly divided into the following types: full backup, incremental backup, differential backup, physical backup, logical backup, cold backup and hot backup. When performing a backup, it is necessary to verify the backup path, extract the metadata and user point-edge data in the library, process the data and write the data. Each step requires multiple communications or data exports. Due to the complexity of the graph relationships stored in the graph database, the relationships between each point and edge are intricate, and the data in each graph database node is related to other nodes. If the data backup in the graph database is stored uniformly or exported, the network and computer resource overhead will be very large.

[0004] In view of this, how to overcome the defects of the prior art and solve the problem of large resource consumption of the existing graph database backup method is a problem to be solved in this technical field. Summary of the invention

[0005] In view of the above defects or improvement needs of the prior art, the present invention solves the problem of large resource consumption of the existing graph database backup method.

[0006] The embodiment of the present invention adopts the following technical solution:

[0007] In the first aspect, the present invention provides a backup method based on a distributed graph database, specifically: generating a corresponding transaction accessible table according to a backup command, and broadcasting the transaction accessible table to all graph library nodes that need to be backed up; each graph library node generates a top-level backup task on this graph library node according to the transaction accessible table, and derives each level of backup tasks layer by layer through the top-level backup task; each graph library node asynchronously executes the bottom-level backup task, obtains user data for backup through the bottom-level backup task, and backs up the user data on this graph library node.

[0008] Preferably, generating a corresponding transaction accessible table according to the backup command specifically includes: when performing a full backup, counting the transaction IDs related to the backup command, obtaining all visible transaction numbers according to the statistical results, and writing all visible transaction numbers into the transaction accessible table; when performing an incremental backup, obtaining all visible transaction numbers according to the backup command, obtaining the previous visible transaction number according to the transaction accessible table at the time of the previous backup, comparing all visible transaction numbers with the previous visible transaction number to obtain the current visible transaction number, and writing the current visible transaction number into the transaction accessible table.

[0009] Preferably, each of the graph library nodes generates a top-level backup task on the graph library node according to the transaction accessible table, specifically including: the graph library node establishes a backup task for each graph, and the backup task set of all graphs is used as the top-level backup task; a query engine is established in the top-level backup task to query the execution status of the backup task of each graph

[0010] Preferably, the backup tasks of each level are derived layer by layer through the top-level backup task, specifically including: obtaining the graph corresponding to the user data in the top-level backup task, and treating the backup task of each graph as a second-level backup task; obtaining the labels and relationships of the graph in each second-level backup task, and treating the backup tasks of the labels and relationships in each graph as a third-level backup task; obtaining the log sequence numbers of the data corresponding to the labels or relationships in each third-level backup task, and treating the backup tasks of each batch of log sequence numbers as a fourth-level backup task; organizing the backup tasks of each level according to a tree relationship, and establishing a query engine in each layer of backup tasks, so that the backup tasks of the upper layer can query the status of the backup tasks of the lower layer.

[0011] Preferably, the method of obtaining user data for backup through the bottom-level backup task and backing up the user data on the gallery node specifically includes: obtaining each log sequence number in the bottom-level backup task, and obtaining the corresponding user data according to the log sequence number; uniformly encoding the obtained user data according to a specified format, and writing the encoded binary stream into the corresponding backup file.

[0012] Preferably, the method of obtaining corresponding user data according to log sequence number also includes: each gallery node obtains point data and edge data in the user data stored on the gallery node according to the log sequence number; when the number of edge data for which endpoint data needs to be queried among the obtained edge data reaches a specified number, the unique identifiers of the endpoint data to be queried are sent in batches to other gallery nodes to obtain the corresponding endpoint data.

[0013] Preferably, the obtained user data is uniformly encoded according to a specified format, specifically including: generating a corresponding operation code for each piece of user data according to a backup method and a recovery operation corresponding to each piece of user data, and using the operation code as a header field of the piece of user data; using a key-value pair of each piece of user data as a data field of the piece of user data; and combining the header field and the data field of each piece of user data into the encoding of the piece of user data.

[0014] Preferably, the method also includes: generating corresponding metadata information according to the backup command, and broadcasting the metadata information to all gallery nodes that need to be backed up; each gallery node generates a metadata backup task on the gallery node according to the metadata information, and backs up the metadata through the metadata backup task.

[0015] In the second aspect, the present invention provides a backup device based on a distributed graph database, specifically: including at least one processor and a memory, at least one processor and the memory are connected through a data bus, the memory stores instructions that can be executed by at least one processor, and after the instructions are executed by the processor, they are used to complete the backup method based on the distributed graph database in the first aspect.

[0016] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method described in the first aspect.

[0017] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing the method provided in the first aspect.

[0018] In a fifth aspect, a computer program product comprising instructions is provided, which, when executed on a computer or a processor, causes the computer or the processor to execute the method provided in the first aspect.

[0019] Compared with the prior art, the beneficial effect of the present invention is that the backup task of the entire graph database is divided into multiple levels, and the backup task is sent to each distributed graph library node, and the specific data backup process is executed in parallel in an asynchronous manner on each graph library node, and each backup data packet is distributed in each graph library node. In this way, the rational use of computer resources can be improved, the effect of fully parallel backup can be achieved, and the efficiency and performance of backup can be greatly improved. It can also reduce network overhead during recovery, greatly improving recovery efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flow chart of a backup method based on a distributed graph database provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a process of initiating backup in a method provided in an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a process of terminating backup in a method provided in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of a task hierarchy structure in a method provided by an embodiment of the present invention;

[0025] Figure 5 A flow chart of another backup method based on a distributed graph database provided by an embodiment of the present invention;

[0026] Figure 6 A flow chart of another backup method based on a distributed graph database provided by an embodiment of the present invention;

[0027] Figure 7 A schematic diagram of a field structure of data encoding in a method provided in an embodiment of the present invention;

[0028] Figure 8 A schematic diagram of a data storage structure in a specific scenario of the method provided by an embodiment of the present invention;

[0029] Fig. 9 A transaction accessible representation intent in a specific scenario of the method provided by an embodiment of the present invention;

[0030] Fig.10 A schematic diagram of a backup thread model in the method provided in an embodiment of the present invention;

[0031] Fig.11 A schematic diagram of a backup structure in a specific scenario of the method provided by an embodiment of the present invention;

[0032] Fig.12 A data schematic diagram of a backup file in a specific scenario of the method provided by an embodiment of the present invention;

[0033] Fig.13 A schematic diagram of the timing of the backup process in the method provided by an embodiment of the present invention;

[0034] Fig.14 A schematic diagram of the structure of a backup device based on a distributed graph database provided by an embodiment of the present invention;

[0035] The reference numerals are as follows:

[0036] 11: processor; 12: memory. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0038] The present invention is an architecture of a specific functional system, so the specific embodiments mainly illustrate the functional logical relationship between the various structural modules, and do not limit the specific software and hardware implementation methods.

[0039] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0040] Some terms used in the embodiments of the present invention are explained as follows:

[0041] (1) Graph Store (abbreviated as GSTORE): a native graph distributed storage engine, hereinafter referred to as "graph store".

[0042] (2) Graph query engine (Cypher Server): A graph query engine that executes based on open cypher syntax rules, hereinafter referred to as the "query engine".

[0043] (3) Management node (Graph Cluster Manager, abbreviated as GCM): The management node of the native graph storage GSTORE engine.

[0044] (4) Distributed library node: A GSTORE node deployed in a distributed library. Multiple GSTOREs, one or more GSTORE nodes and one or more GCM nodes form a distributed cluster, which is hereinafter referred to as a “library node”.

[0045] (5) SEGMENT: A data segmentation device inside GSTORE.

[0046] (6) VLOG: A data storage device inside GSTORE that completely stores metadata, user data, or system data. A piece of data can be uniquely located through a log sequence number (LSN).

[0047] The types of graph database backup methods mainly include:

[0048] (1) Full backup: A complete backup of all data or applications at a certain point in time, including system metadata, user metadata, system data, and user data.

[0049] (2) Incremental backup: Back up data that has changed since the last full backup or incremental backup, including data that has changed due to addition, modification, deletion, etc.

[0050] (3) Differential backup: backup the data that has changed since the last full backup. This shows that the function of incremental backup actually covers differential backup.

[0051] (4) Physical backup: Database file-level backup, the backup content is the database files on the operating system.

[0052] (5) Logical backup: Back up data in a certain logical form and store it in a backup set.

[0053] (6) Cold backup: Also known as offline backup, it refers to a complete database backup performed when the database is closed and cannot be updated. During the backup process, users cannot access the system or database normally.

[0054] (7) Hot backup: Backup is also called online backup or on-line backup. It is the data backup performed when the system is running normally. During the backup process, users can access the system or database normally.

[0055] Embodiment 1:

[0056] This embodiment provides a backup method based on a distributed graph database, which decomposes the backup task of the entire graph library into multiple levels of backup tasks on each graph library node, and backs up the data in the graph library to the current distributed graph library node in a unified data format.

[0057] like Figure 1 As shown, the specific steps of the backup method based on the distributed graph database provided by the embodiment of the present invention are as follows.

[0058] Step 101: Generate a corresponding transaction accessible table according to the backup command, and broadcast the transaction accessible table to all library nodes that need to be backed up.

[0059] When performing a backup, the backup command can be used to indicate configuration attributes such as the backup transaction ID range, backup start time, and backup range. In a specific implementation, the query engine can input a graph backup statement as a backup command through the command line to notify GCM to initiate a full or incremental backup.

[0060] In the method provided in this embodiment, the backup task is deployed separately on each library node, which is conducive to the separate recovery of each node. In a distributed library, GCM can obtain the access status of the data on each library node, while each library node cannot obtain the access status of the data on other library nodes. Therefore, GCM needs to generate a transaction accessible table based on the access status of the data, and each library node can determine the scope of data transactions that need to be backed up based on the transaction accessible table. Furthermore, in order for each library node to obtain the attributes of the backup task and the transaction accessible table, GCM needs to send the corresponding data to each library node by broadcasting.

[0061] Step 102: Each library node generates a top-level backup task on the library node according to the transaction accessible table, and derives backup tasks of each level layer by layer through the top-level backup task.

[0062] After GCM notifies each GSTORE graph library node of the config configuration and the transaction accessible table to initiate a distributed backup, each GSTORE graph library node will perform preparation work, including inspection, generation of metadata (graph, label, relationship) of this backup record, and calculation of the graph object of this backup.

[0063] In the general backup process, usually only the gallery is backed up, and the ordinary tables in the database are not backed up. The backup method provided in this embodiment can support two levels of data backup: graph-level backup and gallery-level backup. Graph-level backup is to back up all user data and the relationship between user data in each graph, that is, to back up the points, edges and metadata under a single graph; gallery-level backup refers to backing up the gallery under the entire cluster, and backing up the ordinary table data such as the label table and relationship table under the gallery, including initiating a full backup, querying the backup progress, initiating an incremental backup, terminating the ongoing backup and deleting the existing backup. Correspondingly, in the method provided in this embodiment, each gallery node is executed concurrently in the form of a multi-layer task queue. The multi-layer task organizes the information of the gallery stored on each node, as well as the hierarchical relationship of the label table and relationship table of all graphs under each gallery, thereby supporting gallery-level backup and graph-level backup.

[0064] In the specific implementation, the backup task at the gallery level can be taken as the top-level backup task first, and then the backup tasks at the graph level, label relationship level, and data level can be derived to the lower levels in turn. The gallery node establishes a backup task for each graph, and the set of backup tasks for all graphs is taken as the top-level backup task; a query engine is established in the top-level backup task to query the execution status of the backup task for each graph. The top-level backup task is a collection of all graph backup tasks, which is mainly used to create graph backup tasks as subtasks and determine whether the subtasks are completed. In the actual implementation, whether the upload task is completed can be judged based on the completion status of the lower-level tasks: when all lower-level tasks are completed, the upper-level task is judged to be completed; when any lower-level task is not completed, the upper-level task is judged to be incomplete.

[0065] Step 103: Each gallery node asynchronously executes the bottom-level backup task, obtains the user data for backup through the bottom-level backup task, and backs up the user data on the gallery node.

[0066] The graph library node obtains the transaction ID in the transaction accessible table, counts the user data corresponding to the transaction according to the transaction ID, and obtains the log sequence number corresponding to each user data, where the user data includes point data and edge data. In the specific implementation, the LSNs of the point data or edge data visible in the transaction accessible table can be assembled into the backup tasks that actually need to be executed (hereinafter referred to as ltask tasks), and distributed to the task queue. One task can store 100,000 point / edge LSNs, and the LSN in each ltask task can only correspond to the operation event log of a certain point data or edge data. In the actual implementation, the ltask tasks that each graph library node needs to execute can be organized into a task queue to facilitate the acquisition and management of the ltask tasks. In the specific implementation, the ltask tasks in the task queue can be executed in the order of joining the queue, or they can be executed concurrently in the form of multi-threaded concurrency. The appropriate concurrency mode can be selected or set as needed to ensure that the execution order does not affect the recovery and post-recovery query.

[0067] The lowest level backup task is a subtask of backing up a batch of data for each label or relationship. After each library node executes the lowest level backup task, it carries a batch of data LSNs, obtains specific data tuples through LSNs, and generates backup Records for these data tuples according to unified coding, and then stores the backup to disk, thus completing the backup of user data. On the other hand, each GSTORE also needs to back up the metadata under the current map / library in an asynchronous manner. Since the amount of metadata is small, there is no need to divide the task hierarchy, and each library node directly generates a metadata backup task for backup. GCM generates corresponding metadata information according to the backup command and broadcasts the metadata information to all library nodes that need to be backed up; each library node generates a metadata backup task on the library node according to the metadata information, and backs up the metadata through the metadata backup task. After the metadata backup is completed, the metadata will also be written into the backup.meta file of this node in a unified coding manner. In addition, the backup.meta file also records the transaction accessible table of this backup.

[0068] After steps 101 to 103 provided in this embodiment, each library node can asynchronously execute the backup task. The process of initiating the backup is as follows: Figure 2 As shown, the backup termination process is as follows Figure 3 As shown, each library node can complete the data interaction between the backup nodes of the user data and metadata on each library node in parallel without blocking other work, saving backup time and resources.

[0069] The method provided in this embodiment can support full backup and incremental backup. Since different backup methods require different data to be backed up, when GCM generates a transaction accessible table, it can generate a corresponding transaction accessible table according to the backup type configured in the backup command.

[0070] (1) When performing a full backup, count the transaction IDs related to the backup command, obtain all visible transaction numbers based on the statistical results, and write all visible transaction numbers into the transaction accessible table.

[0071] When performing a full backup, all data needs to be backed up. GCM will assemble the config configuration for this backup, configure the metadata information of this backup package, and assign the current timestamp as the start time. In addition, GCM will perform transaction overview statistics based on the transaction ID of the current backup command and generate a transaction accessible table, which records the current active transactions, visible transaction numbers, and invisible transaction numbers.

[0072] (2) When performing an incremental backup, obtain all visible transaction numbers according to the backup command, obtain the previous visible transaction number according to the transaction accessible table at the time of the previous backup, compare all visible transaction numbers with the previous visible transaction number to obtain the current visible transaction number, and write the current visible transaction number into the transaction accessible table.

[0073] Incremental backup is based on the backup package A at any time. The metadata of the last time and the transaction accessible table backed up last time are obtained from the backup package A, and compared with the transaction accessible table at the time of this incremental backup, and the transaction accessible table corresponding to the incremental data changed in this incremental backup is obtained. The incremental data is backed up based on this. In actual implementation, the full backup or incremental backup package made at any time can be re-incremented and output as a re-incremental backup package.

[0074] It can be seen that the backup method provided in this embodiment can simultaneously support backup in the form of logical backup and backup in the form of backup statements, and can meet the backup requirements of full backup, incremental backup, and hot backup.

[0075] In this embodiment, Figure 4 As shown, backup tasks at each level can be derived layer by layer through the top-level backup task.

[0076] (1) The top-level backup task is the top-level task for each gallery backup, which can query the backup status of subtasks.

[0077] (2) Obtain the graph corresponding to the user data in the top-level backup task, and treat the backup task of each graph as a second-level backup task. The second-level backup task represents the backup task of each graph in a graph library, which can query the backup status of the subtask. Figure 4 The backup task for 2 graphs in the graph library that contains the top-level backup task.

[0078] (3) Obtain the log sequence number of the data corresponding to the label or relationship in each three-layer backup task, and treat the backup task of each batch of log sequence numbers as a four-layer backup task. The third-layer backup task is to obtain the LSN of the data corresponding to the label or relationship in this task, and pack a batch of LSNs, and then generate the fourth-layer task with the packed LSN data. Figure 4 The backup task contains the labels and relationships of each graph in the second-layer backup task.

[0079] (4) Obtain the data corresponding to the label or relationship in each three-layer backup task, and treat the backup task of the user data corresponding to each label or relationship as a four-layer backup task. The fourth-layer backup task is a subtask of backing up a batch of data LSNs corresponding to a label or relationship. It is the actual data backup process. It carries the LSN of a batch of data and is responsible for obtaining user data through LSNs, and then generates backup records for these data according to the unified coding. It controls the backup to disk. Figure 4 The backup task includes the data corresponding to each label and relationship in the three-layer backup task.

[0080] In actual implementation, each layer of tasks presents a tree-like relationship. Therefore, when deriving tasks at the lower level, it is necessary to organize the backup tasks at each level in a tree-like relationship, and establish a query engine in each layer of backup tasks so that the upper-level backup tasks can query the status of the lower-level backup tasks. In specific implementation, the query engine can initiate a backup status query during the backup process, and each level of tasks can query the execution status of their own lower-level subtasks. For example: the backup status will have a RUNNING status indication when running, and a FINISHED status indication after completion. If the query engine initiates a backup termination command, GCM will notify each GSTORE to immediately stop the backup and mark the backup information as ABORT status.

[0081] like Figure 5 As shown, the user data for backup can be obtained through the bottom-level backup task and the user data can be backed up in the following manner.

[0082] Step 201: Obtain each log sequence number in the bottom-level backup task, and obtain corresponding user data according to the log sequence number.

[0083] The transaction accessible table stores the transaction range of this backup, which invisible transactions are in this range, and the visible transactions outside this range, so as to obtain the LSN of all point data and edge data that need to be backed up based on the visible transactions. During the backup, each graph library node can scan the point data and edge data of the graph library according to the LSN in the transaction accessible table to obtain the point data and edge data that need to be backed up.

[0084] Step 202: uniformly encode the obtained user data according to a specified format, and write the encoded binary stream into a corresponding backup file.

[0085] In order to facilitate data recovery and reduce the space occupied by backup files, in this embodiment, user data is uniformly encoded and written into the backup file in binary form. In specific implementation, the backup file is stored in the backup directory specified by each gallery node configuration file. The path of the backup directory can be specified by configuration modification, or a mapping between the point edge data file name and the backup file name can be provided before writing.

[0086] Furthermore, after each gallery node completes the backup through the backup file, when data recovery is needed, the backed-up data can be read according to the unified code and the data can be restored to the gallery; the backup file can also be uploaded to GCM for unified restoration when idle, or it can be transmitted to other gallery nodes or other storage devices for multi-node distributed restoration.

[0087] After step 201 to step 202 provided in this embodiment, the data backup on each gallery node can be completed.

[0088] In actual implementation, each node in the distributed graph library does not store the start and end vertex keys of each edge data. When the underlying task backup performs edge data backup, it is also necessary to obtain the end point key of the edge data on the remote node through data interaction between nodes. Figure 6 As shown, the corresponding user data can be obtained according to the log sequence number in the following manner.

[0089] Step 301: Each graph library node obtains point data and edge data in the user data stored on the graph library node according to the log sequence number.

[0090] Each library node asynchronously and in parallel starts the execution thread of the lowest-level backup task, obtains the ltask task from the task queue, obtains point data or edge data from the VLOG according to the LSN in the ltask task, and then encodes these data in a unified way.

[0091] In actual implementation, the encoding can be divided into two parts: header and body. According to the backup method and the corresponding recovery operation of each piece of user data, a corresponding operation code is generated for each piece of user data, and the operation code is used as the header field of the user data; the key-value pair of each piece of user data is used as the data field of the user data; the header field and data field of each piece of user data are combined into the encoding of the user data. Figure 7As shown in the figure, the header in the data encoding includes LEN, XOP and XID; LEN indicates the length of the backup data; XOP indicates the operation code of the data. When it is a full backup, the XOP operation code is INSERT. When it is an incremental backup, the XOP position of the header should be one of INSERT, UPDATE, and DELETE; XID is the unique identifier of the backup data. In the data encoding, BODY stores the actual data of each backup data, which can be represented by a key-value pair.

[0092] Step 302: When the number of edge data for which end point data needs to be queried reaches a specified number among the acquired edge data, the unique identifiers of the end point data to be queried are sent in batches to other graph library nodes to obtain the corresponding end point data.

[0093] In the distributed graph library scenario, when GSTORE saves edges, when the starting vertex and the ending vertex of the edge are not on the same node, the end point data of the edge is located at the remote node, and the starting point data is located at the current node. When backing up edge data, when the starting vertex of the backed up edge is not on the same node, the edge data is backed up based on the starting point data. Each node obtains the key of the starting point data on the node, and obtains the key of the end point data at the remote end through data interaction with the remote node, and forms a complete edge based on the keys of the starting point data and the end point data. In the specific implementation, the information sent to the remote node is the unique identifier of the end point data, such as the unique rowid of the end point data in the cluster. In order to reduce the communication consumption between different graph library nodes during the backup process, when a certain graph library node needs to query the end point data in memory for a certain amount of edge data, it will be sent to other nodes in batches according to the information on the edge of the point to obtain the end point data key information, and complete the backup record.

[0094] After step 301 - step 302 provided in this embodiment, the encoding and completion of the backup data can be completed.

[0095] The backup method based on the distributed graph database provided in this embodiment supports data backup without shutting down the database, supports initiating backup operations through data definition language and data operation language during the data backup process, and can ensure the integrity of the data before the start of the backup. The method provided in this embodiment will run independently in each distributed graph library node from the running time, and can be re-incremented according to the full backup or incremental backup package made at any time, and output as a re-incremental backup package. By making full use of the computer resources of each node, it is possible to realize the independent backup operation of each node of the distributed graph library data and reduce unnecessary resource competition.

[0096] Embodiment 2:

[0097] Based on the backup method based on the distributed graph database provided in Example 1, in some specific implementations, it can be implemented through the specific implementation in this embodiment. It can be understood that the specific implementation provided in this embodiment is only used to illustrate the specific implementation process of the method in Example 1 in certain specific scenarios, and is not intended to limit the scope of protection.

[0098] In the distributed image library of this embodiment, the data storage structure is as follows: Figure 8 As shown. In this example, the person tag has three points whose keys are person1, person2, and person3, and the car tag has three points whose keys are car1, car2, and car3; the relationship between person and car is owned (own); person1 owns car3, person2 owns car1, and person3 owns car2. Person1, person2, car1, and car3 are located on the GSTORE1 node, and person3 and car2 are located on GSTORE2.

[0099] This data backup is a full backup at the gallery level, and all the above data needs to be backed up. In actual implementation, users can initiate a specific backup task through the backup execution statement template: BACKUP ALL DATABASES FULLTO "backupName" [COMMENT`descriptions`]. Among them, backupName is the backup name string, which is a required option and can also be used as the file name of the backup file; descriptions is the description of this backup, which is an optional item.

[0100] During this backup, the user first initiates the backup execution through the query engine, with the specific statement: BACKUP ALLDATABASES FULL TO "backup1" "full backup". The query engine sends this statement to GCM to notify GCM to initiate the corresponding backup command

[0101] GCM generates a backup config according to the backup command, generates the ID of this backup, fills in the backup name, backup start time and the transaction accessible table of this backup. In this scenario, the backup transaction ID is 1000, and the active transaction IDs are 300, 890, and 900. The generated transaction accessible table is as follows: Fig. 9 As shown. Fig. 9 As shown, the transaction ID range of the data accessible by this backup is 0-1000, excluding transaction IDs 300, 890, and 900. After obtaining the transaction accessible table, GCM will generate metadata for this backup based on the transaction ID.

[0102] GCM broadcasts the config configuration and the transaction accessible table of this backup to the GSTORE nodes it manages, and starts the timer task to monitor the backup status of each GSTORE. The GCM timer task will periodically obtain the backup status of the GSTORE node so that GCM can update the information of this backup in time, including the backup status, backup progress, backup end time and backup error code.

[0103] After receiving the backup configuration and transaction accessible table, each GSTORE node will generate local backup metadata, and then generate the ltask metadata asynchronous backup task and the ltask user data asynchronous backup top-level task; the metadata backup task carries the metadata that needs to be backed up this time, and its task responsibility is to back up the metadata of this full backup; at this time, GSTORE returns the broadcast result and starts running the metadata backup ltask and the user data top-level ltask. During the backup process, GCM receives the GSTORE broadcast recovery information and immediately returns it to the client so that the user can view the backup status of each GSTORE.

[0104] At this time, the metadata backup task and the user data backup task in the GSTORE node are executed concurrently. The metadata backup task writes the metadata of the backup target to the backup.meta file; and the ltask user data top-level task is derived into a 4-layer full backup task.

[0105] The specific implementation of the ltask task is as follows: Fig.10 shown.

[0106] (1) The ltask top-level backup task carries all graph objects and all information of this backup. When it is executed, it will derive a second-level backup task and is responsible for monitoring the execution status of the second-level backup task. If the second-level backup task of any graph library node fails, the top-level task will modify the backup status and notify other subtask threads to stop the backup immediately; once GCM obtains the backup failure status, it will notify other graph library nodes to stop the backup immediately.

[0107] (2) The second-layer backup task ltask is responsible for expanding the third-layer backup task. It carries the object of a graph to be backed up. When it is executed, it will obtain all the labels or relationships under the graph according to the graph object and derive the third-layer backup tasks one by one according to the labels or relationships.

[0108] (3) The third-layer backup task of ltask is responsible for scanning the label table or relationship table of the backup task of this layer, packaging the LSN of its data in batches, and deriving the fourth-layer backup task.

[0109] (4) ltask The fourth-layer backup task is responsible for obtaining specific data according to the LSN in the batch, generating backup Record data according to the unified coding, and storing the backup Record data batches on the disk. When performing a full backup, the unified coded XOP is INSERT.

[0110] During the backup process, users can query the status of the backup using the SHOW BACKUPS command. The backup results are as follows: Fig.11 If the backup task is completed, the status column will display FINISHED; if the backup task is not completed, the status column will display RUNNING; if the backup fails for some reason, the code column will display the error code. You can check the error code to find out what error caused the failure. When the backup task is completed correctly, the data in the backup file is as follows: Fig.12 shown.

[0111] The complete timing and data interaction process of the above backup method is as follows: Fig.13 As shown. The user inputs an asynchronous backup statement to the query engine, the query engine initiates a backup command to GCM, GCM broadcasts a transaction accessible table to GSTORE, and GSTORE completes the backup of metadata and user data based on the transaction accessible table. GSTORE feedbacks the backup task execution status to GCM, and GCM updates the backup information based on GSTROE. The user inputs a backup information query statement to the query engine, the query engine initiates an information query command to GCM, GCM confirms the current backup information, and feedbacks the backup information to the query engine. The query engine updates the backup information and feedbacks the backup information to the user.

[0112] Embodiment 3:

[0113] Based on the backup method based on the distributed graph database provided in Example 1, in some specific implementations, it can be implemented through the specific implementation in this embodiment. It can be understood that the specific implementation provided in this embodiment is only used to illustrate the specific implementation process of the method in Example 1 in certain specific scenarios, and is not intended to limit the scope of protection.

[0114] In the scenario of Example 2, the user initiated a full backup backup 1. In this embodiment, based on the full backup backup 1 of Example 2, an incremental backup is performed based on the backup result of Example 2.

[0115] The user initiates an incremental backup task using the following statement: BACKUP ALL DATABASES INCREMENT TObackup2 BASE ON backup1. This statement indicates that an incremental backup is performed based on the full backup package backup1. The name of this backup is backup2, and the transaction ID of this incremental backup is 1800.

[0116] Since this backup is an incremental backup, GCM will read the transaction ID of the last backup: 1000, and the active transaction ID of the last backup when generating the transaction accessible table, and generate the transaction accessible table based on the current active transaction and the active transaction of the last backup. After the comparison is completed, the transaction range in the transaction accessible table is [1000, 1800], where the active transactions in this interval need to be excluded and the transaction numbers of the active transactions in the last backup that have ended are included. After obtaining the transaction accessible table, GSTORE will scan and back up the data based on the incremental transaction accessible table generated this time.

[0117] On the other hand, in incremental backup, when backing up metadata, the metadata bakcup.mate file of the last backup will be obtained first according to the name of the last backup task backup1. By comparing the metadata information with the existing system metadata information, the incremental changes in the metadata since the last backup can be obtained. Subsequently, the metadata backup task will generate a backup Record of the metadata, whose XOP is one of UPDATE, DELETE or INSERT, indicating that this metadata has been modified, deleted, or newly added. The user data ltask will scan the data according to the incremental transaction accessible table based on the transaction of the last backup, and then generate the ltask multi-layer task, and complete the backup task with reference to the backup method in Example 2.

[0118] Embodiment 4:

[0119] Based on the backup methods based on the distributed graph database provided in the above-mentioned embodiments 1 to 3, the present invention also provides a backup device based on the distributed graph database that can be used to implement the above-mentioned method, such as Fig.14 , which is a schematic diagram of the device architecture of an embodiment of the present invention. The backup device based on the distributed graph database of this embodiment includes one or more processors 11 and a memory 12. Fig.14 A processor 11 is taken as an example.

[0120] The processor 11 and the memory 12 may be connected via a bus or other means. Fig.14 The example of connecting through bus is taken in the following.

[0121] The memory 12 is a non-volatile computer-readable storage medium for a backup method based on a distributed graph database, and can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the backup method based on a distributed graph database in Examples 1 to 3. The processor 11 executes various functional applications and data processing of the backup device based on the distributed graph database by running the non-volatile software programs, instructions and modules stored in the memory 12, that is, implementing the backup method based on the distributed graph database in Examples 1 to 3.

[0122] The memory 12 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 12 may optionally include a memory remotely arranged relative to the processor 11, and these remote memories may be connected to the processor 11 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0123] The program instructions / modules are stored in the memory 12. When executed by one or more processors 11, the backup method based on the distributed graph database in the above-mentioned embodiments 1 to 3 is executed. For example, the above-described Figure 1 , Figure 5 and Figure 6 The steps shown.

[0124] An embodiment of the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors, such as Fig.14 A processor 11 in the embodiment 1 can enable the above one or more processors to execute the backup method based on the distributed graph database in the embodiment 1 to the embodiment 3, for example, to execute the above described Figure 1 , Figure 5 and Figure 6 The steps shown can also be implemented Figure 8 , Fig.10 The various modules and units described.

[0125] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0126] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0127] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A backup method based on a distributed graph database, characterized in that: include: Generate the corresponding transaction accessible table according to the backup command, and broadcast the transaction accessible table to all graph library nodes that need to be backed up; Each graph library node generates the top-level backup task on the graph library node based on the transaction accessible table, and derives the backup tasks of each level layer by layer through the top-level backup task; Each gallery node asynchronously executes the bottom-level backup task, obtains the user data for backup through the bottom-level backup task, and backs up the user data on the gallery node.

2. The backup method based on a distributed graph database according to claim 1, characterized in that: The generating of the corresponding transaction accessible table according to the backup command specifically includes: When performing a full backup, collect statistics on the transaction IDs related to the backup command, obtain all visible transaction numbers based on the statistical results, and write all visible transaction numbers into the transaction accessible table; When performing an incremental backup, obtain all visible transaction numbers according to the backup command, obtain the previous visible transaction number according to the transaction accessible table at the time of the previous backup, compare all visible transaction numbers with the previous visible transaction number to obtain the current visible transaction number, and write the current visible transaction number into the transaction accessible table.

3. The backup method based on a distributed graph database according to claim 1, characterized in that: Each graph library node generates a top-level backup task on the graph library node according to the transaction accessible table, specifically including: The graph library node creates a backup task for each graph and takes the backup task set of all graphs as the top-level backup task; A query engine is established in the top-level backup task to query the execution status of the backup task of each graph.

4. The backup method based on a distributed graph database according to claim 1, characterized in that: The backup tasks of each level are derived layer by layer from the top-level backup task, specifically including: Get the graph corresponding to the user data in the top-level backup task, and treat the backup task of each graph as a second-level backup task; Obtain the labels and relationships of the graph in each layer 2 backup task, and treat the backup task of the labels and relationships in each graph as a layer 3 backup task; Obtain the log sequence number of the data corresponding to the label or relationship in each three-layer backup task, and treat the backup task of each batch of log sequence numbers as a four-layer backup task; The backup tasks at each level are organized in a tree-like relationship, and a query engine is established in each backup task layer so that the backup tasks at the upper level can query the status of the backup tasks at the lower level.

5. The backup method based on a distributed graph database according to claim 1, characterized in that: The user data for backup is obtained through the backup task at the lowest level, and the user data on the current gallery node is backed up, specifically including: Get each log sequence number in the bottom-level backup task, and get the corresponding user data based on the log sequence number; The obtained user data is uniformly encoded according to the specified format, and the encoded binary stream is written into the corresponding backup file.

6. The backup method based on a distributed graph database according to claim 5, characterized in that: The obtaining of corresponding user data according to the log sequence number also includes: Each graph library node obtains the point data and edge data in the user data stored on the graph library node according to the log sequence number; When the number of edge data for which end point data needs to be queried reaches a specified number among the acquired edge data, the unique identifiers of the end point data to be queried are sent in batches to other graph library nodes to obtain the corresponding end point data.

7. The backup method based on a distributed graph database according to claim 5, characterized in that: The obtained user data is uniformly encoded according to a specified format, specifically including: Generate a corresponding operation code for each piece of user data according to the backup method and the recovery operation corresponding to each piece of user data, and use the operation code as the header field of the piece of user data; The key-value pair of each piece of user data is used as the data field of the piece of user data; The header field and data field of each piece of user data are combined into the encoding of the piece of user data.

8. The backup method based on a distributed graph database according to claim 1, characterized in that: The method further comprises: Generate corresponding metadata information according to the backup command, and broadcast the metadata information to all gallery nodes that need to be backed up; Each library node generates a metadata backup task on the library node according to the metadata information, and backs up the metadata through the metadata backup task.

9. A backup device based on a distributed graph database, characterized in that: It includes at least one processor and a memory, the at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by the at least one processor, and the instructions, after being executed by the processor, are used to complete the backup method based on a distributed graph database as described in any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the backup method based on the distributed graph database as described in any one of claims 1-8.

Images