Database synchronization method based on directed acyclic graph

Abstract

The invention relates to the technical field of electric digital data processing, in particular to a database synchronization method based on a directed acyclic graph. The method comprises the steps of obtaining a target log table; traversing each record of the target log table according to the sequence of the corresponding operation sequence numbers from large to small, and obtaining an initial directed acyclic graph corresponding to the target log table; if the number of the first-layer nodes in the initial directed acyclic graph is greater than a first preset number, selecting the first-layer nodes with the first preset number and adding the first-layer nodes into a ready queue; if the number of nodes included in the initial directed acyclic graph is larger than the number of first-layer nodes selected to be added into the ready queue in the initial directed acyclic graph, obtaining a first updated directed acyclic graph, and if the number of the first-layer nodes in the first updated directed acyclic graph is larger than a first preset number, obtaining a second updated directed acyclic graph; and if yes, selecting a first preset number of first-layer nodes from the first updated directed acyclic graph and adding the first-layer nodes into a ready queue. According to the method, the database synchronization efficiency can be improved, and the synchronization delay is reduced.

Description

Technical Field

[0001] The invention relates to the technical field of electronic digital data processing, and in particular to a database synchronization method based on a directed acyclic graph. Background Art

[0002] Database synchronization refers to executing the changes in the data in the primary database in the standby database to ensure that the data in the standby database is completely consistent with the data in the primary database. When performing database synchronization, in order to make the changes in the primary database complete in the standby database in a timely manner, for a table in the database with only one unique constraint field, the value of the unique constraint field corresponding to each operation is used as the corresponding key value, and then mapped to different buckets. Thus, β operations can be executed in parallel each time, where β is the number of buckets.

[0003] The above method of mapping to different buckets is more suitable for tables with only one unique constraint field. However, for tables with multiple unique constraint fields, the above method of mapping to different buckets is not applicable because the corresponding operations are interdependent. The prior art usually adopts the serial synchronization method, that is, only one operation is executed each time. The above serial synchronization method results in low synchronization efficiency for the database and high synchronization delay. Summary of the invention

[0004] The present invention aims to provide a database synchronization method based on a directed acyclic graph to improve the synchronization efficiency of the database and reduce the synchronization delay.

[0005] According to the present invention, a database synchronization method based on a directed acyclic graph is provided, the method comprising the following steps:

[0006] S100, obtain a target log table; the target log table is used to record operations performed on a table in the first database within a target historical time period; the target log table includes a plurality of records, each record corresponds to an operation, and each record includes a corresponding operation serial number, a corresponding table name, a corresponding operation name, a value of a primary unique key of the corresponding table, values ​​of other unique keys of the corresponding table, and a serial number of a corresponding dependent operation.

[0007] S200, traverse each record of the target log table in descending order of the corresponding operation sequence number, and obtain an initial directed acyclic graph corresponding to the target log table; the initial directed acyclic graph includes several layers of nodes, each layer of nodes includes several nodes, each node represents an operation sequence number, and different nodes represent different operation sequence numbers; the operation corresponding to any child node in the initial directed acyclic graph depends on the operation corresponding to the corresponding parent node, and the operation corresponding to any first-layer node in the initial directed acyclic graph does not depend on the operation corresponding to any other node in the initial directed acyclic graph.

[0008] S300, if the number of first-layer nodes in the initial directed acyclic graph is greater than the first preset number, a first preset number of first-layer nodes are selected from the initial directed acyclic graph to add to the ready queue, so as to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed.

[0009] S400, if the number of nodes included in the initial directed acyclic graph is greater than the number of first-layer nodes selected to be added to the ready queue in the initial directed acyclic graph, obtain the first updated directed acyclic graph and enter S500; the first updated directed acyclic graph is a directed acyclic graph obtained after deleting the first-layer nodes selected to be added to the ready queue from the initial directed acyclic graph; the operation corresponding to any first-layer node in the first updated directed acyclic graph does not depend on the operation corresponding to any other node in the first updated directed acyclic graph.

[0010] S500, if the number of first-layer nodes in the first updated directed acyclic graph is greater than the first preset number, a first preset number of first-layer nodes are selected from the first updated directed acyclic graph to add to the ready queue, so as to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed.

[0011] Compared with the prior art, the present invention has at least the following beneficial effects:

[0012] The present invention obtains a corresponding initial directed acyclic graph according to a target log table, wherein the initial directed acyclic graph is obtained according to the target log table, wherein each node included in the initial directed acyclic graph represents an operation sequence number corresponding to each record in the target log table, the first-layer nodes in the initial directed acyclic graph represent operation sequence numbers corresponding to operations in the target log table that do not depend on any operations, and the operations corresponding to the child nodes in the initial directed acyclic graph depend on the operations corresponding to the corresponding parent nodes; on the basis of obtaining the initial directed acyclic graph, the present invention adds multiple first-layer nodes in the initial directed acyclic graph to a ready queue at one time, so as to The two databases concurrently execute operations corresponding to all first-layer nodes in the ready queue, thereby improving the efficiency of synchronization with the second database and reducing synchronization delays; if not all nodes in the initial directed acyclic graph are added to the ready queue, then a directed acyclic graph consisting of the remaining nodes is further obtained, that is, a first updated directed acyclic graph; for the first updated directed acyclic graph, multiple first-layer nodes therein are also added to the ready queue at one time, so as to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, thereby further improving the efficiency of synchronization with the second database and reducing synchronization delays. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 A flowchart of a database synchronization method based on a directed acyclic graph provided in Embodiment 1 of the present invention;

[0015] Figure 2 A schematic diagram of an initial directed acyclic graph provided in Embodiment 1 of the present invention;

[0016] Figure 3 A schematic diagram of a first updated directed acyclic graph provided in Embodiment 1 of the present invention;

[0017] Figure 4 A schematic diagram of a second updated directed acyclic graph provided in Embodiment 1 of the present invention;

[0018] Figure 5 A schematic diagram of a third updated directed acyclic graph provided in Embodiment 1 of the present invention;

[0019] Figure 6 A schematic diagram of a fourth updated directed acyclic graph provided in Embodiment 1 of the present invention;

[0020] Figure 7A schematic diagram of a fifth updated directed acyclic graph provided in Embodiment 1 of the present invention;

[0021] Figure 8 A schematic diagram of a first updated directed acyclic graph provided in Embodiment 2 of the present invention;

[0022] Fig. 9 A schematic diagram of a second updated directed acyclic graph provided in Embodiment 2 of the present invention;

[0023] Fig.10 A schematic diagram of a third updated directed acyclic graph provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0025] Embodiment 1:

[0026] According to this embodiment, Figure 1 As shown, a database synchronization method based on a directed acyclic graph is provided, and the method comprises the following steps:

[0027] S100, obtain a target log table; the target log table is used to record operations performed on a table in the first database within a target historical time period; the target log table includes a plurality of records, each record corresponds to an operation, and each record includes a corresponding operation serial number, a corresponding table name, a corresponding operation name, a value of a primary unique key of the corresponding table, values ​​of other unique keys of the corresponding table, and a serial number of a corresponding dependent operation.

[0028] In this embodiment, operations performed on the table in the first database during the target historical time period have not yet been synchronized in the second database.

[0029] As a specific implementation, the operation name corresponding to each record is insert new data (insert new) or delete old data (delete old) or delete old data during update (update delete old) or insert new data during update (update insert new). Among them, the delete old data operation during update and the insert new data operation during update together constitute the update operation, and the delete old data operation during update and the insert new data operation during update included in the same update operation share the same operation sequence number.

[0030] As a specific implementation, the first record in the target log table includes an operation number of 1; if the operation corresponding to a record in the target log table does not depend on any other operation, the sequence numbers of the dependent operations corresponding to the record are all composed of 0, and the number of 0 is the number of unique keys of the corresponding table. Those skilled in the art know that no duplicate values ​​are allowed to appear in the column corresponding to any unique key of the table.

[0031] As a specific implementation, the target log table is shown in Table 1, wherein the id field corresponds to the operation serial number, the table field corresponds to the table name, the op field corresponds to the operation name, the PK field corresponds to the value of the primary unique key of the table, the UK field corresponds to the value of the other unique key of the table, and the dependent field corresponds to the serial number of the dependent operation.

[0032] Table 1

[0033] id table op PK UK rely 1 A Insert_new 1 1,1 0,0,0 2 A Insert_new 2 2,2 0,0,0 3 A Insert_new 3 3,3 0,0,0 4 A Update_Delete_old 1 1,1 1,1,1 4 A UPDate_Insert_new 1 4,4 4,0,0 5 A Update_Delete_old 2 2,2 2,2,2 5 A UPDate_Insert_new 2 5,5 5,0,0 6 A Delete_old 3 3,3 3,3,3 7 A Insert_new 3 2,1 6,5,4 8 B Insert_new 11 john 0,0 9 B Insert_new 12 james 0,0 10 B Insert_new 13 david 0,0 11 B Update_Delete_old 11 john 8,8 11 B UPDate_Insert_new 21 john 0,11 12 B Insert_new 11 sophia 11,0 13 B Update_Delete_old 21 john 0,11 13 B UPDate_Insert_new 21 ava 13,0 14 B Insert_new 14 john 0,13 15 B Delete_old 12 james 9,9 16 B Insert_new 15 michael 0,0

[0034] In Table 1, the value of the id field of the first record in the target log table is 1, indicating that the operation sequence number corresponding to the record is 1, that is, the operation corresponding to the record is the first operation performed on the first database in the target historical time period; the value of the table field of the first record in the target log table is A, indicating that the table name corresponding to the record is A, that is, the operation corresponding to the record is an operation in table A; the value of the op field of the first record in the target log table is Insert_new, indicating that the operation name corresponding to the record is Insert_new, that is, the operation corresponding to the record is an insert operation; the first record in the target log table The value of the PK field of the record is 1 and the value of the UK field of the first record in the target log table is 1,1, indicating that the value of the primary unique key of the table corresponding to the record is 1 and the values ​​of other unique keys are 1,1, that is, the operation corresponding to the record is to insert a record into table A (table A includes 2 other unique keys in addition to the primary unique key), and the value of the primary unique key of the inserted record is 1 and the values ​​of the other two unique keys are 1 respectively; the value of the dependent field of the first record in the target log table is 0,0,0, indicating that the sequence number of the dependent operation corresponding to the record is 0,0,0, that is, the operation corresponding to the record does not depend on any other operation.

[0035] In Table 1, the value of the id field of the 15th record in the target log table is 15, indicating that the operation sequence number corresponding to the record is 15, that is, the operation corresponding to the record is the 15th operation performed on the first database in the target historical time period; the value of the table field of the 15th record in the target log table is B, indicating that the table name corresponding to the record is B, that is, the operation corresponding to the record is an operation in table B; the value of the op field of the 15th record in the target log table is Delete_old, indicating that the operation name corresponding to the record is Delete_old, that is, the operation corresponding to the record is a delete operation; the 15th record in the target log table The value of the PK field of the record is 12 and the value of the UK field of the 15th record in the target log table is james, which means that the value of the primary unique key of the table corresponding to the record is 12 and the value of the other unique key is james, that is, the operation corresponding to the record is to delete a record in table B (table B includes 1 other unique key in addition to the primary unique key), and the value of the primary unique key of the deleted record is 12 and the value of the other unique key is james; the value of the dependent field of the 15th record in the target log table is 9,9, which means that the serial number of the dependent operation corresponding to the record is 9,9, that is, the operation corresponding to the record depends on the operation with the serial number 9.

[0036] S200, traverse each record of the target log table in descending order of the corresponding operation sequence number, and obtain an initial directed acyclic graph corresponding to the target log table; the initial directed acyclic graph includes several layers of nodes, each layer of nodes includes several nodes, each node represents an operation sequence number, and different nodes represent different operation sequence numbers; the operation corresponding to any child node in the initial directed acyclic graph depends on the operation corresponding to the corresponding parent node, and the operation corresponding to any first-layer node in the initial directed acyclic graph does not depend on the operation corresponding to any other node in the initial directed acyclic graph.

[0037] In this embodiment, any i+1th layer node in the initial directed acyclic graph is a child node of several ith layer nodes, the value range of i is 1 to n-1, and n is the number of layers included in the initial directed acyclic graph.

[0038] In this example, if the dependent operation number corresponding to the record corresponding to the first operation number includes the second operation number, then the node representing the second operation number is determined as the parent node of the node representing the first operation number, that is, the node representing the first operation number is determined as the child node of the node representing the second operation number. As a specific implementation, the initial directed acyclic graph corresponding to Table 1 is as follows: Figure 2As shown, the nodes corresponding to 1, 2, 3, 8, 9, 10 and 16 are first-layer nodes, the nodes corresponding to 4, 5, 6, 11 and 15 are second-layer nodes, the nodes corresponding to 7, 12 and 13 are third-layer nodes, the node corresponding to 14 is a fourth-layer node, the node corresponding to 1 is the parent node of the node corresponding to 4 (that is, the node corresponding to 4 is the child node of the node corresponding to 1), the node corresponding to 2 is the parent node of the node corresponding to 5, the node corresponding to 3 is the parent node of the node corresponding to 6, the node corresponding to 4, the node corresponding to 5 and the node corresponding to 6 are all parent nodes of the node corresponding to 7, the node corresponding to 8 is the parent node of the node corresponding to 11, the node corresponding to 11 is the parent node of the node corresponding to 12 and the node corresponding to 13, the node corresponding to 13 is the parent node of the node corresponding to 14, and the node corresponding to 9 is the parent node of the node corresponding to 15.

[0039] S300, if the number of first-layer nodes in the initial directed acyclic graph is greater than the first preset number, a first preset number of first-layer nodes are selected from the initial directed acyclic graph to add to the ready queue, so as to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed.

[0040] In this embodiment, if the number of first-layer nodes in the initial directed acyclic graph is less than or equal to a first preset number, all first-layer nodes in the initial directed acyclic graph are added to the ready queue to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and the ready queue is cleared after the execution is completed.

[0041] In this embodiment, the first preset number is a preset concurrent processing number. Optionally, a first preset number of first-layer nodes are randomly selected from the initial directed acyclic graph to add to the ready queue.

[0042] S400, if the number of nodes included in the initial directed acyclic graph is greater than the number of first-layer nodes selected to be added to the ready queue in the initial directed acyclic graph, obtain the first updated directed acyclic graph and enter S500; the first updated directed acyclic graph is a directed acyclic graph obtained after deleting the first-layer nodes selected to be added to the ready queue from the initial directed acyclic graph; the operation corresponding to any first-layer node in the first updated directed acyclic graph does not depend on the operation corresponding to any other node in the first updated directed acyclic graph.

[0043] In this embodiment, any first-layer node in the first updated directed acyclic graph satisfies the following conditions: it is a first-layer node in the initial directed acyclic graph, or it is not a first-layer node in the initial directed acyclic graph and all the corresponding parent nodes in the initial directed acyclic graph have been selected to be added to the ready queue.

[0044] In this embodiment, if the number of nodes included in the initial directed acyclic graph is equal to the number of first-layer nodes selected to be added to the ready queue in the initial directed acyclic graph, it is determined that the synchronization process for the second database is completed and subsequent steps are no longer executed.

[0045] In this embodiment, the first layer nodes selected to be added to the ready queue in the initial directed acyclic graph are different, and the first updated directed acyclic graph obtained is different; as a specific implementation method, the first preset number is 4, Figure 2 The nodes corresponding to 1, 9, 10 and 16 in are added to the ready queue, so the first updated directed acyclic graph is as follows Figure 3 shown.

[0046] S500, if the number of first-layer nodes in the first updated directed acyclic graph is greater than the first preset number, a first preset number of first-layer nodes are selected from the first updated directed acyclic graph to add to the ready queue, so as to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed.

[0047] Optionally, a first preset number of first-layer nodes are randomly selected from the first updated directed acyclic graph to add to the ready queue.

[0048] In this embodiment, if the number of first-layer nodes in the first updated directed acyclic graph is less than or equal to a first preset number, all first-layer nodes in the first updated directed acyclic graph are added to the ready queue to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and the ready queue is cleared after the execution is completed.

[0049] This embodiment obtains the corresponding initial directed acyclic graph according to the target log table, and the initial directed acyclic graph is obtained according to the target log table, wherein each node included in the initial directed acyclic graph represents the operation sequence number corresponding to each record in the target log table, the first-layer nodes in the initial directed acyclic graph represent the operation sequence number corresponding to the operation in the target log table that does not depend on any operation, and the operation corresponding to the child node in the initial directed acyclic graph depends on the operation corresponding to the corresponding parent node; based on the acquisition of the initial directed acyclic graph, this embodiment adds multiple first-layer nodes in the initial directed acyclic graph to the ready queue at one time to The two databases concurrently execute the operations corresponding to all the first-layer nodes in the ready queue. Therefore, this embodiment improves the efficiency of synchronization with the second database and reduces the delay of synchronization. If not all the nodes in the initial directed acyclic graph are added to the ready queue, the directed acyclic graph consisting of the remaining nodes is further obtained, that is, the first updated directed acyclic graph. For the first updated directed acyclic graph, multiple first-layer nodes are also added to the ready queue at one time to concurrently execute the operations corresponding to all the first-layer nodes in the ready queue on the second database. Therefore, this embodiment further improves the efficiency of synchronization with the second database and reduces the delay of synchronization.

[0050] In this embodiment, after S500, the method further includes the following steps:

[0051] S600, if the number of nodes included in the first updated directed acyclic graph is equal to the number of first-layer nodes selected to be added to the ready queue in the first updated directed acyclic graph, it is determined that the synchronization process of the second database is ended; if the number of nodes included in the first updated directed acyclic graph is greater than the number of first-layer nodes selected to be added to the ready queue in the first updated directed acyclic graph, the second updated directed acyclic graph is obtained and enters S700; the second updated directed acyclic graph is a directed acyclic graph obtained after deleting the first-layer nodes selected to be added to the ready queue from the first updated directed acyclic graph; the operation corresponding to any first-layer node in the second updated directed acyclic graph does not depend on the operation corresponding to any other node in the second updated directed acyclic graph.

[0052] In this embodiment, if the first preset number of first-layer nodes selected from the first updated directed acyclic graph is different, the obtained second updated directed acyclic graph is different; as a specific implementation method, the first updated directed acyclic graph is as follows: Figure 3 As shown, the first preset number is 4. Figure 3 The nodes corresponding to 2, 3, 4 and 15 in are added to the ready queue, and the second updated directed acyclic graph is as follows Figure 4 shown.

[0053] S700, if the number of first-layer nodes in the second updated directed acyclic graph is greater than the first preset number, select the first preset number of first-layer nodes from the second updated directed acyclic graph to add to the ready queue, so as to concurrently execute the operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed.

[0054] In this embodiment, if the number of first-layer nodes in the second updated directed acyclic graph is less than or equal to the first preset number, all first-layer nodes in the second updated directed acyclic graph are added to the ready queue to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and the ready queue is cleared after the execution is completed.

[0055] In this embodiment, after S700, the method further includes the following steps:

[0056] S800: If the number of nodes included in the second updated directed acyclic graph is equal to the number of first-layer nodes selected to be added to the ready queue in the second updated directed acyclic graph, it is determined that the synchronization process for the second database is terminated.

[0057] In this embodiment, if the number of nodes included in the second updated directed acyclic graph is greater than the number of first-layer nodes selected to be added to the ready queue in the second updated directed acyclic graph, a third updated directed acyclic graph is obtained, and it is determined whether the number of first-layer nodes in the third updated directed acyclic graph is greater than the first preset number. If it is greater than, the first preset number of first-layer nodes are selected from the third updated directed acyclic graph to be added to the ready queue, so as to concurrently execute the operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed; otherwise, all first-layer nodes in the third updated directed acyclic graph are added to the ready queue, so as to concurrently execute the operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed. If the number of nodes included in the third updated directed acyclic graph is greater than the number of first-layer nodes selected to be added to the ready queue in the third updated directed acyclic graph, a fourth updated directed acyclic graph is obtained. By analogy, until the operations corresponding to all records included in the target log table are executed in the second database, the synchronization process ends.

[0058] As a specific implementation, the second updated directed acyclic graph is as follows Figure 4 As shown, the first preset number is 4. Figure 4 The nodes corresponding to 5, 6, and 8 in are added to the ready queue, and the third updated directed acyclic graph is obtained as follows Figure 5 As shown; Figure 5 The nodes corresponding to 7 and 11 in are added to the ready queue, and the fourth updated directed acyclic graph is obtained as follows Figure 6 As shown; Figure 6The nodes corresponding to 12 and 13 in are added to the ready queue, and the fifth updated directed acyclic graph is obtained as follows Figure 7 As shown; Figure 7 The node corresponding to 14 in the target log table is added to the ready queue, then all operations corresponding to the records included in the target log table are executed in the second database, and the synchronization process ends; for the target log table shown in Table 1, the synchronization process of this specific implementation includes 5 waiting times for other operations to be completed. Compared with the synchronization process using the serial synchronization method in the prior art, which requires 15 waiting times for other operations to be completed, this specific implementation improves the synchronization efficiency of the second database and reduces the synchronization delay.

[0059] Embodiment 2:

[0060] In the above-mentioned first embodiment, the first preset number of first-layer nodes selected from the initial directed acyclic graph to join the ready queue are randomly selected. Although the synchronization efficiency of the second database can be improved compared with the serial synchronization method in the prior art, it still cannot maximize the synchronization efficiency of the second database. In order to further improve the synchronization efficiency of the second database, this embodiment, on the basis of the first embodiment, further optimizes the process of selecting the first preset number of first-layer nodes from the initial directed acyclic graph to join the ready queue. Specifically, the process of selecting the first preset number of first-layer nodes from the initial directed acyclic graph to join the ready queue includes:

[0061] S310, obtaining the descendant node complexity f of the jth first-layer node in the initial directed acyclic graph j , f j =∑ T(j) t＝1 (∏ Q(t) q＝1 num t,q ), num t,q is the number of parent nodes of the qth node of the tth branch connected to the jth first-layer node in the initial directed acyclic graph, the value range of q is 1 to Q(t), Q(t) is the number of nodes included in the tth branch connected to the jth first-layer node in the initial directed acyclic graph; the value range of t is 1 to T(j), T(j) is the number of branches connected to the jth first-layer node in the initial directed acyclic graph; the value range of j is 1 to m, m is the number of first-layer nodes in the initial directed acyclic graph.

[0062] like Figure 2As shown, the number of branches connected to the nodes corresponding to 10 and 16 is 0, the number of branches connected to the nodes corresponding to 1, 2, 3 and 9 is all 1, and the number of branches connected to 8 is 2; wherein, the first node included in the branch connected to the node corresponding to 1 is the node corresponding to 4, and the second node included in the branch connected to the node corresponding to 1 is the node corresponding to 7; the first node included in the branch connected to the node corresponding to 2 is the node corresponding to 5, and the second node included in the branch connected to the node corresponding to 2 is the node corresponding to 7; the first node included in the branch connected to the node corresponding to 3 is the node corresponding to 6, and the second node included in the branch connected to the node corresponding to 3 is the node corresponding to 7; the node corresponding to 9 The first node included in the connected branch is the node corresponding to 15; the first node included in the first branch connected to the node corresponding to 8 is the node corresponding to 11, and the second node included in the first branch connected to the node corresponding to 8 is the node corresponding to 12; the first node included in the second branch connected to the node corresponding to 8 is the node corresponding to 11; the second node included in the second branch connected to the node corresponding to 8 is the node corresponding to 13; the third node included in the second branch connected to the node corresponding to 8 is the node corresponding to 14; the number of parent nodes of the nodes corresponding to 4, 5, 6, 11, 12, 13, 14 and 15 is all 1, and the number of parent nodes of the node corresponding to 7 is 3.

[0063] S320, obtaining the number s of unbound nodes of the jth first-layer node in the initial directed acyclic graph j ; If the jth first-layer node in the initial directed acyclic graph does not have a child node, then s j = 0; if the jth first-layer node in the initial directed acyclic graph has a child node, and there are z children in the jth first-layer node in the initial directed acyclic graph j There is only one parent node for each child node, then s j =z j .

[0064] In this embodiment, the number of unbound nodes of any first-layer node is the number of newly added first-layer nodes after the corresponding first-layer node is deleted. Figure 2 As shown, the number of unbound nodes corresponding to nodes 10 and 16 is 0, and the number of unbound nodes corresponding to nodes 1, 2, 3, 8 and 9 is 1.

[0065] S330, obtaining the first priority value y of the jth first-layer node in the initial directed acyclic graph j ;y j With f j Positive correlation, y j With s j Positive correlation.

[0066] As a preferred embodiment, j=w1×f' j +w2×s' j , w1 and w2 are the weights corresponding to the complexity of descendant nodes and the number of unbound nodes, respectively, f' j and s' j are the normalized f j and the normalized s j , w1 and w2 are both greater than 0 and less than 1, w1+w2=1. The process of obtaining w2 includes:

[0067] S331, obtaining the number g of first-layer nodes in the initial directed acyclic graph whose number of unbound nodes is 0.

[0068] S332, if m-min(h,g)≥4, the first preset weight value is determined to be w2; otherwise, the second preset weight value is determined to be w2; the first preset weight and the second preset weight are both greater than 0 and less than 1, and the second preset weight is greater than the first preset weight; min() is the minimum value, and h is the first preset number.

[0069] Optionally, the first preset weight and the second preset weight are both empirical values; for example, the first preset weight is 0.5, and the second preset weight is 0.7.

[0070] Based on the above preferred specific implementation, the value of w2 is related to the number of first-layer nodes in the initial directed acyclic graph whose number of unbound nodes is 0. If the first-layer nodes in the initial directed acyclic graph whose number of unbound nodes is 0 are preferentially added to the ready queue at one time, it will cause the number of nodes in the remaining nodes of the initial directed acyclic graph that can be added to the ready queue at one time next time to be less than the first preset value. In this way, a larger value is assigned to w2 so that the first-layer nodes in the initial directed acyclic graph whose number of unbound nodes is not 0 are preferentially added to the ready queue, thereby achieving the purpose of increasing the number of nodes in the remaining nodes of the initial directed acyclic graph that can be added to the ready queue at one time next time. Since the number of operations that can be executed concurrently next time is increased, this preferred specific implementation can further speed up the processing speed of operations corresponding to nodes in the initial directed acyclic graph.

[0071] S340, adding a first preset number of first-layer nodes having the largest first priority value in the initial directed acyclic graph to a ready queue.

[0072] As a preferred specific implementation, S340 includes:

[0073] S341, sorting the first layer nodes in the initial directed acyclic graph in descending order according to the corresponding first priority values ​​to obtain a first node sequence D; D = [d1, d2, ..., d r ,…,d m ],d ris the rth first-layer node in D, and the value of r ranges from 1 to m.

[0074] S342, if d h The first priority value is equal to d h+1 The first priority value is entered into S343; h is the first preset number.

[0075] S343, if d h If the first priority value of is 0, the first h first-layer nodes in D are determined as the first first preset number of first-layer nodes with the largest first priority value in the initial directed acyclic graph; otherwise, enter S344.

[0076] S344, obtain the first priority value in D and d h The set E of first-layer nodes with the same first priority value, E={e1,e2,…,e x ,…,e R},e x is the xth first-layer node in E, where x ranges from 1 to R, and R is the first priority value in D and d h The first priority value is equal to the number of first layer nodes.

[0077] S345, traverse E, get e x The second priority value of e x The second priority value is e x The maximum value of the first priority value of all child nodes.

[0078] In this embodiment, if e x There is only one child node, so e x The second priority value is e x The first priority value of the child node; if e x There are more than 2 child nodes, then e x The second priority value is e x The maximum value of the first priority value of all child nodes. x The first priority value of any child node is also based on e x The complexity of descendant nodes and the number of unbound nodes of the corresponding child node are obtained, which will not be repeated here.

[0079] S346, sort the first-layer nodes in E in descending order according to the corresponding second priority values ​​to obtain a second node sequence C; C = [c1, c2, ..., c v ,…,c R ], c v is the vth first-layer node in C, and the value of v ranges from 1 to R.

[0080] S347, if c h-lThe second priority value is not equal to c h-l+1 The first hl first-layer nodes in C and the first l first-layer nodes in D are determined as the first first preset number of first-layer nodes with the largest first priority value in the initial directed acyclic graph; l is the number of nodes in D with the first priority value greater than d h The number of first-layer nodes.

[0081] In this embodiment, if c h-l The second priority value is equal to c h-l+1 The second priority value of is processed in a similar manner to S343-S347, that is, first determine c h-l Is the second priority value of equal to 0? If so, the first hl first-layer nodes in C and the first l first-layer nodes in D are determined as the first preset number of first-layer nodes with the largest first priority value in the initial directed acyclic graph; if not, the second priority value in C and the first priority value in D are obtained. h-l A set F of first-layer nodes whose second priority values ​​are equal to , and then sort the first-layer nodes in F according to the third priority value of each first-layer node in F to obtain a third node sequence G; if the third priority value of the hl-nth first-layer node in G is not equal to the third priority value of the hl-n+1th first-layer node, then the first hl-nth first-layer nodes in G, the first nth first-layer nodes in C, and the first lth first-layer nodes in D are determined as the first first preset number of first-layer nodes with the largest first priority value in the initial directed acyclic graph; n is the number of first-layer nodes in C with a second priority value greater than c h-l If the third priority value of the hl-nth first-layer node in G is equal to the third priority value of the hl-n+1th first-layer node, then the process is similar to S343-S347, which will not be repeated here. The third priority value of any first-layer node is the maximum value of the first priority values ​​of all grandchild nodes of the corresponding first-layer node.

[0082] In this embodiment, the process of selecting a first preset number of first-layer nodes from any updated directed acyclic graph to add to the ready queue is similar to the above-mentioned process of selecting a first preset number of first-layer nodes from the initial directed acyclic graph to add to the ready queue, and will not be repeated here.

[0083] As a specific implementation, the initial directed acyclic graph is as follows Figure 2 As shown, the first preset number is 4. The first layer nodes added to the ready queue in the initial directed acyclic graph determined by the method of this embodiment are nodes corresponding to 1, 2, 3 and 8. Then the first updated directed acyclic graph is as follows: Figure 8As shown; According to the method of this embodiment, the first updated directed acyclic graph determined by the method of this embodiment is added to the ready queue. The first layer of nodes is the nodes corresponding to 4, 5, 9 and 11, then the second updated directed acyclic graph is obtained as follows Fig. 9 As shown; according to the method of this embodiment, the first layer nodes added to the ready queue in the second updated directed acyclic graph are nodes corresponding to 6, 10, 12 and 13, then the third updated directed acyclic graph is obtained as Fig.10 As shown; Fig.10 The nodes corresponding to 7, 14, 15 and 16 in the target log table are added to the ready queue, then the operations corresponding to all the records included in the target log table are executed in the second database, and the synchronization process ends; for the target log table shown in Table 1, the synchronization process of this specific implementation method includes 3 waiting times for other operations to be completed. Compared with the situation in Example 1 where 5 waiting times for other operations to be completed are required, this specific implementation method further improves the synchronization efficiency of the second database and reduces the synchronization delay.

[0084] In addition to the advantages of embodiment one, this embodiment also obtains the complexity of descendant nodes and the number of unbound nodes of each first-layer node in the initial directed acyclic graph, and determines the first priority value of the corresponding first-layer node according to the complexity of the descendant nodes and the number of unbound nodes. The larger the complexity of the descendant nodes, the larger the corresponding first priority value, and the larger the number of unbound nodes, the larger the corresponding first priority value. Therefore, this embodiment adds the first preset number of first-layer nodes with the largest first priority values ​​in the initial directed acyclic graph to the ready queue, so that the operations corresponding to the first-layer nodes with large descendant node complexity and large number of unbound nodes are executed first, avoiding serious node blocking caused by the fact that the operations corresponding to the first-layer nodes with large descendant node complexity and large number of unbound nodes are not executed, which is beneficial to increasing the number of nodes in the remaining nodes of the initial directed acyclic graph that can be added to the ready queue at one time next time, thereby improving the processing speed of operations corresponding to the nodes in the initial directed acyclic graph.

[0085] Although some specific embodiments of the present invention have been described in detail by way of example, it will be appreciated by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will also be appreciated by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A database synchronization method based on a directed acyclic graph, characterized in that: The method comprises the following steps: S100, obtaining a target log table; the target log table is used to record operations performed on a table in the first database within a target historical time period; the target log table includes a plurality of records, each record corresponds to an operation, and each record includes a corresponding operation sequence number, a corresponding table name, a corresponding operation name, a value of a primary unique key of the corresponding table, values ​​of other unique keys of the corresponding table, and a sequence number of a corresponding dependent operation; S200, traverse each record of the target log table in descending order of the corresponding operation sequence number, and obtain an initial directed acyclic graph corresponding to the target log table; the initial directed acyclic graph includes several layers of nodes, each layer of nodes includes several nodes, each node represents an operation sequence number, and different nodes represent different operation sequence numbers; the operation corresponding to any child node in the initial directed acyclic graph depends on the operation corresponding to the corresponding parent node, and the operation corresponding to any first-layer node in the initial directed acyclic graph does not depend on the operation corresponding to any other node in the initial directed acyclic graph; S300, if the number of first-layer nodes in the initial directed acyclic graph is greater than a first preset number, select the first preset number of first-layer nodes from the initial directed acyclic graph to add to the ready queue, so as to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed; S400, if the number of nodes included in the initial directed acyclic graph is greater than the number of first-layer nodes selected to be added to the ready queue in the initial directed acyclic graph, a first updated directed acyclic graph is obtained, and S500 is entered; the first updated directed acyclic graph is a directed acyclic graph obtained after deleting the first-layer nodes selected to be added to the ready queue from the initial directed acyclic graph; the operation corresponding to any first-layer node in the first updated directed acyclic graph does not depend on the operation corresponding to any other node in the first updated directed acyclic graph; S500, if the number of first-layer nodes in the first updated directed acyclic graph is greater than the first preset number, a first preset number of first-layer nodes are selected from the first updated directed acyclic graph to add to the ready queue, so as to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed.

2. The database synchronization method based on directed acyclic graph according to claim 1, characterized in that: After S500, the method further includes the following steps: S600, if the number of nodes included in the first updated directed acyclic graph is equal to the number of first-layer nodes selected to be added to the ready queue in the first updated directed acyclic graph, it is determined that the synchronization process for the second database is terminated; if the number of nodes included in the first updated directed acyclic graph is greater than the number of first-layer nodes selected to be added to the ready queue in the first updated directed acyclic graph, the second updated directed acyclic graph is obtained and enters S700; the second updated directed acyclic graph is a directed acyclic graph obtained after deleting the first-layer nodes selected to be added to the ready queue from the first updated directed acyclic graph; the operation corresponding to any first-layer node in the second updated directed acyclic graph does not depend on the operation corresponding to any other node in the second updated directed acyclic graph; S700, if the number of first-layer nodes in the second updated directed acyclic graph is greater than the first preset number, select the first preset number of first-layer nodes from the second updated directed acyclic graph to add to the ready queue, so as to concurrently execute the operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed.

3. The database synchronization method based on directed acyclic graph according to claim 1, characterized in that: Any first-layer node in the first updated directed acyclic graph satisfies the following conditions: it is a first-layer node in the initial directed acyclic graph, or it is not a first-layer node in the initial directed acyclic graph and all corresponding parent nodes in the initial directed acyclic graph have been selected to be added to the ready queue.

4. The database synchronization method based on directed acyclic graph according to claim 1, characterized in that: S300 also includes: if the number of first-layer nodes in the initial directed acyclic graph is less than or equal to a first preset number, all first-layer nodes in the initial directed acyclic graph are added to a ready queue to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and the ready queue is cleared after the execution is completed.

5. The database synchronization method based on directed acyclic graph according to claim 1, characterized in that: The operation corresponding to each record is named insert new data or delete old data or delete old data when updating or insert new data when updating.

6. The database synchronization method based on directed acyclic graph according to claim 1, characterized in that: S400 further includes: if the number of nodes included in the initial directed acyclic graph is equal to the number of first-layer nodes in the initial directed acyclic graph selected to be added to the ready queue, determining that the synchronization process for the second database is finished.

7. The database synchronization method based on directed acyclic graph according to claim 2, characterized in that: After S700, the method further includes the following steps: S800: If the number of nodes included in the second updated directed acyclic graph is equal to the number of first-layer nodes selected to be added to the ready queue in the second updated directed acyclic graph, it is determined that the synchronization process for the second database is terminated.

8. The database synchronization method based on directed acyclic graph according to claim 1, characterized in that: If the operation corresponding to a record in the target log table does not depend on any other operation, the sequence numbers of the dependent operations corresponding to the record are all composed of 0s, and the number of 0s is the number of unique keys of the corresponding table.

9. The database synchronization method based on directed acyclic graph according to claim 1, characterized in that: Any i+1th layer node in the initial directed acyclic graph is a child node of several ith layer nodes, the value of i ranges from 1 to n-1, and n is the number of layers included in the initial directed acyclic graph.

Images