A database synchronization method based on directed acyclic graph

Abstract

The present application relates to the field of electronic digital data processing technology, and in particular to a database synchronization method based on a directed acyclic graph. The method comprises: obtaining a target log table; traversing each record of the target log table in descending order according to the corresponding operation sequence number to obtain an initial directed acyclic graph corresponding to the target log table; if the number of first-layer nodes in the initial directed acyclic graph is greater than a first preset number, selecting the first preset number of first-layer nodes to join the ready queue; if the number of nodes included in the initial directed acyclic graph is greater than the number of first-layer nodes selected to join the ready queue in the initial directed acyclic graph, obtaining a first updated directed acyclic graph; if the number of first-layer nodes in the first updated directed acyclic graph is greater than the first preset number, selecting the first preset number of first-layer nodes from the first updated directed acyclic graph to join the ready queue. The present invention can improve the synchronization efficiency of the database and reduce synchronization delay.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, in particular to a database synchronization method based on a directed acyclic graph. Background Art

[0002] Database synchronization involves replicating changes to the primary database in the standby database to ensure that the data in the standby is fully consistent with the primary. To ensure that changes to the primary database are promptly replicated in the standby, for tables with only one unique constraint field, each operation is mapped to a different bucket using the value of the unique constraint field as the key. This allows for β operations to be executed in parallel, where β is the number of buckets.

[0003] The above method of mapping to different buckets is more suitable for tables with only one unique constraint field. However, for tables with multiple unique constraint fields, this method is not applicable due to the interdependencies between the corresponding operations. Existing techniques typically use a serial synchronization method, which executes only one operation at a time. This serial synchronization method results in low database synchronization efficiency and high synchronization latency. Summary of the Invention

[0004] The present invention aims to provide a database synchronization method based on a directed acyclic graph to improve the synchronization efficiency of the database and reduce the synchronization delay.

[0005] According to the present invention, a database synchronization method based on a directed acyclic graph is provided, the method comprising the following steps:

[0006] S100, obtain a target log table; the target log table is used to record operations performed on a table in the first database within a target historical time period; the target log table includes a plurality of records, each record corresponds to an operation, and each record includes a corresponding operation serial number, a corresponding table name, a corresponding operation name, a value of a primary unique key of the corresponding table, values ​​of other unique keys of the corresponding table, and a serial number of a corresponding dependent operation.

[0007] S200, traverse each record of the target log table in descending order of the corresponding operation sequence number, and obtain the initial directed acyclic graph corresponding to the target log table; the initial directed acyclic graph includes several layers of nodes, each layer of nodes includes several nodes, each node represents an operation sequence number, and different nodes represent different operation sequence numbers; the operation corresponding to any child node in the initial directed acyclic graph depends on the operation corresponding to the corresponding parent node, and the operation corresponding to any first-layer node in the initial directed acyclic graph does not depend on the operation corresponding to any other node in the initial directed acyclic graph.

[0008] S300, if the number of first-layer nodes in the initial directed acyclic graph is greater than the first preset number, a first preset number of first-layer nodes are selected from the initial directed acyclic graph to add to the ready queue, so as to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed.

[0009] S400, if the number of nodes included in the initial directed acyclic graph is greater than the number of first-layer nodes selected to be added to the ready queue in the initial directed acyclic graph, obtain the first updated directed acyclic graph and enter S500; the first updated directed acyclic graph is a directed acyclic graph obtained after deleting the first-layer nodes selected to be added to the ready queue from the initial directed acyclic graph; the operation corresponding to any first-layer node in the first updated directed acyclic graph does not depend on the operation corresponding to any other node in the first updated directed acyclic graph.

[0010] S500, if the number of first-layer nodes in the first updated directed acyclic graph is greater than the first preset number, a first preset number of first-layer nodes are selected from the first updated directed acyclic graph to be added to the ready queue, so as to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed.

[0011] Compared with the prior art, the present invention has at least the following beneficial effects:

[0012] The present invention obtains the corresponding initial directed acyclic graph according to the target log table. The initial directed acyclic graph is obtained according to the target log table, wherein each node included in the initial directed acyclic graph represents the operation sequence number corresponding to each record in the target log table, the first-layer nodes in the initial directed acyclic graph represent the operation sequence number corresponding to the operation that does not depend on any operation in the target log table, and the operation corresponding to the child node in the initial directed acyclic graph depends on the operation corresponding to the corresponding parent node; based on the acquisition of the initial directed acyclic graph, the present invention adds multiple first-layer nodes in the initial directed acyclic graph to the ready queue at one time to The two databases concurrently execute operations corresponding to all first-layer nodes in the ready queue, thereby improving the efficiency of synchronization with the second database and reducing synchronization delays. If not all nodes in the initial directed acyclic graph are added to the ready queue, a directed acyclic graph consisting of the remaining nodes is further obtained, i.e., a first updated directed acyclic graph. For the first updated directed acyclic graph, multiple first-layer nodes therein are also added to the ready queue at one time, so as to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database. Thus, the present invention further improves the efficiency of synchronization with the second database and reduces synchronization delays. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0014] Figure 1 A flowchart of a database synchronization method based on a directed acyclic graph provided in Example 1 of the present invention;

[0015] Figure 2 A schematic diagram of an initial directed acyclic graph provided in Example 1 of the present invention;

[0016] Figure 3 A schematic diagram of a first updated directed acyclic graph provided in Example 1 of the present invention;

[0017] Figure 4 A schematic diagram of a second updated directed acyclic graph provided in Example 1 of the present invention;

[0018] Figure 5 A schematic diagram of a third updated directed acyclic graph provided in the first embodiment of the present invention;

[0019] Figure 6 A schematic diagram of a fourth updated directed acyclic graph provided in the first embodiment of the present invention;

[0020] Figure 7A schematic diagram of a fifth updated directed acyclic graph provided in the first embodiment of the present invention;

[0021] Figure 8 A schematic diagram of a first updated directed acyclic graph provided in the second embodiment of the present invention;

[0022] Figure 9 A schematic diagram of a second updated directed acyclic graph provided in the second embodiment of the present invention;

[0023] Figure 10 This is a schematic diagram of the third updated directed acyclic graph provided in the second embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0025] Example 1:

[0026] According to this embodiment, Figure 1 As shown, a database synchronization method based on a directed acyclic graph is provided, and the method includes the following steps:

[0027] S100, obtain a target log table; the target log table is used to record operations performed on a table in the first database within a target historical time period; the target log table includes a plurality of records, each record corresponds to an operation, and each record includes a corresponding operation serial number, a corresponding table name, a corresponding operation name, a value of a primary unique key of the corresponding table, values ​​of other unique keys of the corresponding table, and a serial number of a corresponding dependent operation.

[0028] In this embodiment, operations performed on the table in the first database during the target historical time period have not yet been synchronized in the second database.

[0029] As a specific implementation, the operation corresponding to each record is named insert new data (insert new), delete old data (delete old), update delete old data (update delete old), or update insert new data (update insert new). The update delete old data operation and the update insert new data operation together constitute an update operation. The update delete old data operation and the update insert new data operation included in the same update operation share the same operation sequence number.

[0030] As a specific implementation, the first record in the target log table includes an operation with a sequence number of 1. If the operation corresponding to a record in the target log table does not depend on any other operations, the sequence numbers of the dependent operations corresponding to the record are all 0, and the number of 0s is equal to the number of unique keys in the corresponding table. Those skilled in the art will appreciate that duplicate values ​​are not allowed in any column corresponding to a unique key in a table.

[0031] As a specific implementation, the target log table is shown in Table 1, where the id field corresponds to the operation serial number, the table field corresponds to the table name, the op field corresponds to the operation name, the PK field corresponds to the value of the primary unique key of the table, the UK field corresponds to the value of other unique keys of the table, and the dependent field corresponds to the serial number of the dependent operation.

[0032] Table 1

[0033] id table op PK UK rely 1 A Insert_new 1 1,1 0,0,0 2 A Insert_new 2 2,2 0,0,0 3 A Insert_new 3 3,3 0,0,0 4 A Update_Delete_old 1 1,1 1,1,1 4 A UPDate_Insert_new 1 4,4 4,0,0 5 A Update_Delete_old 2 2,2 2,2,2 5 A UPDate_Insert_new 2 5,5 5,0,0 6 A Delete_old 3 3,3 3,3,3 7 A Insert_new 3 2,1 6,5,4 8 B Insert_new 11 John 0,0 9 B Insert_new 12 james 0,0 10 B Insert_new 13 david 0,0 11 B Update_Delete_old 11 John 8,8 11 B UPDate_Insert_new 21 John 0,11 12 B Insert_new 11 sophia 11,0 13 B Update_Delete_old 21 John 0,11 13 B UPDate_Insert_new 21 ava 13,0 14 B Insert_new 14 John 0,13 15 B Delete_old 12 james 9,9 16 B Insert_new 15 michael 0,0

[0034] In Table 1, the value of the id field of the first record in the target log table is 1, indicating that the operation sequence number corresponding to the record is 1, that is, the operation corresponding to the record is the first operation performed on the first database during the target historical time period; the value of the table field of the first record in the target log table is A, indicating that the table name corresponding to the record is A, that is, the operation corresponding to the record is an operation in table A; the value of the op field of the first record in the target log table is Insert_new, indicating that the operation name corresponding to the record is Insert_new, that is, the operation corresponding to the record is an insert operation; the first record in the target log table The value of the PK field of the record is 1 and the value of the UK field of the first record in the target log table is 1,1, indicating that the value of the primary unique key of the table corresponding to the record is 1 and the values ​​of the other unique keys are 1,1. That is, the operation corresponding to the record is to insert a record into table A (table A includes two other unique keys in addition to the primary unique key), and the value of the primary unique key of the inserted record is 1 and the values ​​of the other two unique keys are 1 respectively; the value of the dependent field of the first record in the target log table is 0,0,0, indicating that the sequence number of the dependent operation corresponding to the record is 0,0,0, that is, the operation corresponding to the record does not depend on any other operation.

[0035] In Table 1, the value of the id field of the 15th record in the target log table is 15, indicating that the operation sequence number corresponding to the record is 15, that is, the operation corresponding to the record is the 15th operation performed on the first database during the target historical time period; the value of the table field of the 15th record in the target log table is B, indicating that the table name corresponding to the record is B, that is, the operation corresponding to the record is an operation in table B; the value of the op field of the 15th record in the target log table is Delete_old, indicating that the operation name corresponding to the record is Delete_old, that is, the operation corresponding to the record is a delete operation; the 15th record in the target log table The value of the PK field of the record is 12 and the value of the UK field of the 15th record in the target log table is james, indicating that the primary unique key value of the table corresponding to the record is 12 and the value of the other unique key is james, that is, the operation corresponding to the record is to delete a record in table B (table B includes 1 other unique key in addition to the primary unique key), and the primary unique key value of the deleted record is 12 and the value of the other unique key is james; the value of the dependent field of the 15th record in the target log table is 9,9, indicating that the serial number of the dependent operation corresponding to the record is 9,9, that is, the operation corresponding to the record depends on the operation with operation serial number 9.

[0036] S200, traverse each record of the target log table in descending order of the corresponding operation sequence number, and obtain the initial directed acyclic graph corresponding to the target log table; the initial directed acyclic graph includes several layers of nodes, each layer of nodes includes several nodes, each node represents an operation sequence number, and different nodes represent different operation sequence numbers; the operation corresponding to any child node in the initial directed acyclic graph depends on the operation corresponding to the corresponding parent node, and the operation corresponding to any first-layer node in the initial directed acyclic graph does not depend on the operation corresponding to any other node in the initial directed acyclic graph.

[0037] In this embodiment, any i+1th layer node in the initial directed acyclic graph is a child node of several i-th layer nodes, the value of i ranges from 1 to n-1, and n is the number of layers included in the initial directed acyclic graph.

[0038] In this example, if the dependent operation number corresponding to the record corresponding to the first operation number includes the second operation number, then the node representing the second operation number is determined to be the parent node of the node representing the first operation number, that is, the node representing the first operation number is determined to be the child node of the node representing the second operation number. As a specific implementation method, the initial directed acyclic graph corresponding to Table 1 is as follows: Figure 2As shown, the nodes corresponding to 1, 2, 3, 8, 9, 10 and 16 are first-layer nodes, the nodes corresponding to 4, 5, 6, 11 and 15 are second-layer nodes, the nodes corresponding to 7, 12 and 13 are third-layer nodes, and the node corresponding to 14 is a fourth-layer node. The node corresponding to 1 is the parent node of the node corresponding to 4 (that is, the node corresponding to 4 is the child node of the node corresponding to 1), the node corresponding to 2 is the parent node of the node corresponding to 5, the node corresponding to 3 is the parent node of the node corresponding to 6, the node corresponding to 4, the node corresponding to 5 and the node corresponding to 6 are all parent nodes of the node corresponding to 7, the node corresponding to 8 is the parent node of the node corresponding to 11, the node corresponding to 11 is the parent node of the node corresponding to 12 and the node corresponding to 13, the node corresponding to 13 is the parent node of the node corresponding to 14, and the node corresponding to 9 is the parent node of the node corresponding to 15.

[0039] S300, if the number of first-layer nodes in the initial directed acyclic graph is greater than the first preset number, a first preset number of first-layer nodes are selected from the initial directed acyclic graph to add to the ready queue, so as to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed.

[0040] In this embodiment, if the number of first-layer nodes in the initial directed acyclic graph is less than or equal to a first preset number, all first-layer nodes in the initial directed acyclic graph are added to the ready queue to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and the ready queue is cleared after the execution is completed.

[0041] In this embodiment, the first preset number is a preset number of concurrent processing. Optionally, a first preset number of first-layer nodes are randomly selected from the initial directed acyclic graph and added to the ready queue.

[0042] S400, if the number of nodes included in the initial directed acyclic graph is greater than the number of first-layer nodes selected to be added to the ready queue in the initial directed acyclic graph, obtain the first updated directed acyclic graph and enter S500; the first updated directed acyclic graph is a directed acyclic graph obtained after deleting the first-layer nodes selected to be added to the ready queue from the initial directed acyclic graph; the operation corresponding to any first-layer node in the first updated directed acyclic graph does not depend on the operation corresponding to any other node in the first updated directed acyclic graph.

[0043] In this embodiment, any first-layer node in the first updated directed acyclic graph meets the following conditions: it is a first-layer node in the initial directed acyclic graph, or it is not a first-layer node in the initial directed acyclic graph and all corresponding parent nodes in the initial directed acyclic graph have been selected to be added to the ready queue.

[0044] In this embodiment, if the number of nodes included in the initial directed acyclic graph is equal to the number of first-layer nodes selected to be added to the ready queue in the initial directed acyclic graph, it is determined that the synchronization process for the second database is completed and subsequent steps are no longer executed.

[0045] In this embodiment, the first layer nodes selected to be added to the ready queue in the initial directed acyclic graph are different, and the first updated directed acyclic graph obtained is different; as a specific implementation method, the first preset number is 4, Figure 2 The nodes corresponding to 1, 9, 10 and 16 in the table are added to the ready queue, and the first updated directed acyclic graph is as follows: Figure 3 shown.

[0046] S500, if the number of first-layer nodes in the first updated directed acyclic graph is greater than the first preset number, a first preset number of first-layer nodes are selected from the first updated directed acyclic graph to be added to the ready queue, so as to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed.

[0047] Optionally, a first preset number of first-layer nodes are randomly selected from the first updated directed acyclic graph and added to the ready queue.

[0048] In this embodiment, if the number of first-layer nodes in the first updated directed acyclic graph is less than or equal to a first preset number, all first-layer nodes in the first updated directed acyclic graph are added to the ready queue to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and the ready queue is cleared after the execution is completed.

[0049] This embodiment obtains the corresponding initial directed acyclic graph according to the target log table. The initial directed acyclic graph is obtained according to the target log table, wherein each node included in the initial directed acyclic graph represents the operation sequence number corresponding to each record in the target log table, the first-layer nodes in the initial directed acyclic graph represent the operation sequence number corresponding to the operation that does not depend on any operation in the target log table, and the operation corresponding to the child node in the initial directed acyclic graph depends on the operation corresponding to the corresponding parent node; based on the acquisition of the initial directed acyclic graph, this embodiment adds multiple first-layer nodes in the initial directed acyclic graph to the ready queue at one time to process the first-layer nodes. The two databases concurrently execute operations corresponding to all first-layer nodes in the ready queue. Thus, this embodiment improves the efficiency of synchronization with the second database and reduces synchronization delays. If not all nodes in the initial directed acyclic graph are added to the ready queue, a directed acyclic graph consisting of the remaining nodes is further obtained, i.e., a first updated directed acyclic graph. For the first updated directed acyclic graph, multiple first-layer nodes therein are also added to the ready queue at one time to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database. Thus, this embodiment further improves the efficiency of synchronization with the second database and reduces synchronization delays.

[0050] In this embodiment, after S500, the method further includes the following steps:

[0051] S600, if the number of nodes included in the first updated directed acyclic graph is equal to the number of first-layer nodes selected to be added to the ready queue in the first updated directed acyclic graph, it is determined that the synchronization process of the second database is ended; if the number of nodes included in the first updated directed acyclic graph is greater than the number of first-layer nodes selected to be added to the ready queue in the first updated directed acyclic graph, the second updated directed acyclic graph is obtained and enters S700; the second updated directed acyclic graph is a directed acyclic graph obtained after deleting the first-layer nodes selected to be added to the ready queue from the first updated directed acyclic graph; the operation corresponding to any first-layer node in the second updated directed acyclic graph does not depend on the operation corresponding to any other node in the second updated directed acyclic graph.

[0052] In this embodiment, if the first preset number of first-layer nodes selected from the first updated directed acyclic graph is different, the obtained second updated directed acyclic graph will be different; as a specific implementation method, the first updated directed acyclic graph is as follows: Figure 3 As shown, the first preset number is 4. Figure 3 The nodes corresponding to 2, 3, 4 and 15 in the table are added to the ready queue, and the second updated directed acyclic graph is as follows: Figure 4 shown.

[0053] S700, if the number of first-layer nodes in the second updated directed acyclic graph is greater than the first preset number, select the first preset number of first-layer nodes from the second updated directed acyclic graph to add to the ready queue, so as to concurrently execute the operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed.

[0054] In this embodiment, if the number of first-layer nodes in the second updated directed acyclic graph is less than or equal to the first preset number, all first-layer nodes in the second updated directed acyclic graph are added to the ready queue to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and the ready queue is cleared after the execution is completed.

[0055] In this embodiment, after S700, the method further includes the following steps:

[0056] S800: If the number of nodes included in the second updated directed acyclic graph is equal to the number of first-layer nodes selected to be added to the ready queue in the second updated directed acyclic graph, it is determined that the synchronization process for the second database is completed.

[0057] In this embodiment, if the number of nodes included in the second updated directed acyclic graph is greater than the number of first-layer nodes selected to be added to the ready queue in the second updated directed acyclic graph, a third updated directed acyclic graph is obtained, and it is determined whether the number of first-layer nodes in the third updated directed acyclic graph is greater than a first preset number. If so, the first preset number of first-layer nodes are selected from the third updated directed acyclic graph and added to the ready queue to concurrently execute the operations corresponding to all first-layer nodes in the ready queue on the second database, and the ready queue is cleared after the execution is completed; otherwise, all first-layer nodes in the third updated directed acyclic graph are added to the ready queue to concurrently execute the operations corresponding to all first-layer nodes in the ready queue on the second database, and the ready queue is cleared after the execution is completed. If the number of nodes included in the third updated directed acyclic graph is greater than the number of first-layer nodes selected to be added to the ready queue in the third updated directed acyclic graph, a fourth updated directed acyclic graph is obtained. This process is repeated in this way until the operations corresponding to all records included in the target log table are completed in the second database, and the synchronization process ends.

[0058] As a specific implementation, the second updated directed acyclic graph is as follows Figure 4 As shown, the first preset number is 4. Figure 4 The nodes corresponding to 5, 6 and 8 in are added to the ready queue, and the third updated directed acyclic graph is as follows Figure 5 As shown; Figure 5 The nodes corresponding to 7 and 11 in the list are added to the ready queue, and the fourth updated directed acyclic graph is as follows: Figure 6 As shown; Figure 6The nodes corresponding to 12 and 13 in the list are added to the ready queue, and the fifth updated directed acyclic graph is as follows: Figure 7 As shown; Figure 7 The node corresponding to 14 in the target log table is added to the ready queue, then the operations corresponding to all records included in the target log table are executed in the second database, and the synchronization process is completed; for the target log table shown in Table 1, the synchronization process of this specific implementation method includes 5 waiting times for other operations to be completed. Compared with the synchronization process using the serial synchronization method in the prior art, which requires 15 waiting times for other operations to be completed, this specific implementation method improves the synchronization efficiency of the second database and reduces the synchronization delay.

[0059] Example 2:

[0060] In the above-mentioned first embodiment, the selection of a first preset number of first-layer nodes from the initial directed acyclic graph to be added to the ready queue is random. Although this method can improve the synchronization efficiency of the second database compared to the serial synchronization method in the prior art, it still cannot maximize the synchronization efficiency of the second database. In order to further improve the synchronization efficiency of the second database, this embodiment, based on the first embodiment, further optimizes the process of selecting a first preset number of first-layer nodes from the initial directed acyclic graph to be added to the ready queue. Specifically, the selection of a first preset number of first-layer nodes from the initial directed acyclic graph to be added to the ready queue includes:

[0061] S310, obtaining the descendant node complexity f of the j-th first-layer node in the initial directed acyclic graph j , f j =∑ T(j) t＝1 (∏ Q(t) q＝1 num t,q ), num t,q is the number of parent nodes of the qth node of the tth branch connected to the jth first-layer node in the initial directed acyclic graph, the value range of q is 1 to Q(t), Q(t) is the number of nodes included in the tth branch connected to the jth first-layer node in the initial directed acyclic graph; the value range of t is 1 to T(j), T(j) is the number of branches connected to the jth first-layer node in the initial directed acyclic graph; the value range of j is 1 to m, m is the number of first-layer nodes in the initial directed acyclic graph.

[0062] like Figure 2As shown, the number of branches connected by the nodes corresponding to 10 and 16 is 0, the number of branches connected by the nodes corresponding to 1, 2, 3 and 9 is all 1, and the number of branches connected by 8 is 2; among them, the first node included in the branch connected by the node corresponding to 1 is the node corresponding to 4, and the second node included in the branch connected by the node corresponding to 1 is the node corresponding to 7; the first node included in the branch connected by the node corresponding to 2 is the node corresponding to 5, and the second node included in the branch connected by the node corresponding to 2 is the node corresponding to 7; the first node included in the branch connected by the node corresponding to 3 is the node corresponding to 6, and the second node included in the branch connected by the node corresponding to 3 is the node corresponding to 7; the node corresponding to 9 The first node included in the connected branch is the node corresponding to 15; the first node included in the first branch connected to the node corresponding to 8 is the node corresponding to 11, and the second node included in the first branch connected to the node corresponding to 8 is the node corresponding to 12; the first node included in the second branch connected to the node corresponding to 8 is the node corresponding to 11; the second node included in the second branch connected to the node corresponding to 8 is the node corresponding to 13; the third node included in the second branch connected to the node corresponding to 8 is the node corresponding to 14; the number of parent nodes of the nodes corresponding to 4, 5, 6, 11, 12, 13, 14 and 15 is all 1, and the number of parent nodes of the node corresponding to 7 is 3.

[0063] S320, obtain the number s of unbound nodes of the jth first-layer node in the initial directed acyclic graph j ; If the j-th first-layer node in the initial directed acyclic graph does not have a child node, then s j = 0; if the j-th first-layer node in the initial directed acyclic graph has a child node, and there are z children in the j-th first-layer node in the initial directed acyclic graph j There is only one parent node for each child node, then s j =z j .

[0064] In this embodiment, the number of unbound nodes of any first-layer node is the number of newly added first-layer nodes after the corresponding first-layer node is deleted. Figure 2 As shown, the number of unbound nodes for nodes 10 and 16 is 0, and the number of unbound nodes for nodes 1, 2, 3, 8, and 9 is 1.

[0065] S330, obtaining the first priority value y of the j-th first-layer node in the initial directed acyclic graph j ;y j With f j Positive correlation, y j With s j Positive correlation.

[0066] As a preferred embodiment, j=w1×f' j +w2×s' j , w1 and w2 are the weights corresponding to the complexity of descendant nodes and the number of unbound nodes, f' j and s' j are the normalized f j and the normalized s j , w1 and w2 are both greater than 0 and less than 1, w1+w2=1. The process of obtaining w2 includes:

[0067] S331, obtaining the number g of first-layer nodes in the initial directed acyclic graph whose number of unbound nodes is 0.

[0068] S332, if m-min(h,g)≥4, the first preset weight value is determined to be w2; otherwise, the second preset weight value is determined to be w2; the first preset weight and the second preset weight are both greater than 0 and less than 1, and the second preset weight is greater than the first preset weight; min() is the minimum value, and h is the first preset number.

[0069] Optionally, the first preset weight and the second preset weight are both empirical values; for example, the first preset weight is 0.5, and the second preset weight is 0.7.

[0070] Based on the above preferred specific implementation method, the value of w2 is related to the number of first-layer nodes in the initial directed acyclic graph whose number of unbound nodes is 0. If the first-layer nodes in the initial directed acyclic graph whose number of unbound nodes is 0 are preferentially added to the ready queue at one time, it will cause the number of nodes in the remaining nodes of the initial directed acyclic graph that can be added to the ready queue at one time next time to be less than the first preset value. In this case, a larger value is assigned to w2 so that the first-layer nodes in the initial directed acyclic graph whose number of unbound nodes is not 0 are preferentially added to the ready queue, thereby achieving the purpose of increasing the number of nodes in the remaining nodes of the initial directed acyclic graph that can be added to the ready queue at one time next time. Since the number of operations that can be executed concurrently next time is increased, this preferred specific implementation method can further speed up the processing speed of operations corresponding to nodes in the initial directed acyclic graph.

[0071] S340: Add a first preset number of first-layer nodes with the largest first priority values ​​in the initial directed acyclic graph to a ready queue.

[0072] As a preferred specific implementation, S340 includes:

[0073] S341, sorting the first layer nodes in the initial directed acyclic graph in descending order according to the corresponding first priority values ​​to obtain a first node sequence D; D = [d1, d2, ..., d r ,…,d m ], d ris the rth first-layer node in D, and the value of r ranges from 1 to m.

[0074] S342, if d h The first priority value is equal to d h+1 The first priority value is entered into S343; h is the first preset number.

[0075] S343, if d h If the first priority value of is 0, the first h first-layer nodes in D are determined as the first preset number of first-layer nodes with the largest first priority value in the initial directed acyclic graph; otherwise, go to S344.

[0076] S344, obtain the first priority value in D and d h The set E of first-layer nodes with the same first priority value, E={e1,e2,…,e x ,…,e R}, e x is the xth first-level node in E, where x ranges from 1 to R, and R is the first priority value in D and d h The first priority value is equal to the number of first layer nodes.

[0077] S345, traverse E, get e x The second priority value of e x The second priority value is e x The maximum value of the first priority value of all child nodes.

[0078] In this embodiment, if e x There is only one child node, then e x The second priority value is e x The first priority value of the child node; if e x There are more than 2 child nodes, then e x The second priority value is e x The maximum value of the first priority value of all child nodes. x The first priority value of any child node is also based on e x The complexity of the descendant nodes and the number of unbound nodes of the corresponding child node are obtained, which will not be repeated here.

[0079] S346, sort the first-layer nodes in E in descending order according to the corresponding second priority values ​​to obtain a second node sequence C; C = [c1, c2, ..., c v ,…,c R ], c v is the vth first-layer node in C, and the value of v ranges from 1 to R.

[0080] S347, if c h-lThe second priority value is not equal to c h-l+1 The first hl first-layer nodes in C and the first l first-layer nodes in D are determined as the first first preset number of first-layer nodes with the largest first priority value in the initial directed acyclic graph; l is the first priority value in D greater than d h The number of first-layer nodes.

[0081] In this embodiment, if c h-l The second priority value is equal to c h-l+1 The second priority value of is processed in a similar way to S343-S347, that is, first determine c h-l Is the second priority value of equal to 0? If so, the first hl first-layer nodes in C and the first l first-layer nodes in D are determined as the first preset number of first-layer nodes with the largest first priority value in the initial directed acyclic graph; if not, get the second priority value in C and the first priority value in D. h-l The first-layer nodes F with the same second priority value as , and then sort the first-layer nodes in F according to the third priority value of each first-layer node in F to obtain a third node sequence G; if the third priority value of the hl-nth first-layer node in G is not equal to the third priority value of the hl-n+1th first-layer node, then the first hl-nth first-layer nodes in G, the first nth first-layer nodes in C and the first lth first-layer nodes in D are determined as the first first preset number of first-layer nodes with the largest first priority value in the initial directed acyclic graph; n is the number of nodes in C with the second priority value greater than c h-l If the third priority value of the hl-nth first-layer node in G is equal to the third priority value of the hl-n+1th first-layer node, then the process is similar to S343-S347 and will not be repeated here. The third priority value of any first-layer node is the maximum of the first priority values ​​of all grandchild nodes of the corresponding first-layer node.

[0082] In this embodiment, the process of selecting a first preset number of first-layer nodes from any updated directed acyclic graph to join the ready queue is similar to the above-mentioned process of selecting a first preset number of first-layer nodes from the initial directed acyclic graph to join the ready queue, and will not be repeated here.

[0083] As a specific implementation, the initial directed acyclic graph is as follows Figure 2 As shown, the first preset number is 4. According to the method of this embodiment, the first layer nodes added to the ready queue in the initial directed acyclic graph determined by the method of this embodiment are the nodes corresponding to 1, 2, 3 and 8. Then the first updated directed acyclic graph is as follows: Figure 8As shown; According to the method of this embodiment, the first layer nodes added to the ready queue in the first updated directed acyclic graph are nodes corresponding to 4, 5, 9 and 11, then the second updated directed acyclic graph is as shown Figure 9 As shown; According to the method of this embodiment, the first layer nodes added to the ready queue in the second updated directed acyclic graph are nodes corresponding to 6, 10, 12 and 13, then the third updated directed acyclic graph is obtained as shown Figure 10 As shown; Figure 10 The nodes corresponding to 7, 14, 15, and 16 in the target log table are added to the ready queue, and the operations corresponding to all records included in the target log table are completed in the second database, and the synchronization process ends. For the target log table shown in Table 1, the synchronization process of this specific implementation includes three waiting times for other operations to be completed. Compared with the situation in Example 1 where five waiting times are required for other operations to be completed, this specific implementation further improves the synchronization efficiency of the second database and reduces the synchronization delay.

[0084] In addition to the advantages of embodiment one, this embodiment also obtains the complexity of descendant nodes and the number of unbound nodes of each first-layer node in the initial directed acyclic graph, and determines the first priority value of the corresponding first-layer node based on the complexity of the descendant nodes and the number of unbound nodes. The greater the complexity of the descendant nodes, the greater the corresponding first priority value, and the greater the number of unbound nodes, the greater the corresponding first priority value. Therefore, this embodiment adds the first preset number of first-layer nodes with the largest first priority values ​​in the initial directed acyclic graph to the ready queue, so that the operations corresponding to the first-layer nodes with large descendant node complexity and a large number of unbound nodes are executed first, avoiding serious node blocking caused by the fact that the operations corresponding to the first-layer nodes with large descendant node complexity and a large number of unbound nodes are not executed, which is beneficial to increasing the number of nodes in the remaining nodes of the initial directed acyclic graph that can be added to the ready queue at one time next time, thereby improving the processing speed of operations corresponding to the nodes in the initial directed acyclic graph.

[0085] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A database synchronization method based on a directed acyclic graph, characterized in that: The method comprises the following steps: S100, obtaining a target log table; the target log table is used to record operations performed on a table in the first database during a target historical time period; the target log table includes a plurality of records, each record corresponding to an operation, and each record includes a corresponding operation sequence number, a corresponding table name, a corresponding operation name, a value of a primary unique key of the corresponding table, values ​​of other unique keys of the corresponding table, and a sequence number of a corresponding dependent operation; S200, traversing each record of the target log table in descending order of the corresponding operation sequence number to obtain an initial directed acyclic graph corresponding to the target log table; the initial directed acyclic graph includes several layers of nodes, each layer of nodes includes several nodes, each node represents an operation sequence number, and different nodes represent different operation sequence numbers; the operation corresponding to any child node in the initial directed acyclic graph depends on the operation corresponding to the corresponding parent node, and the operation corresponding to any first-layer node in the initial directed acyclic graph does not depend on the operation corresponding to any other node in the initial directed acyclic graph; S300, if the number of first-layer nodes in the initial directed acyclic graph is greater than a first preset number, selecting the first preset number of first-layer nodes from the initial directed acyclic graph and adding them to a ready queue, so as to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and clearing the ready queue after the execution is completed; S400, if the number of nodes included in the initial directed acyclic graph is greater than the number of first-layer nodes selected to be added to the ready queue in the initial directed acyclic graph, a first updated directed acyclic graph is obtained, and the process proceeds to S500; the first updated directed acyclic graph is a directed acyclic graph obtained by deleting the first-layer nodes selected to be added to the ready queue from the initial directed acyclic graph; the operation corresponding to any first-layer node in the first updated directed acyclic graph does not depend on the operation corresponding to any other node in the first updated directed acyclic graph; any first-layer node in the first updated directed acyclic graph satisfies the following conditions: it is a first-layer node in the initial directed acyclic graph, or it is not a first-layer node in the initial directed acyclic graph and all corresponding parent nodes in the initial directed acyclic graph have been selected to be added to the ready queue; S500, if the number of first-layer nodes in the first updated directed acyclic graph is greater than the first preset number, a first preset number of first-layer nodes are selected from the first updated directed acyclic graph to be added to the ready queue, so as to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed.

2. The database synchronization method based on directed acyclic graph according to claim 1, characterized in that: After S500, the method further includes the following steps: S600, if the number of nodes included in the first updated directed acyclic graph is equal to the number of first-layer nodes selected to be added to the ready queue in the first updated directed acyclic graph, it is determined that the synchronization process for the second database is terminated; if the number of nodes included in the first updated directed acyclic graph is greater than the number of first-layer nodes selected to be added to the ready queue in the first updated directed acyclic graph, a second updated directed acyclic graph is obtained and the process proceeds to S700; the second updated directed acyclic graph is a directed acyclic graph obtained by deleting the first-layer nodes selected to be added to the ready queue from the first updated directed acyclic graph; the operation corresponding to any first-layer node in the second updated directed acyclic graph does not depend on the operation corresponding to any other node in the second updated directed acyclic graph; S700, if the number of first-layer nodes in the second updated directed acyclic graph is greater than the first preset number, select the first preset number of first-layer nodes from the second updated directed acyclic graph to add to the ready queue, so as to concurrently execute the operations corresponding to all first-layer nodes in the ready queue on the second database, and clear the ready queue after the execution is completed.

3. The database synchronization method based on directed acyclic graph according to claim 1, characterized in that: S300 also includes: if the number of first-layer nodes in the initial directed acyclic graph is less than or equal to a first preset number, all first-layer nodes in the initial directed acyclic graph are added to a ready queue to concurrently execute operations corresponding to all first-layer nodes in the ready queue on the second database, and the ready queue is cleared after the execution is completed.

4. The database synchronization method based on directed acyclic graph according to claim 1, characterized in that: The operation corresponding to each record is named insert new data or delete old data or delete old data when updating or insert new data when updating.

5. The database synchronization method based on directed acyclic graph according to claim 1, characterized in that: S400 further includes: if the number of nodes included in the initial directed acyclic graph is equal to the number of first-layer nodes in the initial directed acyclic graph selected to be added to the ready queue, determining that the synchronization process for the second database is finished.

6. The database synchronization method based on directed acyclic graph according to claim 2, characterized in that: After S700, the method further includes the following steps: S800: If the number of nodes included in the second updated directed acyclic graph is equal to the number of first-layer nodes selected to be added to the ready queue in the second updated directed acyclic graph, it is determined that the synchronization process for the second database is completed.

7. The database synchronization method based on directed acyclic graph according to claim 1, characterized in that: If the operation corresponding to a record in the target log table does not depend on any other operation, the sequence numbers of the dependent operations corresponding to the record are all composed of 0, and the number of 0 is the number of unique keys of the corresponding table.

8. The database synchronization method based on directed acyclic graph according to claim 1, characterized in that: Any i+1th layer node in the initial directed acyclic graph is a child node of several i-th layer nodes, where the value of i ranges from 1 to n-1, and n is the number of layers included in the initial directed acyclic graph.

Images