Database change data analysis method and related product

Abstract

The invention provides a database change data analysis method and related products, and the database change data analysis method comprises the steps: reading part of change data of a source database, and obtaining a plurality of analysis data corresponding to different source tables of the source database; the minimum serial number data in each piece of analysis data is searched for, whether at least one piece of analysis data and the unread change data comprise the same submission transaction or not is judged, and the minimum serial number data is the data with the minimum start log serial number of the submission transaction on the transaction log in the analysis data to which the minimum serial number data belongs; and if not, brushing the minimum serial number data according to the sequence of the starting log serial numbers. Under the condition that the data of the source database is changed, the consistency of the data after the data of the source database is changed and the actually changed data is ensured, and the data consistency of the target database and the source database is ensured.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a database change data parsing method and related products. Background Art

[0002] With the continuous expansion of enterprise information systems and the growing demand for real-time data processing, capturing and parsing database change data has become a crucial foundation for achieving critical business scenarios such as data synchronization, data integration, and real-time analysis. In practice, some database management systems enable the CDC (Change Data Capture) feature, generating a corresponding CT (Change Tracking) table for each database table to record changes to that table. During database synchronization, change data must be read from each CT table sequentially. Once the change data is parsed, it is written to the source database's disk for subsequent data synchronization.

[0003] Currently, in existing technologies, user data is typically stored across multiple tables in a database. When modifying stored data, different operations within the same cross-table transaction may be performed on different tables, generating cross-table transactions. Consequently, during database synchronization, cross-table transactions can cause the parsing of modified data to become out of order, disrupting the atomicity of cross-table transactions and causing inconsistencies between the stored data in the source or target database and the actual modified data. Summary of the Invention

[0004] An object of the present invention is to provide a database change data parsing method and related products that can overcome at least one of the above-mentioned defects in the prior art.

[0005] A further object of the present invention is to ensure consistency between the data after the source database is changed and the actual changed data, and to ensure data consistency between the target database and the source database when the data in the source database is changed.

[0006] In particular, the present invention provides a method for parsing changed data of a database, comprising:

[0007] Read some of the changed data in the source database to obtain multiple parsed data corresponding to different source tables in the source database;

[0008] Find the minimum sequence number data in each parsed data, and determine whether at least one of them includes the same committed transaction as the unread change data, wherein the minimum sequence number data is the data with the smallest start log sequence number of the committed transaction in the transaction log in the corresponding parsed data;

[0009] If not, flush the data with the smallest sequence number to disk in the order of the starting log sequence number.

[0010] Furthermore, the step of searching for the minimum sequence number data in each parsed data and determining whether at least one of the data includes the same committed transaction as the unread change data includes:

[0011] Each parsed data is stored in a minimum heap, where the key value of each node in the minimum heap is the start log sequence number of different committed transactions in the parsed data;

[0012] Determine whether the root nodes of each minimum heap are consistent;

[0013] If not, it is determined that each minimum sequence number data does not include the same committed transaction as the unread change data.

[0014] Furthermore, when the root nodes of the minimum heaps are consistent, the step of searching for the minimum sequence number data in each parsed data and determining whether at least one of the data includes the same committed transaction as the unread change data further includes:

[0015] Determine whether there is a minimum heap with a depth greater than 1;

[0016] If so, also confirm that each minimum sequence number data does not include the same committed transaction as the unread change data;

[0017] If not, it is determined that the minimum sequence number data and the unread change data include the same committed transaction.

[0018] Furthermore, after the step of flushing the minimum sequence number data to disk in the order of the start log sequence number, the change data parsing method further includes:

[0019] Update each minimum heap that has been submitted with the minimum sequence number data;

[0020] Execute the step of determining whether the root nodes of each minimum heap are consistent.

[0021] Furthermore, the structure of the minimum heap includes the table name of the source table, the depth of the minimum heap and multiple nodes. The structure of each node includes a start log sequence number and a data list. The data list is used to store the parsed data corresponding to the start log sequence number.

[0022] Furthermore, in the case where the minimum sequence number data and the unread change data are included in the same committed transaction, the change data parsing method further includes:

[0023] Sleep for a first preset time period, and then perform the step of searching for the minimum sequence number data in each parsed data.

[0024] Furthermore, the steps of reading part of the changed data of the source database include:

[0025] Read some of the changed data from the blocking queue through the source database's parsing thread; and

[0026] Before the step of reading some of the changed data from the blocking queue by the parsing thread of the source database, the changed data parsing method further includes:

[0027] Obtain the breakpoint structure of the reading thread of the source database, where the breakpoint structure includes the start and end log sequence numbers of the change data last read by the reading thread;

[0028] Assign the value of the end log sequence number to the start log sequence number;

[0029] Query the latest log sequence value in the source database. The latest log sequence value is the start log sequence number of the latest committed transaction in the source database on the transaction log.

[0030] Determine whether the starting log sequence number is greater than or equal to the latest log sequence number;

[0031] If not, assign the latest log sequence value to the terminal log sequence number;

[0032] The change data between the starting end log sequence number value and the ending end log sequence number value is put into the blocking queue through the reading thread; and

[0033] If the starting end log sequence number is greater than or equal to the latest log sequence number, return to the step of querying the latest log sequence number in the source database.

[0034] Furthermore, when the reading thread is running for the first time, and before the step of obtaining the breakpoint structure of the reading thread of the source database, the change data parsing method further includes:

[0035] Initialize the breakpoint structure to assign preset values ​​to the start log sequence number and the end log sequence number respectively; and,

[0036] After the step of querying the latest log sequence value in the source database and before the step of determining whether the value of the starting end log sequence number is greater than or equal to the latest log sequence value, the change data parsing method further includes:

[0037] Determine whether the value of the starting end log sequence number is the preset value;

[0038] If so, assign the latest log sequence value to the starting end log sequence number, and then perform the step of determining whether the value of the starting end log sequence number is greater than or equal to the latest log sequence value;

[0039] If not, the step of determining whether the value of the starting end log sequence number is greater than or equal to the latest log sequence number is executed.

[0040] Furthermore, the change data is stored in the change tracking table corresponding to the source table; and

[0041] The steps of putting the changed data between the start log sequence number value and the end log sequence number value into the blocking queue through the reading thread include:

[0042] Read the change data between the start log sequence number value and the end log sequence number value on multiple change tracking tables in parallel through the reading thread;

[0043] The reading thread puts the changed data between the value of the starting log sequence number and the value of the ending log sequence number into the blocking queue.

[0044] In particular, the present invention also provides a computer program product, including a computer program, which implements the above-mentioned database change data parsing method when executed by a processor.

[0045] In particular, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned method for parsing changed data of a database when the computer program is executed by a processor.

[0046] In particular, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, and the processor implements the above-mentioned database change data analysis method when executing the computer program.

[0047] The database change data parsing method of the present invention can first obtain the parsed data corresponding to different source tables in the read change data, then find the minimum sequence number data in the different parsed data, and then determine whether at least one of them includes the same committed transaction as the unread change data. When all the minimum sequence number data do not include the same committed transaction as the unread change data, each minimum sequence number data is flushed to disk according to the starting log sequence number; that is, the minimum sequence number data with the oldest starting log sequence number in the parsed data corresponding to different source tables and which may belong to the same cross-table transaction are grouped together, and it is further determined whether these minimum sequence number data include the same cross-table transaction as the unread change data. When these minimum sequence number data do not include the same cross-table transaction as the unread change data, that is, when the data including the same cross-table transaction are all in these minimum sequence number data, these minimum sequence number data are persisted on the disk of the source database in the order of the starting log sequence number, so as to ensure the atomicity of the cross-table transaction and suppress or even avoid the occurrence of disorder problems caused by cross-table transactions. Therefore, when the data in the source database changes, the database change data parsing method of the present invention can well ensure the consistency of the data after the source database data change and the actual changed data, and ensure the data consistency between the target database and the source database.

[0048] The computer program product of the present invention can implement the above-mentioned method for analyzing database change data. Therefore, the computer program product of the present invention also has the beneficial technical effects that can be achieved by the above-mentioned method for analyzing database change data.

[0049] The computer-readable storage medium of the present invention can implement the above-mentioned method for analyzing database change data. Therefore, the computer-readable storage medium of the present invention also has the beneficial technical effects that can be achieved by the above-mentioned method for analyzing database change data.

[0050] The computer device of the present invention can implement the database change data analysis method of the above embodiment. Therefore, the computer device of the present invention also has the beneficial technical effects that can be achieved by the above database change data analysis method.

[0051] Based on the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will become more aware of the above and other objects, advantages and features of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Hereinafter, some specific embodiments of the present invention will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:

[0053] Figure 1 1 is a flow chart of a method for parsing database change data according to an embodiment of the present invention;

[0054] Figure 2 is a flowchart of a method for parsing database change data according to another embodiment of the present invention;

[0055] Figure 3 1 is a flow chart of placing changed data into a blocking queue in a database changed data parsing method according to another embodiment of the present invention;

[0056] Figure 4 1 is a schematic diagram of a process for reading change data in a database change data parsing method according to another embodiment of the present invention;

[0057] Figure 5 1 is a schematic diagram of a process for parsing changed data in a method for parsing changed data of a database according to another embodiment of the present invention;

[0058] Figure 6 is a schematic structural block diagram of a computer program product according to an embodiment of the present invention;

[0059] Figure 7 is a schematic block diagram of a computer-readable storage medium according to an embodiment of the present invention;

[0060] Figure 8 FIG. 1 is a structural block diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In the description of this embodiment, it should be understood that, in the description of the present invention, "a plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined. When a feature "includes or comprises" one or more of the features it covers, unless otherwise specifically described, this indicates that other features are not excluded and may further include other features.

[0062] Unless otherwise specified or limited, the term "connection" and other terms should be understood broadly. For example, it can mean fixed connection, detachable connection, or integration; it can mean mechanical connection or electrical connection; it can mean direct connection or indirect connection through an intermediate medium; it can mean internal communication between two elements or interaction between two elements, unless otherwise specified. Those skilled in the art should be able to understand the specific meanings of the above terms in the present invention based on the specific circumstances.

[0063] Unless otherwise defined, all terms (including technical terms and scientific terms) used in the description of this embodiment have the same meaning as commonly understood by ordinary technicians in the technical field to which this application belongs.

[0064] In the description of the present embodiment, reference to the term "embodiment" or the like indicates that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0065] The following is based on Figures 1 to 5 The database change data parsing method of this embodiment is described in detail.

[0066] Figure 1 1 is a flow chart of a method for analyzing database change data according to an embodiment of the present invention. Figure 1 In this embodiment, the database change data parsing method may include:

[0067] Step S101: read part of the changed data in the source database to obtain a plurality of parsed data corresponding to different source tables in the source database. This step may include step S210 of the following embodiment.

[0068] Step S102, search for the minimum sequence number data in each parsed data, and determine whether at least one of them includes the same committed transaction as the unread change data, wherein the minimum sequence number data is the data with the smallest starting log sequence number of the committed transaction on the transaction log in the corresponding parsed data; if not, step S103 can be executed; if so, that is, when it is determined that the minimum sequence number data and the unread change data include the same committed transaction, the step of sleeping for the first preset time or step S218 of the following embodiment can be executed; and this step S102 can include steps S211 to S213, step S216 and step S217 of the following embodiment.

[0069] Step S103: flush the data with the minimum sequence number to disk in the order of the starting log sequence number.

[0070] Since the database change data parsing method of this embodiment can first obtain the parsed data corresponding to different source tables in the read change data, then find the minimum sequence number data in the different parsed data, and then determine whether at least one of them includes the same committed transaction as the unread change data, if all the minimum sequence number data do not include the same committed transaction as the unread change data, each minimum sequence number data is flushed to disk according to the starting log sequence number; that is, the minimum sequence number data with the oldest starting log sequence number in the parsed data corresponding to different source tables and which may belong to the same cross-table transaction are grouped together, and it is further determined whether these minimum sequence number data include the same cross-table transaction as the unread change data. When these minimum sequence number data do not include the same cross-table transaction as the unread change data, that is, when the data including the same cross-table transaction are all in these minimum sequence number data, these minimum sequence number data are persisted on the disk of the source database in the order of the starting log sequence number, so as to ensure the atomicity of the cross-table transaction and suppress or even avoid the occurrence of disorder problems caused by cross-table transactions. Therefore, when the data in the source database changes, the database change data parsing method of this embodiment can well ensure the consistency of the data after the source database data change and the actual changed data, and ensure the data consistency between the target database and the source database.

[0071] In this embodiment, when the minimum sequence number data and the unread change data are included in the same committed transaction, the change data parsing method may further include:

[0072] Sleep for a first preset time period, and then perform the step of searching for the minimum sequence number data in each parsed data.

[0073] It can be understood that, when at least one of the minimum sequence number data includes the same committed transaction as the unread change data, you can wait for the first preset time, at which time new parsed data may be obtained, and then you can return to execute step S102 until the change data that includes the same committed transaction (cross-table transaction) as at least one of the minimum sequence number data is read or the parsed data is obtained. Then, at this time, all the minimum sequence number data include a complete cross-table transaction, and all the minimum sequence number data can be flushed to the disk in the order of the starting log sequence number to ensure the atomicity of the cross-table transaction, suppress or even avoid the occurrence of disorder problems caused by cross-table transactions, and ensure the data consistency of the database.

[0074] In addition, in an alternative embodiment, when the minimum sequence number data and the unread change data include the same committed transaction, the process can directly return to step S101 to similarly ensure the atomicity of cross-table transactions, suppress or even avoid the occurrence of disorder problems caused by cross-table transactions, and ensure the data consistency of the database.

[0075] In an alternative embodiment of step S103, step S103 may be to sort the minimum sequence number data in the order of the start log sequence number, and flush the sorted minimum sequence number data to the disk of the source database.

[0076] Figure 2 FIG. 1 is a flow chart of a method for analyzing database change data according to another embodiment of the present invention. Figure 2 In this embodiment, the database change data parsing method may include:

[0077] Step S201 : Initialize the breakpoint structure to assign preset values ​​to the start log sequence number and the end log sequence number respectively. This step may be performed when the reading thread runs for the first time and before step S202 .

[0078] Step S202: Obtain the breakpoint structure of the source database's read thread; the breakpoint structure includes the starting and ending log sequence numbers of the change data last read by the read thread. This step S202, along with steps S203, S204, S207, 208, and 209, can be performed before step 210, in which the source database's parsing thread reads some change data from the blocking queue.

[0079] Step S203: assign the value of the end log sequence number to the start log sequence number.

[0080] Step S204 , querying the latest log sequence value in the source database, where the latest log sequence value is the value of the start log sequence number of the latest committed transaction in the source database on the transaction log.

[0081] Step S205 determines whether the starting log sequence number is equal to the preset value. If so, step S206 is executed. If not, step S207 is executed to determine whether the starting log sequence number is greater than or equal to the latest log sequence number. Furthermore, steps 205 and 206 can be executed after querying the source database for the latest log sequence number and before determining whether the starting log sequence number is greater than or equal to the latest log sequence number, i.e., between steps S204 and S207.

[0082] Step S206 , assigning the latest log sequence value to the starting end log sequence number; and after this step, executing the step of determining whether the value of the starting end log sequence number is greater than or equal to the latest log sequence value, ie executing step S207 .

[0083] Step S207, determine whether the value of the starting end log sequence number is greater than or equal to the latest log sequence value; if not, execute step S208; if so, that is, when the value of the starting end log sequence number is greater than or equal to the latest log sequence value, return to the step of querying the latest log sequence value in the source database, that is, return to execute step S204.

[0084] Step S208: assign the latest log sequence value to the end log sequence number.

[0085] Step S209: put the changed data between the value of the starting end log sequence number and the value of the ending log sequence number into a blocking queue through the reading thread.

[0086] Step S210 : Reading part of the changed data from the blocking queue through the parsing thread of the source database to obtain a plurality of parsed data corresponding to different source tables of the source database.

[0087] In step S211, each parsed data item is stored in a min-heap. The key value of each node in the min-heap is the start log sequence number of each committed transaction in the parsed data item. The min-heap structure includes the source table name, the min-heap depth, and multiple nodes. Each node structure includes the start log sequence number and a data list. The data list is used to store the parsed data item corresponding to the start log sequence number.

[0088] Step S212, determining whether the root nodes of the minimum heaps are consistent; if not, executing step S213; if yes, that is, when the root nodes of the minimum heaps are consistent, executing step S216.

[0089] Step S213: Determine that each minimum sequence number data does not include the same committed transaction as the unread change data. Step S214 may be executed after this step.

[0090] Step S214: flush the data with the smallest sequence number in the order of the starting log sequence number. Step S215 may be executed after this step S214.

[0091] Step S215, updating each minimum heap to which the minimum sequence number data is submitted; and after this step, executing the step of determining whether the root nodes of each minimum heap are consistent, that is, executing step S212 after this step; or executing step S210 or step S211 after this step.

[0092] Step S216, determine whether there is a minimum heap with a depth greater than 1; if so, also determine that each minimum sequence number data does not include the same committed transaction as the unread change data, that is, step S213 can be executed; if not, step S217 can be executed.

[0093] Step S217: Determine whether the minimum sequence number data and the unread change data include the same committed transaction. After this step, step S218 or step S210 may be executed.

[0094] Step S218: Sleep for a first preset time period; after this step, the process may return to step S212 or step S211.

[0095] It is understandable that the minimum heap can quickly sort the data stored therein, and the root node of the minimum heap is the data with the smallest key value stored in the minimum heap, and then step S211 stores each parsed data according to the minimum heap, and the search for the minimum sequence number data in the parsed data can be quickly completed. And because the start log sequence number of the cross-table transaction corresponding to the operation on different source tables is the same, step S212 determines whether the root nodes of each minimum heap are consistent, and specifically, it can be determined based on whether the key values ​​of the root nodes of each minimum heap are consistent. What needs to be understood about step S216 is that, when the root nodes of each minimum heap are consistent, the minimum sequence number data stored in these root nodes may still include the same cross-table transaction as the unread change data. We can then further determine whether the depth of the minimum heap where these root nodes are located is greater than 1. If there is a minimum heap with a depth greater than 1, it means that in addition to the node storing the minimum sequence number data including the cross-table transaction, the minimum heap also stores other nodes, that is, in addition to the minimum sequence number data including the cross-table transaction, the minimum heap also stores the parsed data including other committed transactions, which means that the committed transactions included in the minimum sequence number data stored in the root nodes of all the minimum heaps at this time are complete cross-table transactions. In the case of a minimum heap with a depth greater than 1, it can be determined that each minimum sequence number data does not include the same committed transaction as the unread change data; on the contrary, if there is no minimum heap with a depth greater than 1, when the root nodes of each minimum heap are consistent, it means that the minimum sequence number data stored in these root nodes may still include the same cross-table transaction as the unread change data, that is, it is determined that the minimum sequence number data and the unread change data include the same committed transaction. Therefore, steps S211 to S213, step S216 and step S217 can facilitate the quick and efficient completion of step S102, and ensure that the database can use fewer computing resources when running step S102, thereby ensuring the operational performance of the database.

[0096] It can also be understood that, in step S214, after the minimum sequence number data is flushed to disk in the order of the starting log sequence number, step S215 can update each minimum heap to which the minimum sequence number data has been submitted, and the minimum sequence number data stored in the root node of the minimum heap will be updated, so as to judge whether the root nodes of each minimum heap are consistent in the next round, search for the minimum sequence number data in each parsed data in the next round, and judge whether at least one of the minimum sequence number data includes the same committed transaction as the unread change data, and then each minimum sequence number data can be flushed to disk round by round, and the flushing of the parsed data can be completed round by round, and further, the flushing and synchronization of the change data can be completed step by step.

[0097] It is also understood that steps S202, S203, S204, S207, S208, and S209 can read the changed data within the specified interval. Specifically, the configuration of the breakpoint structure can ensure the continuity of reading the changed data in steps S202, S203, S204, S207, S208, and S209, and avoid repeated reading of the changed data, thereby ensuring data consistency in the database.

[0098] Steps S201, S205, S206, and S207 can prompt the database change data parsing method of this embodiment to read the change data when it is run for the first time, specify the starting position for reading the change data when the database change data parsing method of this embodiment is run for the first time, and ensure the normal operation of the database change data parsing method.

[0099] In addition, under the configuration of the blocking queue, when the blocking queue is empty, the parsing thread will be blocked and stop reading the change data from the group match queue until new change data is added to the blocking queue; when the blocking queue is full, the reading thread that adds change data to the blocking queue will be blocked until there is a vacancy in the blocking queue. Therefore, the database change data parsing method of this embodiment can effectively coordinate the speed difference between the parsing thread and the reading thread by transferring change data through the blocking queue, suppress or even avoid busy waiting and waste of database resources, and ensure the performance of the database. Moreover, in this embodiment, the capacity of the blocking queue can be configured to be larger, or configured as an unbounded blocking queue, so that all the change data read by the reading thread can be placed in the blocking queue. In some other alternative embodiments, when the capacity of the blocking queue is small, the specific implementation steps of step S101, reading part of the change data of the source database may be (or step S209 and step S210 may be correspondingly replaced by), putting the part of the change data between the value of the starting log sequence number and the value of the ending log sequence number into the blocking queue through the reading thread, and reading (part of) the change data from the blocking queue through the parsing thread of the source database, to obtain multiple parsed data corresponding to different source tables of the source database.

[0100] In this embodiment, the breakpoint structure also includes a commit sequence number, and after step S214 and before step S215, the commit sequence number may be assigned the value of the largest starting log sequence number among the minimum sequence number data. Furthermore, if the database change data parsing method experiences a failure or goes offline during operation and is restarted, the database change data parsing method of this embodiment may further include assigning the commit sequence number value to the starting end log sequence number and executing step S204. This ensures that the database change data parsing method of this embodiment reads change data continuously, avoids repeated reads of change data, and ensures database data consistency.

[0101] In this embodiment, after step S209 and before step S210, the database change data parsing method of this embodiment may further include assigning the end log sequence number to an invalid value; before step S208, and specifically, when the value of the starting end log sequence number is less than the latest log sequence value and before step S208, the database change data parsing method of this embodiment may further include determining whether the value of the end log sequence number is an invalid value. If the end log sequence number is an invalid value, step S208 may be executed; if the end log sequence number is not an invalid value, it may be determined that an abnormality has occurred in the operation of the database change data parsing method.

[0102] It is understandable that when the terminal log sequence number is an invalid value, it indicates that the database change data parsing method is in the state of reading the change data for the first time, or is in the state of reading the change data multiple times, that is, when the terminal log sequence number is an invalid value, it indicates that the database change data parsing method is in a normal operating state. Furthermore, when the terminal log sequence number is not an invalid value, it can be determined that an abnormality has occurred in the operation of the database change data parsing method. Therefore, the step of determining whether the value of the terminal log sequence number is an invalid value can realize the detection of the operating state of the database change data parsing method of this embodiment, and further ensure that the database change data parsing method of this embodiment can read the change data normally, further ensuring the data consistency of the database.

[0103] Reference Figure 3 In this embodiment, the change data is stored in the change tracking table corresponding to the source table; and the step of placing the change data between the value of the starting end log sequence number and the value of the ending log sequence number into the blocking queue by the reading thread includes:

[0104] Step S301, reading the change data between the value of the starting end log sequence number and the value of the ending log sequence number on multiple change tracking tables in parallel through a reading thread;

[0105] Step S302: put the changed data between the value of the start log sequence number and the value of the end log sequence number into a blocking queue through a reading thread.

[0106] It can be understood that the database change data parsing method of this embodiment can prompt the reading thread to read the change data on multiple change tracking tables in parallel. Compared with the prior art method of reading the change tracking table by polling all change tracking tables, the database change data parsing method of this embodiment can efficiently and quickly complete the reading of the change data, breaking through the performance bottleneck of reading caused by reading the change tracking table by polling all change tracking tables in the prior art, thereby improving the operating performance of the database.

[0107] Figure 4 and Figure 5 are all flowcharts of a method for parsing database change data according to yet another embodiment of the present invention. Specifically, Figure 4 1 is a schematic diagram of a process for reading change data in a database change data parsing method according to another embodiment of the present invention; Figure 5 The figure is a schematic diagram of a process for parsing changed data in a method for parsing changed data of a database according to another embodiment of the present invention.

[0108] Reference Figure 4 In this embodiment, the database change data parsing method may include:

[0109] Step S401, obtain the last breakpoint event and initialize the breakpoint information (fromLSN, toLSN, commitLSN), where fromLSN is the starting end log sequence number, toLSN is the end log sequence number, and commitLSN is the commit sequence number.

[0110] Step S402, query the maximum maxLSN (latest log sequence value) in the database. If there is no data, it means that no change data has been generated. Step S402 is executed again after a period of time.

[0111] Step S403, determine whether fromLSN is null (invalid value). If fromLSN is null, it means that the program has not recorded a breakpoint, and execute step S404; if fromLSN is not null, directly execute step S405.

[0112] Step S404, fromLSN is set to the current maxLSN.

[0113] Step S405, determine whether fromLSN is greater than or equal to maxLSN. If fromLSN is greater than or equal to maxLSN, it means that the latest data has been read, and step S402 is executed again after a period of time; if fromLSN is less than maxLSN, step S406 is executed.

[0114] In step S406, it is determined whether toLSN is null (which will be reset in step S407). If toLSN is null (which will be reset in step S408), it indicates that the program has not recorded a breakpoint, or a round of LSN (log sequence number) interval reading has been performed after startup, and step S407 is executed. If toLSN is not null, the database change data parsing method of this embodiment can be stopped.

[0115] Step S407: assign the current maxLSN to toLSN, and then execute step S408.

[0116] Step S408: query all change tables for change data within the specified range from fromLSN to toLSN through multiple threads.

[0117] Step S409: After all queries are completed, all changed data are put into the changed data blocking queue through the reading thread, toLSN is assigned to fromLSN, and then toLSN is assigned to null, and then step S402 is executed.

[0118] Reference Figure 5 In this embodiment, the database change data parsing method may further include:

[0119] Step S501: store the changed data of each change table in a minimum heap.

[0120] Step S502: Obtain the minimum node minNode (root node) of all minimum heaps.

[0121] Step S503, determine whether the root nodes of all minimum heaps are consistent; if inconsistent, it means that the transaction data corresponding to the minNode has been parsed, and then execute step S504; if all minimum nodes are consistent, execute step S506.

[0122] Step S504: collect the obtained Lists (data lists) in all root nodes and reorder them in the order of the starting log sequence number.

[0123] Step S505 , committing the data (ie persisting the changed data in all the sorted root nodes on the disk of the source database), recording the largest LSN among the changed data in all the root nodes as commitLSN; and executing step S502 after this step.

[0124] Step S506, determine whether there is a minimum heap with a depth greater than 1; if there is a minimum heap with a depth greater than 1, it means that the transaction data corresponding to the root node of the minimum heap greater than 1 has been parsed, and then execute step S504; if the depth of all current small heaps is not greater than 1, it means that there is no committed transaction at present, and then execute step S507.

[0125] Step S507: sleep for a period of time, and then execute step S502.

[0126] It should be noted that the steps of this embodiment can achieve or be replaced by the beneficial technical effects that can be achieved by the corresponding steps of the aforementioned embodiment. For example, steps S401 to S409 can replace steps S201 to S209 of the aforementioned embodiment, and steps S501 to S507 can achieve steps S101 to S103 of the aforementioned embodiment, or be replaced by steps S210 to S218. The corresponding beneficial technical effects are not repeated here.

[0127] The following combination Figure 6 The computer program product of this embodiment will be described in detail. Figure 6 Schematic diagram of the structure of a computer program product according to an embodiment of the present invention. Figure 6 In this embodiment, the computer program product 10 includes a computer program 11. When the computer program 11 is executed by a processor, the database change data parsing method of the above embodiment is implemented.

[0128] The computer program product 10 of this embodiment can implement the above-mentioned method for analyzing database change data. Therefore, the computer program product 10 of this embodiment also has the beneficial technical effects achieved by the above-mentioned method for analyzing database change data.

[0129] The following combination Figure 7 The computer-readable storage medium of this embodiment is described in detail. Figure 7 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention, with reference to Figure 7 In this embodiment, a computer program 11 is stored on a computer-readable storage medium 20. When the computer program 11 is executed by a processor, the database change data parsing method of the above embodiment is implemented.

[0130] The computer-readable storage medium 20 of this embodiment can implement the above-mentioned method for analyzing database change data. Therefore, the computer-readable storage medium 20 of this embodiment also has the beneficial technical effects achieved by the above-mentioned method for analyzing database change data.

[0131] It should be noted that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of instructions for implementing logical functions, and can be specifically implemented in any computer-readable storage medium 20 for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0132] For the purposes of the description of this embodiment, the computer-readable storage medium 20 can be any device that can contain, store, communicate, propagate, or transmit a program for use with an instruction execution system, device, or apparatus, or in conjunction with such an instruction execution system, device, or apparatus. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection portion having one or more wirings (electronic device), a portable computer disk cartridge (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable storage medium 20 can even be paper or other suitable medium on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, and then editing, interpreting, or processing in other suitable ways as necessary, and then stored in a computer memory.

[0133] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.

[0134] The following combination Figure 8 The computer device of this embodiment will be described in detail. Figure 8 is a structural block diagram of a computer device according to an embodiment of the present invention, referring to Figure 8 In this embodiment, the computer device 30 includes a memory 31, a processor 32 and a computer program 11 stored in the memory 31, and the processor 32 implements the above-mentioned database change data analysis method when executing the computer program 11.

[0135] The computer device 30 of this embodiment can implement the database change data parsing method of the above embodiment. Therefore, the computer device 30 of this embodiment also has the beneficial technical effects that can be achieved by the above database change data parsing method.

[0136] The computer device 30 may be, for example, a server, a desktop computer, a laptop computer, a tablet computer, or a smartphone. In some examples, the computer device 30 may be a cloud computing node. The computer device 30 may be described in the general context of computer system instructions (such as program modules) executed by a computer system. Typically, a program module may include routines, programs, object programs, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. The computer device 30 may be implemented in a distributed cloud computing environment where remote processing devices linked via a communication network perform tasks. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media, including storage devices.

[0137] The computer device 30 may include a processor 32 adapted to execute stored instructions, and a memory 31 that provides temporary storage space for the instructions during operation. The processor 32 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 31 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0138] The processor 32 can be connected to an I / O interface (input / output interface) suitable for connecting the computer device 30 to one or more I / O devices (input / output devices) via a system interconnect (e.g., PCI, PCI-Express, etc.). The I / O devices may include, for example, a keyboard and a pointing device, wherein the pointing device may include a touchpad or a touch screen, etc. The I / O devices may be built-in components of the computer device 30, or may be devices externally connected to the computing device.

[0139] Processor 32 can also be linked to the display interface that is suitable for connecting computer device 30 to display device through system interconnection.Display device can comprise the display screen that is built-in component of computer device 30.Display device can also comprise the computer monitor, television or projector etc. that are externally connected to computer device 30.In addition, network interface controller (network interface controller, NIC) can be suitable for connecting computer device 30 to network through system interconnection.In certain embodiments, NIC can use any suitable interface or protocol (such as Internet Small Computer System Interface etc.) to transmit data.Network can be cellular network, radio network, wide area network (WAN)), local area network (LAN) or Internet etc.Remote device can be connected to computing device through network.

[0140] At this point, those skilled in the art will recognize that, although a number of exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications consistent with the principles of the present invention may be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention should be understood and deemed to cover all such other variations or modifications.

Claims

1. A method for parsing database change data, comprising: Reading part of the changed data of the source database to obtain a plurality of parsed data corresponding to different source tables of the source database; Searching for the minimum sequence number data in each of the parsed data, and determining whether at least one of the data includes the same committed transaction as the unread change data, wherein the minimum sequence number data is the data with the smallest start log sequence number of the committed transaction in the transaction log in the corresponding parsed data; If not, the data with the minimum sequence number is flushed to disk in the order of the starting log sequence number.

2. The database change data analysis method according to claim 1, wherein: The step of searching for the minimum sequence number data in each of the parsed data and determining whether at least one of the data includes the same committed transaction as the unread change data includes: storing each of the parsed data according to a minimum heap, wherein the key value of each node of the minimum heap is the start log sequence number of a different committed transaction in the parsed data; Determine whether the root nodes of the minimum heaps are consistent; If not, it is determined that each of the minimum sequence number data does not include the same commit transaction as the unread change data.

3. The database change data analysis method according to claim 2, wherein: In the case that the root nodes of the minimum heaps are consistent, the step of searching for the minimum sequence number data in each of the parsed data and determining whether at least one of the data includes the same committed transaction as the unread change data further includes: Determine whether there is a minimum heap with a depth greater than 1; If so, similarly determining that each of the minimum sequence number data does not include the same committed transaction as the unread change data; If not, it is determined that the minimum sequence number data and the unread change data include the same committed transaction.

4. The database change data analysis method according to claim 3, wherein: After the step of flushing the minimum sequence number data to disk in the order of the start log sequence numbers, the change data parsing method further includes: Updating each of the minimum heaps to which the minimum sequence number data is submitted; Execute the step of determining whether the root nodes of the minimum heaps are consistent.

5. The database change data analysis method according to claim 2, wherein: The structure of the minimum heap includes the table name of the source table, the depth of the minimum heap and multiple nodes. The structure of each node includes the starting log sequence number and a data list. The data list is used to store the parsed data corresponding to the starting log sequence number.

6. The database change data analysis method according to claim 1, wherein: In a case where the minimum sequence number data and the unread change data include the same committed transaction, the change data parsing method further includes: Sleep for a first preset time period, and then perform the step of searching for the minimum sequence number data in each of the parsed data.

7. The database change data analysis method according to claim 1, wherein: The step of reading part of the changed data of the source database includes: Reading part of the change data from the blocking queue through the parsing thread of the source database; and Before the step of reading part of the change data from the blocking queue by the parsing thread of the source database, the change data parsing method further includes: Obtaining a breakpoint structure of a reading thread of the source database, wherein the breakpoint structure includes a start log sequence number and an end log sequence number of the change data last read by the reading thread; Assign the value of the end log sequence number to the start log sequence number; Query the latest log sequence value in the source database, where the latest log sequence value is the value of the start log sequence number of the latest committed transaction in the source database on the transaction log; Determine whether the value of the starting end log sequence number is greater than or equal to the latest log sequence value; If not, assign the latest log sequence value to the end log sequence number; The change data between the value of the starting end log sequence number and the value of the ending log sequence number are put into a blocking queue by the reading thread; and In a case where the value of the starting end log sequence number is greater than or equal to the latest log sequence value, the process returns to the step of querying the latest log sequence value in the source database.

8. The database change data analysis method according to claim 7, wherein: When the reading thread is run for the first time, and before the step of obtaining the breakpoint structure of the reading thread of the source database, the change data parsing method further includes: Initializing the breakpoint structure to assign preset values ​​to the starting end log sequence number and the ending end log sequence number respectively; and After the step of querying the latest log sequence value in the source database and before the step of determining whether the value of the starting end log sequence number is greater than or equal to the latest log sequence value, the change data parsing method further includes: Determine whether the value of the starting end log sequence number is the preset value; If so, assign the latest log sequence value to the starting end log sequence number, and then perform the step of determining whether the value of the starting end log sequence number is greater than or equal to the latest log sequence value; If not, the step of determining whether the value of the starting end log sequence number is greater than or equal to the latest log sequence value is performed.

9. The database change data analysis method according to claim 7, wherein: The change data is stored in a change tracking table corresponding to the source table; and The step of placing the changed data between the value of the starting end log sequence number and the value of the ending log sequence number into a blocking queue by the reading thread includes: Reading the change data between the value of the starting end log sequence number and the value of the ending log sequence number on a plurality of the change tracking tables in parallel by the reading thread; The changed data between the value of the starting end log sequence number and the value of the ending log sequence number are put into a blocking queue through a reading thread.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for parsing database change data according to any one of claims 1 to 9 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for parsing database change data according to any one of claims 1 to 9 is implemented.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the database change data parsing method according to any one of claims 1 to 9 when executing the computer program.

Images