Data synchronization method, device, electronic device and storage medium

Abstract

The embodiments of the present application disclose a data synchronization method, device, electronic device and storage medium, which relate to the field of computer technology, including: adding a second synchronization link according to a synchronization request, the second synchronization link and the first synchronization link share a log reading node; using a data extraction node to obtain the current log sequence number of the log file of the original database, and sending a new breakpoint message to the log reading node; using the data extraction node to determine the target breakpoint of the second synchronization link according to the current log sequence number and the current read sequence number, and using the log reading node to save the target breakpoint; using a transaction parsing node to obtain candidate log records based on the target breakpoint, and performing transaction parsing on the candidate log records to obtain transaction data to be synchronized; using a data loading node to synchronize the transaction data to be synchronized to a second target database, thereby realizing a one-to-many data synchronization function, reducing the risk of data loss, and improving the efficiency and accuracy of data synchronization.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a data synchronization method, device, electronic device and storage medium. Background Art

[0002] Currently, when realizing data synchronization between the original database and the target database, the data of the original database is first synchronized to the target database in full through the data extraction node, and the log file of the original database is refreshed and read through the log reading node, and the read log record is sent to the transaction parsing node for transaction parsing. Then, when it is determined that the log record is an operation to change the data of the original database, the transaction data corresponding to the log record is sent to the data loading node, and then the transaction data is synchronized to the target database through the data loading node.

[0003] The above method can accurately realize one-to-one synchronization requirements, but when realizing one-to-many synchronization requirements, for example, synchronizing the data of the original database to multiple target databases, or synchronizing the data of different data tables of the original database to different target databases, since the log file of the database is a single continuous file, and the reading of the log file can only be read sequentially from a specific position, if the one-to-many data synchronization is realized according to the above method, multiple log reading nodes will repeatedly read the same log file, resulting in a waste of resources; and, when a new synchronization link is dynamically added, the data extraction node of the synchronization link will query the current log sequence number of the log file, and determine the current log sequence number as the starting point for incremental synchronization of the synchronization link, but the query process of the current log sequence number takes a certain amount of time, and all synchronization links are asynchronously operated, that is, after the data extraction node of the synchronization link completes the query process of the current log sequence number, the log reading nodes of other synchronization links may have read and processed the log records corresponding to the current log sequence number. At this time, if the current log sequence number is directly determined as the starting point for incremental synchronization of the synchronization link, there is a risk of data loss.

[0004] Therefore, when implementing one-to-many synchronization requirements, the data synchronization method in the prior art has the situation where multiple log reading nodes repeatedly read the same log file, and there may be a risk of data loss, which in turn causes problems such as reduced reading efficiency and data synchronization confusion, resulting in low efficiency and accuracy of data synchronization. Summary of the invention

[0005] The embodiments of the present application provide a data synchronization method, device, electronic device and storage medium, which realize the data synchronization function and reduce the risk of data loss, so as to solve the problems of reduced reading efficiency and data synchronization confusion in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a data synchronization method, which is applied to an electronic device, wherein the electronic device includes a first synchronization link for synchronizing data in an original database to a first target database, and the method includes:

[0007] A second synchronization link is added according to the synchronization request, the second synchronization link is used to synchronize the data of the original database to the second target database, the second synchronization link and the first synchronization link share a log reading node, and the second synchronization link also includes a data extraction node, a transaction parsing node and a data loading node;

[0008] The data extraction node is used to obtain the current log sequence number of the log file of the original database, and a new breakpoint message is sent to the log reading node. The new breakpoint message is used to notify the log reading node to feedback the current read sequence number to the data extraction node;

[0009] Determine the target breakpoint of the second synchronization link according to the current log sequence number and the current read sequence number using the data extraction node, and save the target breakpoint using the log reading node;

[0010] Use the transaction parsing node to obtain candidate log records based on the target breakpoint, and perform transaction parsing on the candidate log records to obtain the transaction data to be synchronized;

[0011] The transaction data to be synchronized is synchronized to the second target database using the data loading node.

[0012] In an embodiment of the present application, a second synchronization link can be added according to a synchronization request; a data extraction node is used to obtain the current log sequence number of the log file of the original database, and a new breakpoint message is sent to the log reading node; the data extraction node is used to determine the target breakpoint of the second synchronization link according to the current log sequence number and the current read sequence number, and the log reading node is used to save the target breakpoint; a transaction parsing node is used to obtain candidate log records based on the target breakpoint, and transaction parsing is performed on the candidate log records to obtain transaction data to be synchronized; a data loading node is used to synchronize the transaction data to be synchronized to the second target database, thereby realizing a one-to-many data synchronization function. In the prior art, when realizing one-to-many synchronization requirements, each synchronization link corresponds to a log reading node. At this time, there are multiple log reading nodes that repeatedly read the same log file. In the above technical scheme, the first synchronization link and the second synchronization link share the log reading node, that is, multiple synchronization links share the same log reading node, which can avoid the situation where multiple log reading nodes repeatedly read the same log file, save computing resources, and thus improve the reading efficiency, solve the problem of reduced reading efficiency in the prior art, thereby improving the efficiency of data synchronization; and, in the above technical scheme, the target breakpoint is determined by the current log sequence number and the current reading sequence number, which can avoid the situation in the prior art that after the data extraction node completes the query process of the current log sequence number, the log reading nodes of other synchronization links may have read and processed the log records corresponding to the current log sequence number, effectively reducing the risk of data loss, thereby improving the accuracy of data synchronization, solving the problem of data synchronization confusion in the prior art, and thus improving the user experience.

[0013] In a second aspect, an embodiment of the present application provides a data synchronization device, which is applied to an electronic device, wherein the electronic device includes a first synchronization link for synchronizing data in an original database to a first target database, and the device includes:

[0014] A link adding module, used to add a second synchronization link according to a synchronization request, the second synchronization link is used to synchronize the data of the original database to the second target database, the second synchronization link and the first synchronization link share a log reading node, and the second synchronization link also includes a data extraction node, a transaction parsing node and a data loading node;

[0015] An acquisition module is used to obtain the current log sequence number of the log file of the original database by using the data extraction node, and send a new breakpoint message to the log reading node, and the new breakpoint message is used to notify the log reading node to feedback the current read sequence number to the data extraction node;

[0016] A breakpoint determination module, used to determine the target breakpoint of the second synchronization link according to the current log sequence number and the current read sequence number using the data extraction node, and save the target breakpoint using the log reading node;

[0017] A parsing module is used to obtain candidate log records based on the target breakpoint using the transaction parsing node, and perform transaction parsing on the candidate log records to obtain transaction data to be synchronized;

[0018] The synchronization module is used to synchronize the transaction data to be synchronized to the second target database by using the data loading node.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0020] at least one processor; and a memory communicatively coupled to the at least one processor;

[0021] The memory stores a computer program that can be executed by at least one processor, and the computer program is executed by at least one processor so that the at least one processor can execute the data synchronization method of any embodiment of the present application.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, a data synchronization method as in any embodiment of the present application is implemented.

[0023] The description of the second, third and fourth aspects in this application can refer to the detailed description of the first aspect; and the beneficial effects described in the second, third and fourth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0024] In this application, the name of the above-mentioned data synchronization device does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies.

[0025] These and other aspects of the present application will become more apparent from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 It is a flowchart of a data synchronization method provided by an embodiment of the present application;

[0028] Figure 2is another flowchart of the data synchronization method provided in an embodiment of the present application;

[0029] Figure 3 is a structural diagram of a data synchronization device provided in an embodiment of the present application;

[0030] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0032] It should be noted that the terms "first", "second", "target", and "original" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having", and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0033] Figure 1 It is a flow chart of the data synchronization method provided in an embodiment of the present application. This embodiment can be applied to scenarios where it is necessary to synchronize data in an original database to different target databases when realizing one-to-many synchronization requirements. A data synchronization method provided in this embodiment can be executed by a data synchronization device provided in an embodiment of the present application, and the device can be implemented in software and / or hardware. In a specific embodiment, the data synchronization device can be integrated in an electronic device, and the electronic device can include a first synchronization link for synchronizing data in the original database to a first target database. For example, the electronic device can be a computer or a server. The execution subject of this method can be an electronic device, see Figure 1 The data synchronization method of this embodiment includes but is not limited to the following steps:

[0034] S110: Add a second synchronization link according to the synchronization request.

[0035] Among them, the synchronization request is a signal or instruction that triggers the data synchronization operation. The user can send a synchronization request to the electronic device according to actual usage requirements; the synchronization request may include the source database and target database that need to be synchronized, that is, the synchronization request may include the original database and the second target database.

[0036] The second synchronization link is used to synchronize the data of the original database to the second target database to maintain data consistency or meet specific data usage requirements. At this time, the electronic device can synchronize the data of the original database to different target databases; the second synchronization link shares the log reading node with the first synchronization link, and the second synchronization link also includes a data extraction node, a transaction parsing node and a data loading node; the log reading node is used to continuously refresh and read the log files of the original database in sequence; the data extraction node is used for full synchronization, that is, to obtain historical data from the original database and synchronize these historical data to the second target database; the transaction parsing node is used to perform in-depth analysis of the log records obtained from the log reading node to understand the transaction logic therein and parse the transaction data; the data loading node is used to load the transaction data processed by the transaction parsing node into the second target database.

[0037] Specifically, the electronic device has established a first synchronization link to synchronize the data of the original database to the first target database, and when the electronic device executes the first synchronization link, the user can send a new synchronization request to the electronic device according to actual usage needs to request the electronic device to establish and execute a new synchronization link, wherein the synchronization request is used to request the electronic device to synchronize the data of the original database to the second target database, that is, the synchronization request may include the original database and the second target database.

[0038] Afterwards, the electronic device can receive the synchronization request sent by the user, and add a second synchronization link based on the synchronization request, that is, the synchronization request can be parsed, and the original database and the second target database can be extracted, and then a second synchronization link is established to synchronize the data of the original database to the second target database, and the first synchronization link and the second synchronization link are controlled to share the log reading node to prevent the problem of multiple log reading nodes repeatedly reading the same log file, and then an identifier of the second synchronization link is generated, and the data extraction node, transaction parsing node and data loading node of the second synchronization link are initialized, for example, the data extraction node is configured to be connected to the original database to ensure that the data extraction node can access the original database and extract data from it; the data loading node is configured to be connected to the second target database to ensure that the data loading node can make data changes to the second target database; a communication connection is established between the data extraction node and the log reading node, a communication connection is established between the transaction parsing node and the log reading node, and a communication connection is established between the transaction parsing node and the data loading node; at this time, there are two synchronization links in the electronic device, namely the first synchronization link and the second synchronization link, and the first synchronization link and the second synchronization link are executed asynchronously.

[0039] It should be noted that the first synchronization link and the second synchronization link share a log reading node, but do not share a data extraction node, a transaction parsing node, and a data loading node.

[0040] S120: Utilize the data extraction node to obtain the current log sequence number of the log file of the original database, and send a new breakpoint message to the log reading node.

[0041] The log file is used to record all transaction operation information in the original database; illustratively, the log file may be a transaction log file.

[0042] The log sequence number is a number used to identify the log record position in the log file, and the log sequence number in the log file increases gradually, that is, it increases in one direction. The current log sequence number is the maximum log sequence number in the log file when the data extraction node is started.

[0043] The newly added breakpoint message is used to notify the log reading node to feedback the current reading sequence number to the data extraction node; the current reading sequence number is the log sequence number fed back by the log reading node to the data extraction node, representing the log record position that the log reading node is reading.

[0044] Specifically, after adding a second synchronization link, the data extraction node of the second synchronization link can be started, and the data extraction node can be used to obtain the largest log sequence number, that is, the current log sequence number, from the log file of the original database. That is, the data extraction node obtains the current log sequence number of the log file through the interface or query statement provided by the original database (such as the DBCC LOGINFO command provided by the SQL Server database); then the data extraction node can be used to generate a new breakpoint message based on the identifier of the second synchronization link, and send the new breakpoint message to the log reading node to notify the log reading node to feedback the current reading sequence number to the data extraction node.

[0045] After receiving the new breakpoint message, the log reading node can parse the new breakpoint message and extract the identifier of the second synchronization link to determine the synchronization link that sends the new breakpoint message, and then obtain the log sequence number corresponding to the log record currently being read, that is, the current reading sequence number, and send the current reading sequence number to the data extraction node of the corresponding synchronization link based on the identifier of the second synchronization link, that is, send the current reading sequence number to the data extraction node of the second synchronization link.

[0046] Optionally, when the log reading node receives a new breakpoint message, it indicates that there are multiple synchronization links. At this time, the log reading node can be used to store the log records read from the log file into the cache area to save the read log records.

[0047] S130: Determine the target breakpoint of the second synchronization link according to the current log sequence number and the current read sequence number using the data extraction node, and save the target breakpoint using the log reading node.

[0048] Among them, the target breakpoint is the starting point for incremental synchronization of the second synchronization link, which ensures that the second synchronization link can start incremental synchronization from a suitable log record position, reduces the risk of data loss or repeated synchronization, and ensures the accuracy and completeness of data synchronization; at the same time, the target breakpoint is also the reference point for the second synchronization link to perform full synchronization.

[0049] Specifically, when the data extraction node obtains the current log sequence number and sends a new breakpoint message to the log reading node, the log reading node will continuously read the log file, that is, the data extraction node and the log reading node asynchronously perform their respective operations. It can be seen that when the data extraction node obtains the current log sequence number, but the log reading node has not received the new breakpoint message, the log reading node will continuously read the log file and send the log sequence number currently being read to the data extraction node only when the new breakpoint message is received. At this time, it is possible that the data extraction node has read and processed the log file after it obtains the current log sequence number, but the log reading node has not received the new breakpoint message. After sorting out the log records corresponding to the current log sequence number, after the data extraction node obtains the current log sequence number and the current read sequence number, it is necessary to use the data extraction node to determine the target breakpoint of the second synchronization link according to the current log sequence number and the current read sequence number to reduce the risk of data loss. For example, the data extraction node can compare the current log sequence number and the current read sequence number, and determine the target breakpoint of the second synchronization link according to the size relationship between the current log sequence number and the current read sequence number. For example, when the current log sequence number is greater than the current read sequence number, the target breakpoint is determined to be the current log sequence number; when the current log sequence number is not greater than the current read sequence number, the target breakpoint is determined to be the current read sequence number.

[0050] After using the data extraction node to determine the target breakpoint, the log reading node can be used to save the target breakpoint, that is, the data extraction node can generate a save breakpoint message based on the identifier and target breakpoint of the second synchronization link, and send the save breakpoint message to the log reading node; after receiving the save breakpoint message, the log reading node can parse the save breakpoint message to obtain the identifier and target breakpoint of the second synchronization link, and encapsulate the identifier and target breakpoint of the second synchronization link according to a preset format, such as encapsulating the identifier and target breakpoint of the second synchronization link according to the JS key-value pair data (JavaScript Object Notation, JSON) format, and then saving the encapsulated data to a preset storage structure; the preset storage structure is used to store the mapping relationship between the identifier of the synchronization link and the breakpoint.

[0051] Optionally, after determining the target breakpoint using the data extraction node, the data extraction node can be used to perform full synchronization of the second synchronization link, that is, the data extraction node can obtain from the original database all the data that has been saved in the original database before the log record corresponding to the target breakpoint occurs, and synchronize these data to the second target database based on the preset data mapping relationship; at the same time, the transaction parsing node can be used to perform incremental synchronization starting from the log record corresponding to the target breakpoint. The preset data mapping relationship is a pre-set mapping relationship used to describe the corresponding relationship between the data structure and data content between the original database and the second target database.

[0052] S140: Utilize the transaction parsing node to obtain candidate log records based on the target breakpoint, and perform transaction parsing on the candidate log records to obtain transaction data to be synchronized.

[0053] Among them, the candidate log record is a log record whose log sequence number is not less than the target breakpoint, and the candidate log records include log records that need to be synchronized to the second target database and log records that do not need to be synchronized to the second target database. Exemplarily, when the candidate log record is an operation to change data in the original database, the candidate log record is a log record that needs to be synchronized to the second target database; when the candidate log record is not an operation to change data in the original database, the candidate log record is a log record that does not need to be synchronized to the second target database.

[0054] The transaction data to be synchronized is the transaction data that needs to be synchronized to the second target database. The transaction data to be synchronized may include information such as the change operation type, the change object (such as the change data table and the change field) and the change field data.

[0055] Specifically, after using the log reading node to save the target breakpoint, the transaction parsing node can be used to obtain candidate log records based on the target breakpoint. For example, the transaction parsing node can obtain log records with log sequence numbers not less than the target breakpoint from the cache area to obtain candidate log records.

[0056] Afterwards, the transaction parsing node can be used to perform transaction parsing on the candidate log records to obtain the transaction data to be synchronized. For example, the transaction parsing node can parse the candidate log records and determine whether the candidate log records are log records that need to be synchronized to the second target database. If the candidate log record is an operation to change data in the original database, it indicates that the candidate log record is a log record that needs to be synchronized to the second target database, that is, the change operation that the second synchronization link needs to synchronize. At this time, the transaction data corresponding to the candidate log record is determined as the transaction data to be synchronized; otherwise, it can be determined whether other candidate log records are log records that need to be synchronized to the second target database.

[0057] S150: Synchronize the transaction data to be synchronized to the second target database using the data loading node.

[0058] Specifically, after obtaining the transaction data to be synchronized, the data loading node can be used to synchronize the transaction data to be synchronized to the second target database, that is, the data loading node can parse the transaction data to be synchronized, and extract the change operation type, change object (such as change data table and change field) and change field data, and then query the preset data mapping relationship based on the change object to obtain the object to be changed of the second target database, that is, the data table to be changed and the field to be changed, and determine the data corresponding to the field to be changed based on the preset data mapping relationship and the correspondence between the change object and the change field data, and then determine the data change operation of the second target database based on the change operation type, the data table to be changed, the field to be changed and the data corresponding to the field to be changed, and then control the second target database to execute the data change operation to synchronize the transaction data to be synchronized to the second target database.

[0059] The technical solution of the embodiment of the present application can add a second synchronization link according to the synchronization request; use the data extraction node to obtain the current log sequence number of the log file of the original database, and send a new breakpoint message to the log reading node; use the data extraction node to determine the target breakpoint of the second synchronization link according to the current log sequence number and the current read sequence number, and use the log reading node to save the target breakpoint; use the transaction parsing node to obtain candidate log records based on the target breakpoint, and perform transaction parsing on the candidate log records to obtain transaction data to be synchronized; use the data loading node to synchronize the transaction data to be synchronized to the second target database, thereby realizing a one-to-many data synchronization function. In the prior art, when realizing one-to-many synchronization requirements, each synchronization link corresponds to a log reading node. At this time, there are multiple log reading nodes that repeatedly read the same log file. In the above technical scheme, the first synchronization link and the second synchronization link share the log reading node, that is, multiple synchronization links share the same log reading node, which can avoid the situation where multiple log reading nodes repeatedly read the same log file, save computing resources, and thus improve the reading efficiency, solve the problem of reduced reading efficiency in the prior art, thereby improving the efficiency of data synchronization; and, in the above technical scheme, the target breakpoint is determined by the current log sequence number and the current reading sequence number, which can avoid the situation in the prior art that after the data extraction node completes the query process of the current log sequence number, the log reading nodes of other synchronization links may have read and processed the log records corresponding to the current log sequence number, effectively reducing the risk of data loss, thereby improving the accuracy of data synchronization, solving the problem of data synchronization confusion in the prior art, and thus improving the user experience.

[0060] The following further describes a data synchronization method provided by an embodiment of the present application. Figure 2 1 is another flow chart of the data synchronization method provided by the embodiment of the present application. The embodiment of the present application is optimized based on the above embodiments. Figure 2 The method of this embodiment includes but is not limited to the following steps:

[0061] S201. Add a second synchronization link according to a synchronization request.

[0062] Optionally, the second synchronization link is used to synchronize the data of the original data table in the original database to the second target database. In this case, the electronic device can synchronize the data of different data tables in the original database to different target databases.

[0063] S202: Using the data extraction node to obtain the current log sequence number of the log file of the original database, and sending a new breakpoint message to the log reading node.

[0064] Specifically, after the log reading node receives the new breakpoint message sent by the data extraction node, S203 and S206 can be executed synchronously, that is, the log reading node can be used to feedback the current reading sequence number to the data extraction node, and then execute S203; at the same time, the log reading node can be used to synchronously execute S206.

[0065] S203: Use the data extraction node to determine whether the current log sequence number is greater than the current read sequence number.

[0066] Specifically, after the data extraction node obtains the current read sequence number returned by the log reading node, the data extraction node can be used to compare the current log sequence number and the current read sequence number. If the current log sequence number is greater than the current read sequence number, it indicates that the log reading node did not read and process the log record corresponding to the current log sequence number when receiving the new breakpoint message. At this time, S204 can be executed; if the current log sequence number is not greater than the current read sequence number, it indicates that the log reading node has read and processed the log record corresponding to the current log sequence number when receiving the new breakpoint message. At this time, it can return to execute S202 to use the data extraction node to re-obtain the current log sequence number and the current read sequence number until the obtained current log sequence number is greater than the current read sequence number, and when the current log sequence number is greater than the current read sequence number, execute S204.

[0067] S204: When the current log sequence number is greater than the current read sequence number, the target breakpoint is determined to be the current log sequence number using the data extraction node.

[0068] Specifically, when the current log sequence number is greater than the current read sequence number, it indicates that the log reading node has not yet read the log record corresponding to the current log sequence number. At this time, there is no risk of data loss. Therefore, the data extraction node can be used to determine the current log sequence number as the target breakpoint.

[0069] S205: Use the log reading node to save the target breakpoint.

[0070] Specifically, S207 is then executed.

[0071] S206: Use the log reading node to store the log records read from the log file to the disk.

[0072] Specifically, after the log reading node receives the newly added breakpoint message sent by the data extraction node, the log reading node can be used to store the log records read from the log file to the disk, that is, when the file size of the first preset file on the disk does not exceed the preset threshold, the log reading node is used to store the log records read from the log file to the first preset file; when the file size of the first preset file on the disk exceeds the preset threshold, the log reading node is used to store the log records read from the log file to the second preset file on the disk, and the file name of the first preset file is determined as the maximum log sequence number in the first preset file. Wherein, the first preset file and the second preset file are files in the disk, which are used to store the log records read from the log file by the log reading node, and the log sequence numbers of the log records stored in the first preset file and the second preset file are unidirectionally increasing; the preset threshold is a preset file size limit value, which is used to determine whether the first preset file is full, so as to determine whether the log reading node stores the read log records to the first preset file or the second preset file. The user can adjust and set the preset threshold according to actual use requirements, and the embodiment of the present application does not make specific restrictions on this.

[0073] Specifically, when the file size of the first preset file on the disk does not exceed the preset threshold, the log reading node serializes the log records read from the log file and stores the serialized log records in the first preset file. Then, when the file size of the first preset file on the disk exceeds the preset threshold, the log reading node stores the serialized log records in the second preset file and determines the file name of the first preset file as the maximum log sequence number in the first preset file.

[0074] It should be noted that when the file size of the second preset file exceeds the preset threshold, the log reading node can be used to store the log records read from the log file to other preset files on the disk. The embodiment of the present application does not specifically limit the number of preset files on the disk; and the storage order of the first preset file and the second preset file can be changed. The embodiment of the present application does not specifically limit this, that is, the log reading node can first store the read log records to the second preset file, and when the file size of the second preset file exceeds the preset threshold, the read log records can be stored in the first preset file, and the file name of the second preset file is determined as the maximum log sequence number in the second preset file.

[0075] In an embodiment of the present application, by setting a preset threshold, the file size of the preset file can be controlled to prevent the preset file from being too large and affecting the I / O performance of the disk, and the log records can be stored in a dispersed manner, which helps to maintain efficient use of disk space; the file name of the first preset file is determined as the maximum log sequence number in the first preset file, so that the transaction parsing node can quickly determine the file where the target breakpoint is located based on the file name, thereby improving the acquisition rate of candidate log records.

[0076] After the log record read from the log file is stored in the disk by using the log reading node, S207 may be executed.

[0077] It should be noted that the execution order of S203 to S205 and S206 is synchronous.

[0078] S207: Using the transaction parsing node, obtain the log record whose log sequence number is not less than the target breakpoint from the disk to obtain the candidate log record.

[0079] Specifically, after determining the target breakpoint and using the log reading node to store the log records read from the log file to the disk, the transaction parsing node can be used to obtain the log records whose log sequence numbers are not less than the target breakpoint from the disk. That is, the transaction parsing node can determine the file where the target breakpoint is located based on the target breakpoint and the file name on the disk, and record it as the target file. Then, the target file is opened in read-only mode, and the query starts from the log sequence number of the first log record of the target file, filtering out all log records with log sequence numbers less than the target breakpoint until a log record with a log sequence number not less than the target breakpoint is found. The log record is then deserialized to obtain a candidate log record.

[0080] S208: Perform transaction analysis on the candidate log record using the transaction analysis node, and when the change data table of the candidate log record is the original data table, determine the transaction data corresponding to the candidate log record as the transaction data to be synchronized.

[0081] Specifically, after obtaining the candidate log record, the transaction parsing node can be used to perform transaction parsing on the candidate log record and extract the change object, namely the change data table and the change field, and then determine whether the change data table is the original data table. If the change data table is the original data table, it indicates that the candidate log record is a log record that needs to be synchronized to the second target database. At this time, the transaction data corresponding to the candidate log record can be determined as the transaction data to be synchronized; if the change data table is not the original data table, it indicates that the candidate log record is not a log record that needs to be synchronized to the second target database. At this time, the transaction parsing node can be used to obtain the log record located after the candidate log record from the disk, and repeat the above judgment process until the change data table of the candidate log record is the original data table.

[0082] S209: Synchronize the transaction data to be synchronized to the second target database using the data loading node.

[0083] S210: Determine a new breakpoint using a transaction parsing node according to a log sequence number of a log record corresponding to the transaction data to be synchronized.

[0084] Specifically, after synchronizing the transaction data to be synchronized to the second target database using the data loading node, the transaction parsing node can be used to obtain the next log record of the log record corresponding to the transaction data to be synchronized from the disk based on the log sequence number of the log record corresponding to the transaction data to be synchronized, and determine the log sequence number of the next log record as the new breakpoint. The next log record is the next log record of the log record corresponding to the transaction data to be synchronized in the disk.

[0085] S211 . Using the log reading node, update the saved target breakpoint based on the new breakpoint.

[0086] Specifically, after obtaining the new breakpoint, the transaction parsing node can be used to generate an update breakpoint message based on the new breakpoint and the identifier of the second synchronization link, and send the update breakpoint message to the log reading node; then, the log reading node can receive the update breakpoint message, and parse the identifier of the second synchronization link and the new breakpoint from the update breakpoint message, and then based on the identifier of the second synchronization link, update the target breakpoint saved in the preset storage structure to the new breakpoint, thereby updating the breakpoint corresponding to the second synchronization link.

[0087] Optionally, after the saved target breakpoint is updated based on the new breakpoint using the log reading node, S207 to S211 may be repeatedly executed based on the updated target breakpoint to synchronize the incremental data in the original data table to the second target database.

[0088] Optionally, the log reading node can be used to regularly delete preset files in the disk according to the breakpoints of all synchronization links at a preset time interval, that is, the log reading node can obtain the breakpoints of all synchronization links from the preset storage structure, and determine the minimum value therefrom, and then delete the preset files whose file names are less than the minimum value from the disk. Useless files can be cleaned up in time, thereby improving the efficient use of disk space.

[0089] Optionally, if a synchronization link is added to synchronize the data of the original database to other target databases, S201 to S211 may be repeatedly performed.

[0090] The technical solution of the embodiment of the present application can add a second synchronization link according to the synchronization request, and then use the data extraction node to obtain the current log sequence number of the log file of the original database, and send a new breakpoint message to the log reading node. Then, when the current log sequence number is greater than the current read sequence number, use the data extraction node to determine that the target breakpoint is the current log sequence number; when the current log sequence number is not greater than the current read sequence number, return to execute the use of the data extraction node to obtain the current log sequence number of the log file of the original database, until the obtained current log sequence number is greater than the current read sequence number. By comparing the current log sequence number and the current read sequence number, the target breakpoint corresponding to the second synchronization link can be accurately determined, effectively avoiding This eliminates the situation where the log reading node has already read and processed the log records corresponding to the current log sequence number when receiving the new breakpoint message, provides an accurate reference point for subsequent data synchronization, and reduces the risk of data loss, thereby improving the efficiency and accuracy of data synchronization; after sending the new breakpoint message to the log reading node, the log reading node stores the log records read from the log file to the disk, and the log records read by the log reading node can be backed up. At this time, all synchronization links can use their respective transaction parsing nodes to obtain log records from the disk, effectively avoiding data synchronization confusion and reducing the risk of data loss, thereby improving the efficiency and accuracy of data synchronization.

[0091] Afterwards, the transaction parsing node is used to obtain log records with log sequence numbers not less than the target breakpoint from the disk to obtain candidate log records, so that each synchronization link can use its own transaction parsing node to obtain log records from the disk, thereby improving the efficiency and accuracy of determining the candidate log records, thereby reducing the risk of data loss and providing an accurate data foundation for subsequent synchronization work; the transaction parsing node is used to perform transaction parsing on the candidate log records, and when the change data table of the candidate log record is the original data table, the transaction data corresponding to the candidate log record is determined as the transaction data to be synchronized, which can avoid the problem of confusion in the second target database due to erroneous synchronization of other irrelevant data, thereby improving the accuracy and efficiency of determining the transaction data to be synchronized, thereby improving the pertinence and accuracy of data synchronization.

[0092] After the transaction data to be synchronized is synchronized to the second target database using the data loading node, the new breakpoint is determined according to the log sequence number of the log record corresponding to the transaction data to be synchronized using the transaction parsing node, and the saved target breakpoint is updated based on the new breakpoint using the log reading node. The breakpoint corresponding to the synchronization link can be updated in time, so that no matter when the data synchronization process is interrupted or resumed, data synchronization can be continued from the correct position, avoiding data omission or repeated synchronization, thereby improving the accuracy of data synchronization.

[0093] Figure 3 is a structural diagram of a data synchronization device provided in an embodiment of the present application, referring to Figure 3 , the data synchronization device may include:

[0094] A link adding module 310 is used to add a second synchronization link according to the synchronization request, the second synchronization link is used to synchronize the data of the original database to the second target database, the second synchronization link and the first synchronization link share a log reading node, and the second synchronization link also includes a data extraction node, a transaction parsing node and a data loading node;

[0095] The acquisition module 320 is used to obtain the current log sequence number of the log file of the original database by using the data extraction node, and send a new breakpoint message to the log reading node, the new breakpoint message is used to notify the log reading node to feed back the current read sequence number to the data extraction node;

[0096] The breakpoint determination module 330 is used to determine the target breakpoint of the second synchronization link according to the current log sequence number and the current read sequence number using the data extraction node, and save the target breakpoint using the log reading node;

[0097] The parsing module 340 is used to obtain candidate log records based on the target breakpoint using the transaction parsing node, and perform transaction parsing on the candidate log records to obtain transaction data to be synchronized;

[0098] The synchronization module 350 is used to synchronize the transaction data to be synchronized to the second target database by using the data loading node.

[0099] In one embodiment, the breakpoint determination module 330 determines the target breakpoint of the second synchronization link based on the current log sequence number and the current read sequence number, including: when the current log sequence number is greater than the current read sequence number, determining the target breakpoint to be the current log sequence number; when the current log sequence number is not greater than the current read sequence number, returning to execute the current log sequence number of the log file of the original database obtained by the data extraction node until the obtained current log sequence number is greater than the current read sequence number.

[0100] In one embodiment, the data synchronization device further includes a cache module, which is specifically used to: after sending a new breakpoint message to the log reading node, use the log reading node to store the log records read from the log file to the disk.

[0101] In one embodiment, the cache module is specifically used to: when the file size of the first preset file on the disk does not exceed a preset threshold, use the log reading node to store the log records read from the log file to the first preset file; when the file size of the first preset file on the disk exceeds the preset threshold, use the log reading node to store the log records read from the log file to the second preset file on the disk, and determine the file name of the first preset file as the maximum log sequence number in the first preset file.

[0102] In one embodiment, the parsing module 340 uses the transaction parsing node to obtain candidate log records based on the target breakpoint, including: using the transaction parsing node to obtain log records with log sequence numbers not less than the target breakpoint from the disk to obtain candidate log records.

[0103] In one embodiment, the second synchronization link is used to synchronize the data of the original data table in the original database to the second target database, and the parsing module 340 performs transaction parsing on the candidate log records to obtain the transaction data to be synchronized, including: performing transaction parsing on the candidate log records, and when the change data table of the candidate log record is the original data table, determining the transaction data corresponding to the candidate log record as the transaction data to be synchronized.

[0104] In one embodiment, the data synchronization device also includes a breakpoint update module, which is specifically used to: after synchronizing the transaction data to be synchronized to the second target database using the data loading node, use the transaction parsing node to determine the new breakpoint according to the log sequence number of the log record corresponding to the transaction data to be synchronized; use the log reading node to update the saved target breakpoint based on the new breakpoint.

[0105] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0106] The data synchronization device provided in this embodiment can be applied to the data synchronization method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0107] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. Figure 4 A block diagram of an exemplary electronic device 11 suitable for implementing the embodiments of the present application is shown. Figure 4 The electronic device 11 shown is only an example and should not bring any limitation to the function and scope of use of the present embodiment.

[0108] like Figure 4 As shown, the electronic device 11 is in the form of a general purpose computing electronic device. The components of the electronic device 11 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0109] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0110] The electronic device 11 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 11, including volatile and non-volatile media, removable and non-removable media.

[0111] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 11 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4not shown, usually called a "hard drive"). Although Figure 4 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical medium) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0112] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0113] The electronic device 11 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 11, and / or may communicate with any device that enables the electronic device 11 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed through an input / output (I / O) interface 22. Furthermore, the electronic device 11 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 20.

[0114] like Figure 4 As shown, the network adapter 20 communicates with other modules of the electronic device 11 via the bus 18. It should be understood that although Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 11, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0115] The processing unit 16 executes various functional applications and page displays by running programs stored in the system memory 28, such as implementing a data synchronization method provided in any embodiment of the present application.

[0116] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements a data synchronization method provided by any embodiment of the present application, for example.

[0117] The computer storage medium of this embodiment can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to: electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing programs, which can be used by instruction execution systems, devices or devices or used in combination with them.

[0118] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0119] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0120] Computer program code for performing the operation of the present application can be written in one or more programming languages ​​or combinations thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0121] It should be understood by those skilled in the art that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0122] In addition, the acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0123] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the inventive concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A data synchronization method, applied to an electronic device, wherein the electronic device comprises a first synchronization link for synchronizing data of an original database to a first target database, characterized in that: The method comprises: A second synchronization link is added according to the synchronization request, the second synchronization link is used to synchronize the data of the original database to the second target database, the second synchronization link and the first synchronization link share a log reading node, and the second synchronization link also includes a data extraction node, a transaction parsing node and a data loading node; Utilize the data extraction node to obtain the current log sequence number of the log file of the original database, and send a new breakpoint message to the log reading node, wherein the new breakpoint message is used to notify the log reading node to feed back the current read sequence number to the data extraction node; Determine a target breakpoint of the second synchronization link according to the current log sequence number and the current read sequence number using the data extraction node, and save the target breakpoint using the log reading node; Using the transaction parsing node to obtain candidate log records based on the target breakpoint, and performing transaction parsing on the candidate log records to obtain transaction data to be synchronized; The transaction data to be synchronized is synchronized to the second target database using the data loading node.

2. The data synchronization method according to claim 1, characterized in that: The determining the target breakpoint of the second synchronization link according to the current log sequence number and the current read sequence number includes: When the current log sequence number is greater than the current read sequence number, determining the target breakpoint to be the current log sequence number; When the current log sequence number is not greater than the current read sequence number, return to executing the step of using the data extraction node to obtain the current log sequence number of the log file of the original database until the obtained current log sequence number is greater than the current read sequence number.

3. The data synchronization method according to claim 1, characterized in that: After sending the newly added breakpoint message to the log reading node, the method further includes: The log reading node is used to store the log records read from the log file to a disk.

4. The data synchronization method according to claim 3, characterized in that: The step of using the log reading node to store the log records read from the log file to a disk includes: When the file size of the first preset file on the disk does not exceed a preset threshold, using the log reading node to store the log records read from the log file into the first preset file; When the file size of the first preset file on the disk exceeds a preset threshold, the log reading node is used to store the log records read from the log file to the second preset file on the disk, and the file name of the first preset file is determined as the maximum log sequence number in the first preset file.

5. The data synchronization method according to claim 3, characterized in that: The obtaining the candidate log record based on the target breakpoint by using the transaction parsing node includes: The transaction parsing node is used to obtain log records whose log sequence numbers are not less than the target breakpoint from the disk to obtain the candidate log records.

6. The data synchronization method according to claim 1, characterized in that: The second synchronization link is used to synchronize the data of the original data table in the original database to the second target database, and the transaction parsing of the candidate log record to obtain the transaction data to be synchronized includes: Transaction parsing is performed on the candidate log record, and when the changed data table of the candidate log record is the original data table, transaction data corresponding to the candidate log record is determined as the transaction data to be synchronized.

7. The data synchronization method according to claim 1, characterized in that: After synchronizing the transaction data to be synchronized to the second target database using the data loading node, the method further includes: Determine a new breakpoint using the transaction parsing node according to the log sequence number of the log record corresponding to the transaction data to be synchronized; The saved target breakpoint is updated based on the new breakpoint using the log reading node.

8. A data synchronization device, applied to an electronic device, the electronic device comprising a first synchronization link for synchronizing data of an original database to a first target database, characterized in that: The device comprises: A link adding module, used for adding a second synchronization link according to a synchronization request, wherein the second synchronization link is used for synchronizing the data of the original database to the second target database, the second synchronization link and the first synchronization link share a log reading node, and the second synchronization link also includes a data extraction node, a transaction parsing node and a data loading node; An acquisition module, used to use the data extraction node to obtain the current log sequence number of the log file of the original database, and send a new breakpoint message to the log reading node, wherein the new breakpoint message is used to notify the log reading node to feed back the current read sequence number to the data extraction node; A breakpoint determination module, configured to determine a target breakpoint of the second synchronization link according to the current log sequence number and the current read sequence number using the data extraction node, and save the target breakpoint using the log reading node; A parsing module, configured to obtain candidate log records based on the target breakpoint using the transaction parsing node, and perform transaction parsing on the candidate log records to obtain transaction data to be synchronized; A synchronization module is used to synchronize the transaction data to be synchronized to the second target database using the data loading node.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data synchronization method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data synchronization method as described in any one of claims 1 to 7 is implemented.

Images