Incremental data synchronization acceleration method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to an incremental data synchronization acceleration method and device, computer equipment, a computer readable storage medium and a computer program product. The method is applied to a data transmission service, and comprises the following steps: acquiring a current incremental log record from a source database by using a first mode through a connector module, and recording a current site; analyzing the current incremental log record through the current data dictionary, acquiring a security site under the condition of analysis failure, and switching the first mode into a second mode; using a second mode to re-acquire historical and current incremental log records starting from the security site; obtaining table structure change information of the current data dictionary according to the historical increment log record; executing rollback on the current data dictionary according to the table structure change information to obtain a data dictionary after rollback; and analyzing the historical and current incremental log records through the returned data dictionary, and generating and sending a first change event. By adopting the method, the data synchronization efficiency and reliability can be improved.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to an incremental data synchronization acceleration method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Data integration open-source tools such as Flink CDC are currently mainstream data integration frameworks that support real-time change capture from various data sources, including Oracle databases. Its embedded Debezium component's Oracle connector supports three Oracle log parsing tools: Logminer, XStream API, and OpenLogReplicator. XStream API requires a license from GoldenGate. OpenLogReplicator is a third-party open-source project and is not yet fully mature.

[0003] In traditional technologies, database migration projects like Oracle often employ a long-term dual-track operation mechanism. Using GoldenGate incurs long-term license fees, leading to high costs, while using the third-party open-source OpenLogReplicator carries unpredictable risks. Therefore, the data transfer service uses only Logminer for log parsing, saving on development and license costs, and offering greater stability and reliability than OpenLogReplicator. Within Logminer's log parsing methods, Debezium provides three strategies: online_catalog, redo_log_catalog, and hybrid.

[0004] However, when using the Logminer log parsing tool, the parsing strategy configured for the log parsing tool often results in parsing errors when using the online_catalog mode, affecting the performance of data transmission synchronization; using the redo_log_catalog mode requires constantly rebuilding the data dictionary, which also affects the performance of data transmission synchronization; using the hybrid mode will persist an additional DDL version library locally on the connector, which is redundant with the storage module store design of the data transmission service and has the risk of losing metadata, which also affects the performance of data transmission synchronization. Summary of the Invention

[0005] Therefore, it is necessary to provide an incremental data synchronization acceleration method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve data transmission synchronization performance in response to the above-mentioned technical problems.

[0006] In a first aspect, this application provides an incremental data synchronization acceleration method, applied to a data transmission service, the method comprising:

[0007] The connector module uses a first mode to retrieve the current incremental log record from the source database and records the current position of the current incremental log record; wherein, the position is used to identify the order in which each change occurs in the source database; the first mode is the parsing mode using the current data dictionary;

[0008] The current incremental log record is parsed using the current data dictionary. If the parsing of the current incremental log record fails, a safe position is obtained, and the first mode is switched to the second mode. The safe position is the position that was successfully parsed in the last time corresponding to the current position. The second mode is the parsing mode using the rolled-back data dictionary.

[0009] Use the second mode to reacquire the historical incremental log records and the current incremental log records starting from the secure location;

[0010] The table structure change information of the current data dictionary is obtained from the historical incremental log records; the current data dictionary is rolled back based on the table structure change information to obtain the rolled-back data dictionary;

[0011] The historical incremental log record and the current incremental log record are parsed using the rolled-back data dictionary to generate and send a first change event.

[0012] In one embodiment, after generating and sending the first change event, the method further includes:

[0013] Obtain the next location from the current location;

[0014] Starting from the next location, the second mode will be switched back to the first mode;

[0015] The new incremental log record is obtained from the source database using the first mode, and the new incremental log record is parsed using the current data dictionary.

[0016] In one embodiment, the method further includes:

[0017] If the current incremental log record is successfully parsed, a second change event is generated;

[0018] The second change event is sent to the storage module; wherein the storage module is used to store the incremental log records of the source database.

[0019] In one embodiment, incremental logging includes data definition language logging; the current data dictionary is updated in the following ways:

[0020] Obtain the Data Definition Language (DDL) log record; wherein the position corresponding to the DDL log record is before the current position;

[0021] Parse the data definition language log records to obtain table structure change information of the data dictionary;

[0022] The data dictionary is updated based on the table structure change information to obtain the current data dictionary.

[0023] In one embodiment, before obtaining the secure location in the event of parsing failure, the method further includes:

[0024] When the metadata required for the current incremental log record is not found in the current data dictionary, at least one of the table name and field name in the current incremental log record is marked as a degradation feature and the parsing is determined to be a failure.

[0025] In one embodiment, incremental logging includes data manipulation language logging; the method further includes:

[0026] If the content recorded in the current data manipulation language is a current incremental transaction with an event count greater than a threshold, the event count of the current incremental transaction is obtained through the connector module.

[0027] Store the first target transaction in the current incremental transaction whose event count is less than or equal to the threshold in the first storage location; store the second target transaction in the current incremental transaction whose event count is greater than the threshold in the second storage location;

[0028] The first target transaction and the second target transaction are read from the first storage location and the second storage location in the order of the event count, and then sent to the storage module.

[0029] Secondly, this application also provides an incremental data synchronization acceleration device for use in data transmission services, the device comprising:

[0030] The record acquisition module is used to acquire the current incremental log record from the source database through the connector module using a first mode, and record the current position of the current incremental log record; wherein, the position is used to identify the order in which each change occurs in the source database; the first mode is the parsing mode using the current data dictionary;

[0031] The parsing switching module is used to parse the current incremental log record using the current data dictionary. If the parsing of the current incremental log record fails, a safe position is obtained, and the first mode is switched to the second mode. The safe position is the position that was successfully parsed in the last time corresponding to the current position. The second mode is the parsing mode using the rolled-back data dictionary.

[0032] The reacquisition module is used to reacquire the historical incremental log records and the current incremental log records starting from the security point using the second mode;

[0033] The rollback module is used to obtain the table structure change information of the current data dictionary from the historical incremental log records; and to perform a rollback on the current data dictionary based on the table structure change information to obtain the rolled-back data dictionary.

[0034] The parsing and sending module is used to parse the historical incremental log record and the current incremental log record through the rolled-back data dictionary, and generate and send the first change event.

[0035] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0036] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0037] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0038] The aforementioned incremental data synchronization acceleration method, apparatus, computer equipment, computer-readable storage medium, and computer program product prioritize the use of a first mode to maintain high efficiency and performance during incremental data synchronization. If parsing fails in the first mode, it automatically switches to a second mode for parsing, rebuilding the data dictionary as needed. Therefore, it eliminates the need for local persistence and storage of redundant data definition language (DML) version libraries. Since no DML version library exists, the problem of metadata loss in the DML version library is also avoided. By prioritizing the use of the efficient and high-performance first mode, the efficiency and reliability of synchronization are improved, while also avoiding the risks of redundancy and loss associated with locally persisted DML version libraries. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is an application environment diagram of an incremental data synchronization acceleration method in one embodiment;

[0041] Figure 2 This is a schematic diagram of the internal architecture of the connector in one embodiment;

[0042] Figure 3 This is a flowchart illustrating a method for improving the efficiency and reliability of synchronization in one embodiment;

[0043] Figure 4 This is a flowchart illustrating the mode switching process in one embodiment;

[0044] Figure 5 This is a schematic diagram of the automatic cache switching process in one embodiment;

[0045] Figure 6 This is a schematic diagram of the automatic cache switching process in another embodiment;

[0046] Figure 7 This is a structural block diagram of an incremental data synchronization acceleration device in one embodiment;

[0047] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] The incremental data synchronization acceleration method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, the Oracle database connector module of the Data Transaction Service (DTS) connects to the Oracle database via a log parsing tool. The connector module also connects to the DTS storage module, `store`. The log parsing tool, Logminer, extracts redo logs and archive logs from the Oracle database. The storage module stores the incremental events sent by the connector, which are then consumed by multiple downstream endpoints.

[0050] The internal architecture of the connector module is as follows Figure 2 As shown, the connector module receives and processes the parsed logs through the poll thread, then stores them in the Transactions data structure. The sink thread then submits the transactions to the Store sequentially from this data structure. Internally, the connector module also includes Java heap memory, Java off-heap memory, and local cache storage.

[0051] In one exemplary embodiment, such as Figure 3 As shown, an incremental data synchronization acceleration method is provided, which can be applied to... Figure 1 Taking the connector module as an example, the explanation includes the following steps S302 to S310. Wherein:

[0052] Step S302: Use the connector module to retrieve the current incremental log record from the source database using the first mode, and record the current position of the current incremental log record.

[0053] The position is used to identify the order in which each change occurs in the source database, and the position is a monotonically increasing sequence; the current position is used to identify the order in which the current incremental log record changes in the source database occur.

[0054] The first mode uses the current data dictionary for parsing. This mode can be `online_catalog`, also known as online data dictionary parsing mode. In this mode, the log parsing tool queries the database's currently online, up-to-date system data dictionary to parse change records in the redo logs. This mode offers high parsing performance, but when parsing delayed log records, the online data dictionary has evolved and may not reflect the historical table structure at the time the log record was created, potentially leading to metadata loss and parsing errors.

[0055] The current incremental log record can be either a Data Definition Language (DDL) record or a Data Manipulation Language (DML) record. DDL is a subset of SQL specifically used to define and manage all structure and schema objects in the database. DML is the part of SQL used to manipulate the actual data in the database tables.

[0056] Optionally, an Oracle database generates a current incremental log record, such as a DML record, triggered by a business operation; its corresponding position is 1000. The Data Transmission Service (DTS) retrieves the current incremental log record from the source database using the first mode through the connector module, and records the current position of the current incremental log record, i.e., retrieves the DML record, and records it at position 1000.

[0057] Step S304: Parse the current incremental log record using the current data dictionary. If parsing the current incremental log record fails, obtain the safe location and switch the first mode to the second mode.

[0058] The safe site is the site that was successfully resolved in the last time, for example, site 980.

[0059] The second mode is the parsing mode using the rolled-back data dictionary. Examples of this mode include `redo_log_catalog`, also known as the log-in-the-log data dictionary parsing mode. This means that the log parsing tool does not query the online database, but instead reconstructs the historical system data dictionary from the information recorded in the transaction log itself, and then parses the redo log accordingly. This mode ensures the accuracy of the parsed history and avoids metadata loss, but because it requires real-time reconstruction of the data dictionary, the parsing performance is significantly reduced, resulting in low synchronization efficiency. The working principle of the `redo_log_catalog` mode is: instead of reading the data dictionary from the current database, it reconstructs the data dictionary for a specific historical point in time from the log file itself. This is a slow but accurate process.

[0060] The reason for the current incremental log record parsing failure is as follows: For example, at position 990, the database executed a DDL statement, resulting in a change to the database table structure. However, the current incremental log record shows a new DML operation occurring at position 1000. The connector's parsing thread (poll thread) is still using the online_catalog mode to parse the current incremental log record at position 1000. This current incremental log record is recorded under the old table structure (the database table changed at position 990). The Logminer cannot correctly map back to the old table structure using the new data dictionary, leading to parsing failure.

[0061] Optionally, the connector module parses the current incremental log record using the current data dictionary, i.e., the latest data dictionary. If parsing the current incremental log record fails, it obtains a safe point, such as 980, and switches from the first mode to the second mode, changing `online_catalog` to `redo_log_catalog`. The parsing thread then re-digs the log starting from point 980. Note that this does not start from the error point 1000, but from the previous safe point, ensuring that no changes that might have been incorrectly parsed during the delay are missed.

[0062] Step S306: Use the second mode to reacquire historical incremental log records and current incremental log records starting from the safe location.

[0063] Optionally, the connector module uses redo_log_catalog to re-acquire historical incremental log records starting from safe point 980, with points 981-999; and the current incremental log record, with point 1000.

[0064] Step S308: Obtain the table structure change information of the current data dictionary from the historical incremental log records; perform a rollback on the current data dictionary based on the table structure change information to obtain the rolled-back data dictionary.

[0065] Since the redo_log_catalog mode works by reconstructing the data dictionary from a specific point in history instead of reading it from the current database, the parsing module in the connector obtains the table structure change information of the current data dictionary from the historical incremental log records. Based on the table structure change information, it performs a rollback on the current data dictionary to obtain the rolled-back data dictionary.

[0066] Step S310: Parse the historical incremental log records and the current incremental log records using the rolled-back data dictionary, generate and send the first change event.

[0067] Optionally, when the parsing module in the connector replays the previously erroneous DML operation (position 1000), the Logminer now possesses the correct, historical version of the data dictionary (the rolled-back data dictionary). Therefore, it can successfully parse the record, generate a structured first change event, and send the first change event to the downstream storage module (store) via the sending thread. At this point, incremental data synchronization is complete.

[0068] In another embodiment, such as Figure 4As shown. Before starting the DST incremental task, the task mining strategy is configured to optimistic mode. In optimistic mode, DTS uses Logminer's native strategy online_catalog to start a parsing thread poll thread to obtain and parse the current incremental log records, which can be DDL records or DML records, and continuously perform incremental synchronization.

[0069] If parsing the current incremental log record fails, and the reason for failure is "data dictionary not found," then DTS switches to Logminer's native redo_log_catalog mode. The parsing thread does not retrieve incremental log records for candidate positions but instead restarts synchronization from the last incremental position. Upon completion of parsing the current incremental log record, a first change event is generated and sent to the storage module via the sink thread. The storage module stores the first change event. It should be noted that if the reason for failure is not "data dictionary not found," the parsing thread retryes or sends an exception message.

[0070] If the current incremental log record fails to be parsed, it is converted into a structured second change event, and the second change event is sent to the storage module through the sink thread, where the storage module stores the second change event.

[0071] When DTS completes this cycle, it switches back to online_catalog mode starting from the next location.

[0072] In the aforementioned incremental data synchronization acceleration method, the first mode is prioritized during incremental data synchronization to maintain high efficiency and performance. If parsing fails in the first mode, it automatically switches to the second mode for parsing, rebuilding the data dictionary as needed. Therefore, there is no need to persist and store redundant data definition language (DDL) versions locally. Since there is no DDL version library, the problem of metadata loss from the DDL version library is also eliminated. By prioritizing the efficient and high-performance first mode, the efficiency and reliability of synchronization are improved, while also avoiding the risks of redundancy and loss from locally persisted DDL versions.

[0073] In an exemplary embodiment, after generating and sending the first change event, the mode switching process further includes: obtaining the next location of the current location; starting from the next location, switching the second mode back to the first mode; using the new incremental log record obtained from the source database using the first mode, and parsing the new incremental log record through the current data dictionary.

[0074] Optionally, in the second mode, such as redo_log_catalog mode, the connector module successfully processed all logs corresponding to position 1000 without any further errors. The connector module determined that the crisis had been resolved and the data dictionary had caught up to the latest state through replay; it automatically switched the Logminer's mining mode from the second mode (redo_log_catalog) back to the high-performance first mode (online_catalog), continuing to use the fast first mode (online_catalog) for incremental data synchronization starting from position 1001.

[0075] In this embodiment, by continuing to use the fast first mode for incremental data synchronization, the data synchronization efficiency can be improved.

[0076] In an exemplary embodiment, the method further includes: generating a second change event if the current incremental log record is successfully parsed; and sending the second change event to a storage module; wherein the storage module is used to store the incremental log records of the source database.

[0077] Optionally, if the current incremental log record at point 1000 is successfully parsed, a structured second change event is generated; the structured JSON event is successfully sent to the downstream storage module by the sink thread. The storage module stores the incremental log records and the second change event from the source database. At this point, data synchronization is complete. The synchronization point advances to 1001, everything is normal, and the connector continues to parse the current incremental log record using the first mode, maintaining high efficiency.

[0078] In this embodiment, by continuously parsing the current incremental log record through the first mode, a high incremental data synchronization efficiency can be maintained.

[0079] In an exemplary embodiment, the incremental log record includes a Data Definition Language (DDL) log record; the current data dictionary is updated by: obtaining the DDL log record; wherein the position corresponding to the DDL log record is before the current position; parsing the DDL log record to obtain the table structure change information of the data dictionary; and updating the data dictionary based on the table structure change information to obtain the current data dictionary.

[0080] Incremental logging includes Data Definition Language (DDL) logging.

[0081] Optionally, the connector retrieves the Data Definition Language (DDL) log records corresponding to sites preceding the current site using a poll thread; it parses these DDL log records to obtain table structure change information for the data dictionary, such as adding fields; the connector then updates the existing data dictionary based on this information to obtain the current data dictionary. The connector parses the change records in the redo log by querying the latest online data dictionary in the database.

[0082] In this embodiment, efficient parsing of the first mode is ensured by continuously updating the current data dictionary.

[0083] In an exemplary embodiment, before obtaining the secure location in the event of parsing failure, the method further includes: when the metadata required for the current incremental log record is not present in the current data dictionary, marking at least one of the table name and field name in the current incremental log record as a degenerate feature and determining that parsing has failed.

[0084] Among these, degradation features include adding specific identifiers to table names and / or field names to indicate that the name cannot be resolved by the current data dictionary.

[0085] Optionally, when Logminer encounters metadata mismatches while parsing in online_catalog mode, the connector will convert unresolved table names to a format including the "#OBJ" suffix, such as converting the USER_ACCOUNT table to USER_ACCOUNT#OBJ_12345, and converting field names to common formats such as COL_1 and COL_2. After marking degradation characteristics and determining that parsing has failed, the connector automatically saves the context state of the current incremental log records, including the parsed incremental log record sequence, memory usage, and cache state, preparing for subsequent mode switching and data recovery.

[0086] In this embodiment, by detecting degradation features in log records in real time, the loss of metadata can be accurately identified, providing a reliable trigger signal for the subsequent automatic recovery mechanism.

[0087] In one exemplary embodiment, the cache is automatically switched, such as Figure 5 As shown, the incremental log recording includes data manipulation language records; the method also includes steps S502 to S506. Wherein:

[0088] Step S502: If the content of the current data manipulation language record is the current incremental transaction with an event count greater than the threshold, obtain the event count of the current incremental transaction through the connector module.

[0089] In traditional methods, Debezium uses the Infinispan secondary storage location to handle large transactions (such as infinispan_embedded or infinispan_remote modes). Even when configured for asynchronous write transactions, this can result in up to a 6x performance penalty, leading to synchronization latency and the risk of lost data. Furthermore, pure memory operations may cause heap memory overflow. In other words, while Debezium natively provides infinispan_embedded and infinispan_remote modes for handling large transaction synchronization, even with asynchronous persistent transactions, there is still a performance penalty, resulting in low synchronization efficiency.

[0090] Optionally, before starting the data transfer server to execute the enhanced task, the transaction storage strategy is configured to hybrid mode. In hybrid mode, DTS directly uses Java heap memory to store transactions, and incremental synchronization continues under this setting. The event count of the current incremental transaction is obtained through the connector module. When DTS caches each incremental event, it increments the event counter in the transaction corresponding to that event by 1, calculates the event count for that event, and adds it to the data volume statistics in that transaction.

[0091] Step S504: Store the first target transaction in the current incremental transaction whose event count is less than or equal to the threshold in the first storage location; store the second target transaction in the current incremental transaction whose event count is greater than the threshold in the second storage location.

[0092] The first storage location can be Java heap memory; the second storage location can be Java off-heap memory.

[0093] Optionally, if the current Data Manipulation Language (DML) record contains data from the current incremental transaction where the event count is greater than the threshold (i.e., when the event counter or data volume statistician exceeds the configured threshold), the configuration is switched to any of the Infinispan modes, and Infinispan is configured to use off-heap memory, allowing subsequent events to be written asynchronously using a cache. The first target transaction, where the event count in the previous incremental transaction is less than or equal to the threshold, is stored in Java heap memory. The second target transaction, where the event count in the current incremental transaction is greater than the threshold, is stored in Java off-heap memory.

[0094] Step S506: Read the first target transaction and the second target transaction from the first storage location and the second storage location respectively according to the order of event count, and send them to the storage module.

[0095] Optionally, the connector module reads the first target transaction from the first storage location (Java heap memory) and the second target transaction from the second storage location (Java off-heap memory) in the order of event count, and sends them to the storage module store for downstream use.

[0096] Optionally, the content of the current data manipulation language record is obtained through the connector module. If the content is the event count or data volume of the current incremental transaction, it is determined whether the event count of the current incremental transaction is greater than the counting threshold or the data volume is greater than the data volume threshold. If neither the event count nor the data volume is greater than the counting threshold, the first target transaction with an event count less than or equal to the counting threshold or a data volume less than or equal to the data volume threshold is stored in the first storage location. If either the event count or the data volume is greater than the counting threshold, the second target transaction with an event count greater than the counting threshold or a data volume greater than the data volume threshold is stored in the second storage location.

[0097] Optionally, such as Figure 6 As shown. The transaction storage strategy is configured as hybrid, and then the incremental service of the data transfer service is started. The connector module obtains the content of the current data manipulation language record through the parsing thread. Memory mode is prioritized for storing and parsing the current data manipulation language record. The current transaction corresponding to the previous data manipulation language record stored in memory mode is the first target transaction. The event count of the current data manipulation language record is checked. If the event count is less than a threshold, memory mode is prioritized, and the first target transaction is stored in the Java heap memory. If the event count is greater than or equal to the threshold, the storage mode is switched to infiniteispan mode, and the second target transaction is stored in off-heap memory. If the current incremental transaction is a commit event, the first target transaction is read from the Java heap memory; the second target transaction is read from off-heap memory and sent to the storage store module.

[0098] It should be noted that the amount of data and the data volume threshold can also be used to determine whether the data should be stored in the first or second storage location.

[0099] In this embodiment, high performance is maintained under normal transactions, avoiding unnecessary caching overhead. Under large transactions, the system automatically switches to caching mode to prevent heap memory overflow, thus balancing efficiency and memory usage.

[0100] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0101] Based on the same inventive concept, this application also provides an incremental data synchronization acceleration device for implementing the incremental data synchronization acceleration method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the incremental data synchronization acceleration device provided below can be found in the limitations of the incremental data synchronization acceleration method described above, and will not be repeated here.

[0102] In one exemplary embodiment, such as Figure 7 As shown, an incremental data synchronization acceleration device is provided for data transmission services, including: a record acquisition module 701, a parsing switching module 702, a re-acquisition module 703, a rollback module 704, and a parsing and sending module 705, wherein:

[0103] The record acquisition module 701 is used to acquire the current incremental log record from the source database through the connector module using the first mode, and record the current position of the current incremental log record; wherein, the position is used to identify the order in which each change occurs in the source database; the first mode is the parsing mode of the current data dictionary.

[0104] The parsing switching module 702 is used to parse the current incremental log record using the current data dictionary. If the parsing of the current incremental log record fails, a safe position is obtained, and the first mode is switched to the second mode. The safe position is the position that was successfully parsed in the last time corresponding to the current position. The second mode is the parsing mode using the rolled-back data dictionary.

[0105] The reacquisition module 703 is used to reacquire historical incremental log records and current incremental log records starting from a safe location using the second mode.

[0106] The rollback module 704 is used to obtain the table structure change information of the current data dictionary from the historical incremental log records; and to perform a rollback on the current data dictionary based on the table structure change information to obtain the rolled-back data dictionary.

[0107] The parsing and sending module 705 is used to parse historical incremental log records and current incremental log records through the rolled-back data dictionary, generate and send the first change event.

[0108] In an exemplary embodiment, the parsing switching module 702 is further configured to obtain the next site of the current site; starting from the next site, switch the second mode back to the first mode; use the first mode to obtain new incremental log records from the source database, and parse the new incremental log records through the current data dictionary.

[0109] In an exemplary embodiment, the parsing and sending module 705 is further configured to generate a second change event if the current incremental log record is successfully parsed; and send the second change event to the storage module; wherein the storage module is configured to store the incremental log record of the source database.

[0110] In one exemplary embodiment, an incremental data synchronization acceleration device includes incremental log records comprising data definition language log records; it also includes an update module for acquiring data definition language log records; wherein the position corresponding to the data definition language log record is before the current position; parsing the data definition language log records to obtain table structure change information of the data dictionary; and updating the data dictionary based on the table structure change information to obtain the current data dictionary.

[0111] In an exemplary embodiment, an incremental data synchronization acceleration device further includes a judgment module, which is used to mark at least one of the table name and field name in the current incremental log record as a degradation feature and determine that the parsing has failed when the metadata required by the current incremental log record is not found in the current data dictionary.

[0112] In an exemplary embodiment, an incremental data synchronization acceleration device further includes a large transaction processing module, configured to: obtain the event count of the current incremental transaction through a connector module when the content of the current data manipulation language record is a current incremental transaction with an event count greater than a threshold; store a first target transaction in the current incremental transaction with an event count less than or equal to the threshold in a first storage location; store a second target transaction in the current incremental transaction with an event count greater than the threshold in a second storage location; and read the first target transaction and the second target transaction from the first storage location and the second storage location in the order of event counts, and send them to the storage module.

[0113] Each module in the aforementioned incremental data synchronization acceleration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0114] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores current incremental log records. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an incremental data synchronization acceleration method.

[0115] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0116] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0117] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0118] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0119] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0120] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0121] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for accelerating incremental data synchronization, characterized in that, Applied to data transmission services, the method includes: The connector module uses a first mode to retrieve the current incremental log record from the source database and records the current position of the current incremental log record; wherein, the position is used to identify the order in which each change occurs in the source database; the first mode is the parsing mode using the current data dictionary; The current incremental log record is parsed using the current data dictionary. If the parsing of the current incremental log record fails, a safe position is obtained, and the first mode is switched to the second mode. The safe position is the position that was successfully parsed in the last time corresponding to the current position. The second mode is the parsing mode using the rolled-back data dictionary. Use the second mode to reacquire the historical incremental log records and the current incremental log records starting from the secure location; The table structure change information of the current data dictionary is obtained from the historical incremental log records; the current data dictionary is rolled back based on the table structure change information to obtain the rolled-back data dictionary; The historical incremental log record and the current incremental log record are parsed using the rolled-back data dictionary to generate and send a first change event.

2. The method according to claim 1, characterized in that, Following the generation and sending of the first change event, the process also includes: Obtain the next location from the current location; Starting from the next location, the second mode will be switched back to the first mode; The new incremental log record is obtained from the source database using the first mode, and the new incremental log record is parsed using the current data dictionary.

3. The method according to claim 1, characterized in that, The method further includes: If the current incremental log record is successfully parsed, a second change event is generated; The second change event is sent to the storage module; wherein the storage module is used to store the incremental log records of the source database.

4. The method according to claim 1, characterized in that, Incremental logging includes Data Definition Language (DDL) logging; the current data dictionary is updated in the following ways: Obtain the Data Definition Language (DDL) log record; wherein the position corresponding to the DDL log record is before the current position; Parse the data definition language log records to obtain table structure change information of the data dictionary; The data dictionary is updated based on the table structure change information to obtain the current data dictionary.

5. The method according to claim 1, characterized in that, Before obtaining the safe location in the event that the current incremental log record parsing fails, the method also includes: When the metadata required for the current incremental log record is not found in the current data dictionary, at least one of the table name and field name in the current incremental log record is marked as a degradation feature, and the current incremental log record is determined to have failed to be parsed.

6. The method according to claim 1, characterized in that, Incremental logging includes data manipulation language logging; the method also includes: If the current data manipulation language record contains an event count greater than the threshold for the current incremental transaction, the event count for the current incremental transaction is obtained through the connector module. Store the first target transaction in the current incremental transaction whose event count is less than or equal to the threshold in the first storage location; store the second target transaction in the current incremental transaction whose event count is greater than the threshold in the second storage location; The first target transaction and the second target transaction are read from the first storage location and the second storage location respectively in the order of the event count, and then sent to the storage module.

7. An incremental data synchronization acceleration device, characterized in that, The apparatus is used for data transmission services and includes: The record acquisition module is used to acquire the current incremental log record from the source database through the connector module using a first mode, and record the current position of the current incremental log record; wherein, the position is used to identify the order in which each change occurs in the source database; the first mode is the parsing mode using the current data dictionary; The parsing switching module is used to parse the current incremental log record using the current data dictionary. If the parsing of the current incremental log record fails, a safe position is obtained, and the first mode is switched to the second mode. The safe position is the position that was successfully parsed in the last time corresponding to the current position. The second mode is the parsing mode using the rolled-back data dictionary. The reacquisition module is used to reacquire the historical incremental log records and the current incremental log records starting from the security point using the second mode; The rollback module is used to obtain the table structure change information of the current data dictionary from the historical incremental log records; and to perform a rollback on the current data dictionary based on the table structure change information to obtain the rolled-back data dictionary. The parsing and sending module is used to parse the historical incremental log record and the current incremental log record through the rolled-back data dictionary, and generate and send the first change event.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images