Data incremental synchronization methods, devices and terminal equipment

Abstract

This invention provides a data incremental synchronization method, apparatus, and terminal device. The method includes: receiving multiple incremental logs sent by a source database; extracting table data involved in DML operations from the multiple incremental logs, generating compressed logs based on the table data, and sending the compressed logs to various target databases; receiving a first request or a second request sent by each target database, and determining the incremental logs corresponding to each target database based on the received first request or second request; converting the incremental logs corresponding to each target database based on a preset mapping relationship between each target database and the source database, and sending the converted incremental logs to each target database. This invention can improve the efficiency of data incremental synchronization.

Description

Technical Field

[0001] This invention belongs to the field of data synchronization technology, and more specifically, relates to a data incremental synchronization method, apparatus and terminal equipment. Background Technology

[0002] The development of information technology has led to an increasing number of data collection, storage, and processing tasks, and network-based data sharing has become an indispensable way to acquire data resources.

[0003] To better achieve data sharing, data synchronization is necessary. For example, to facilitate data sharing among various civil affairs systems, data synchronization between these systems is required. However, the civil affairs system involves a massive amount of data, and the data table structures of each system differ. Synchronizing data across civil affairs systems nationwide would lead to a significant increase in bandwidth pressure and reduce the efficiency of incremental data synchronization. Summary of the Invention

[0004] The purpose of this invention is to provide a data incremental synchronization method, apparatus, and terminal device to improve the efficiency of data incremental synchronization.

[0005] A first aspect of the present invention provides a data incremental synchronization method, comprising:

[0006] Receive multiple incremental logs sent by the source database;

[0007] The table data involved in the DML operation is extracted from the multiple incremental logs, and compressed logs are generated based on the table data. The compressed logs are then sent to each target database. The compressed logs are used to instruct the target database to issue a first request when it determines that the compressed logs match the query requests it receives, or to instruct the target database to issue a second request after receiving the compressed logs for a preset time.

[0008] Receive a first request or a second request sent by each target database, and determine the incremental log corresponding to each target database based on the received first request or second request;

[0009] Based on the preset mapping relationship between each target database and the source database, the incremental logs corresponding to each target database are transformed, and the transformed incremental logs are sent to each target database.

[0010] In one possible implementation, the transformation of incremental logs corresponding to each target database based on a preset mapping relationship between each target database and the source database includes:

[0011] Based on the pre-stored log parsing file of the source database, the incremental logs of each target database are parsed to obtain the table data and operation types of each target database.

[0012] Based on the pre-stored log files corresponding to the target databases, the table data corresponding to each target database, and the operation types of the table data, the transformed incremental logs for each target database are generated.

[0013] In one possible implementation, the compressed log carries a compressed log number; the compressed log is used to instruct the target database to extract the table data involved in the DML operation from the compressed log. If the query request received by the target database contains the table data involved in the DML operation, it is determined that the compressed log matches the received query request, and a first request containing the compressed log number is issued.

[0014] Accordingly, the method for sending the transformed incremental logs to the corresponding target database upon receiving the first request is as follows:

[0015] Determine the starting point for incremental synchronization based on the compressed log number;

[0016] Determine the incremental log corresponding to the first request based on the starting point of incremental synchronization;

[0017] Based on the preset mapping relationship between the target database and the source database, the incremental log corresponding to the first request is transformed, and the transformed incremental log is sent to the corresponding target database.

[0018] In one possible implementation, the method for determining the compressed log number is as follows:

[0019] The first incremental log among the multiple incremental logs sent by the source database is parsed to obtain the starting point of incremental synchronization in the first incremental log.

[0020] A compressed log number is generated based on the starting point of incremental synchronization in the first incremental log entry.

[0021] In one possible implementation, the compressed log carries a compressed log number, and the second request carries the compressed log number.

[0022] The method for sending the transformed incremental log to the corresponding target database upon receiving the second request is as follows:

[0023] Determine the starting point for incremental synchronization based on the compressed log number;

[0024] Determine the incremental log corresponding to the second request based on the starting point of incremental synchronization;

[0025] Based on the preset mapping relationship between the target database and the source database, the incremental log corresponding to the second request is transformed, the transformed incremental log is compressed, and the compressed incremental log is sent to the corresponding target database.

[0026] In one possible implementation, before sending the transformed incremental logs to each target database, the incremental data synchronization method further includes:

[0027] The transformed incremental logs are optimized, and the optimized incremental logs are sent to the corresponding target database.

[0028] The optimization process refers to eliminating the reverse operation from the incremental synchronization start point to the first request, or eliminating the reverse operation from the incremental synchronization start point to the second request.

[0029] A second aspect of the present invention provides a data incremental synchronization device, comprising:

[0030] The data acquisition module is used to receive multiple incremental logs sent by the source database;

[0031] The first data sending module is used to extract table data involved in DML operations from the multiple incremental logs, generate compressed logs based on the table data, and send the compressed logs to each target database. The compressed logs are used to instruct the target database to issue a first request when it determines that the compressed logs match the query requests it receives, or to instruct the target database to issue a second request after receiving the compressed logs for a preset time.

[0032] The data selection module is used to receive a first request or a second request sent by each target database, and determine the incremental log corresponding to each target database based on the received first request or second request.

[0033] The second data sending module is used to transform the incremental logs corresponding to each target database based on the preset mapping relationship between each target database and the source database, and then send the transformed incremental logs to each target database.

[0034] In one possible implementation, the data incremental synchronization device further includes:

[0035] Data optimization module;

[0036] The data optimization module is used to optimize the converted incremental logs before sending them to each target database, and then send the optimized incremental logs to the corresponding target databases.

[0037] The optimization process refers to eliminating the reverse operation from the incremental synchronization start point to the first request, or eliminating the reverse operation from the incremental synchronization start point to the second request.

[0038] A third aspect of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described data incremental synchronization method.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described data incremental synchronization method.

[0040] The beneficial effects of the data incremental synchronization method, apparatus, and terminal device provided in this invention are as follows:

[0041] Unlike existing technologies that distribute incremental logs for incremental synchronization immediately upon receipt, this invention performs incremental synchronization in two scenarios: basic periodic synchronization and temporary synchronization when users query relevant data. In practical applications, different incremental synchronization cycles can be set for each target database to reduce bandwidth pressure and improve data synchronization efficiency. Furthermore, unlike existing technologies that map and transform data tables, this invention directly transforms the incremental logs before sending them, converting them into incremental logs that the target database can directly use. This solves the problem of heterogeneous data tables between different systems and further improves data synchronization efficiency. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating a data incremental synchronization method provided in an embodiment of the present invention;

[0044] Figure 2 This is a structural block diagram of a data incremental synchronization device provided in an embodiment of the present invention;

[0045] Figure 3 This is a schematic block diagram of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0046] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0047] To make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments will be described below in conjunction with the accompanying drawings.

[0048] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a data incremental synchronization method according to an embodiment of the present invention. The method includes:

[0049] S101: Receives multiple incremental logs sent by the source database.

[0050] S102: Extract table data involved in DML (Data Manipulation Language) operations from multiple incremental logs, generate compressed logs based on the table data, and send the compressed logs to each target database. The compressed logs are used to instruct the target database to issue a first request when it determines that the compressed logs match its received query requests, or to instruct the target database to issue a second request after a preset time has elapsed since receiving the compressed logs.

[0051] S103: Receive the first or second request sent by each target database, and determine the incremental log corresponding to each target database based on the received first or second request.

[0052] S104: Based on the preset mapping relationship between each target database and the source database, the incremental logs corresponding to each target database are transformed, and the transformed incremental logs are sent to each target database.

[0053] In this embodiment, the execution subject of steps S101 to S104 is a terminal device, which acts as an intermediate server.

[0054] In this embodiment, the preset time is determined based on the table data involved in the DML operations in the compressed log. That is, the compressed log also instructs the target database to extract the table data involved in the DML operations from the compressed log, and determines the preset time based on the usage frequency of the table data involved in the DML operations. This usage frequency can be obtained from historical data statistics.

[0055] In this embodiment, the compressed log is the data obtained by compressing the log generated based on the table data. In this embodiment, while generating the compressed log based on the table data, a corresponding number for the compressed log is also generated, i.e., the compressed log number, and this compressed log number is sent to each target database.

[0056] In one possible implementation, the method for determining the compressed log number is as follows:

[0057] The first incremental log among the multiple incremental logs sent by the source database is parsed to obtain the starting point of incremental synchronization in the first incremental log.

[0058] A compressed log number is generated based on the starting point of incremental synchronization in the first incremental log entry.

[0059] In one possible implementation of this embodiment, after receiving multiple incremental logs sent by the source database, the data incremental synchronization method further includes:

[0060] Multiple incremental logs are merged to obtain the merged incremental log.

[0061] In this embodiment, during actual implementation, multiple incremental logs can be merged and stored in a contiguous storage space. After all target databases have synchronized data based on a certain incremental log, the incremental log is then saved to another storage space and deleted from the contiguous storage space.

[0062] Correspondingly, the extraction of table data involved in DML operations from multiple incremental logs can also be completed periodically. That is, the latest incremental logs can be retrieved from the contiguous storage space periodically, the table data involved in DML operations in the latest incremental logs can be extracted, compressed logs can be generated, and the compressed logs can be sent to the databases of each target end.

[0063] In this embodiment, a pre-stored mapping relationship between the data tables of each target database and the data tables of the source database can be stored in the execution entity. Before sending incremental logs to each target database, the incremental logs are directly transformed based on the pre-stored mapping relationship, and the transformed incremental logs are sent to each target database. Each target database can then directly perform incremental data synchronization based on the transformed incremental logs. This embodiment of the invention adopts a centralized management approach for the pre-stored mapping relationship. In a master-slave database application scenario, when the data table structure of the master database changes, there is no need to synchronize the mapping relationships in each slave database, which can effectively ensure the mapping accuracy of each data table and improve the efficiency and accuracy of data synchronization.

[0064] As can be seen from the above, unlike existing technologies that distribute incremental logs for incremental synchronization immediately upon receiving them, this invention performs incremental synchronization in two scenarios: basic periodic synchronization and temporary synchronization when users query relevant data. In practical applications, different incremental synchronization cycles can be set for each target database to reduce bandwidth pressure and improve the efficiency of incremental data synchronization. Furthermore, unlike existing technologies that map and transform data tables, this invention directly transforms the incremental logs before sending them, converting them into incremental logs that the target database can directly use. This solves the problem of heterogeneous data tables between different systems and further improves the efficiency of data synchronization.

[0065] In one possible implementation, the incremental logs corresponding to each target database are transformed based on a pre-defined mapping relationship between each target database and the source database, including:

[0066] Based on the pre-stored log parsing file of the source database, the incremental logs of each target database are parsed to obtain the table data and operation types of each target database.

[0067] Based on the pre-stored log files corresponding to the target databases, the table data corresponding to each target database, and the operation types of the table data, the transformed incremental logs for each target database are generated.

[0068] In this embodiment, log parsing files of the source database and log generation files of each target database can be pre-stored to complete the conversion of incremental logs by combining preset mapping relationships.

[0069] In one possible implementation, the compressed log carries a compressed log number. The compressed log is used to instruct the target database to extract the table data involved in the DML operation from the compressed log. If the query request received by the target database contains the table data involved in the DML operation, it determines that the compressed log matches the received query request and issues a first request containing the compressed log number.

[0070] Accordingly, the method for sending the transformed incremental logs to the corresponding target database upon receiving the first request is as follows:

[0071] The starting point for incremental synchronization is determined based on the compressed log number.

[0072] The incremental log corresponding to the first request is determined based on the starting point of the incremental synchronization.

[0073] Based on the preset mapping relationship between the target database and the source database, the incremental log corresponding to the first request is transformed, and the transformed incremental log is sent to the corresponding target database.

[0074] In other words, the condition for the target database to issue the first request is that the user's query request for data contains table data involved in the DML operation, that is, data that has not been incrementally synchronized.

[0075] In other words, if a user queries relevant data before the target database sends a second request, the target database will immediately send the first request to synchronize the data.

[0076] In one possible implementation, the compressed log carries a compressed log number, which is also carried in the second request.

[0077] The method for sending the transformed incremental log to the corresponding target database upon receiving the second request is as follows:

[0078] The starting point for incremental synchronization is determined based on the compressed log number.

[0079] The incremental log corresponding to the second request is determined based on the starting point of the incremental synchronization.

[0080] Based on the preset mapping relationship between the target database and the source database, the incremental log corresponding to the second request is transformed, the transformed incremental log is compressed, and the compressed incremental log is sent to the corresponding target database.

[0081] In this embodiment, the converted incremental logs are compressed to reduce the amount of data transmitted and improve data synchronization efficiency. Specifically, after the target database issues a first request or a second request and obtains the corresponding incremental logs, it marks the compressed logs corresponding to the first or second request to indicate that the compressed logs have been used and the corresponding data has been synchronized.

[0082] In one possible implementation, the incremental data synchronization method further includes the following steps before sending the transformed incremental logs to the respective target databases:

[0083] The transformed incremental logs are optimized and then sent to the corresponding target database.

[0084] Optimization processing refers to eliminating the reverse operation from the incremental synchronization start point to the first request, or eliminating the reverse operation from the incremental synchronization start point to the second request.

[0085] In this embodiment, for example, if data in a certain table is added or deleted sequentially during the period from the start of incremental synchronization to the time the target database issues its first request, then all the addition and deletion operations in the transformed incremental log will be deleted to achieve optimization. Of course, to ensure data traceability, even after optimizing the transformed incremental log, a copy of the original incremental log will be retained.

[0086] In one possible implementation, table data involved in DML operations is extracted from multiple incremental logs, and compressed logs are generated based on the table data, including:

[0087] Extract all table data involved in the DML operations from multiple incremental logs.

[0088] All table data is deduplicated, and compressed logs are generated based on the deduplicated table data.

[0089] When extracting all table data involved in DML operations from multiple incremental logs, which may involve multiple operations on the same table, only one backup of the table data needs to be retained, as long as it indicates that the table data has been updated.

[0090] Corresponding to the data incremental synchronization method in the above embodiment, Figure 2 This is a structural block diagram of a data incremental synchronization device provided according to an embodiment of the present invention. For ease of explanation, only the parts relevant to the embodiment of the present invention are shown. References Figure 2 The data incremental synchronization device 20 includes:

[0091] Among them, the data acquisition module 21 is used to receive multiple incremental logs sent by the source database.

[0092] The first data sending module 22 is used to extract table data involved in DML operations from multiple incremental logs, generate compressed logs based on the table data, and send the compressed logs to each target database. The compressed logs are used to instruct the target database to issue a first request when it determines that the compressed logs match its received query requests, or to instruct the target database to issue a second request after receiving the compressed logs for a preset time.

[0093] The data selection module 23 is used to receive the first request or the second request sent by each target database, and determine the incremental log corresponding to each target database based on the received first request or the second request.

[0094] The second data sending module 24 is used to convert the incremental logs corresponding to each target database based on the preset mapping relationship between each target database and the source database, and send the converted incremental logs to each target database.

[0095] In one possible implementation, the data incremental synchronization device 20 further includes:

[0096] Data optimization module 25.

[0097] The data optimization module 25 is used to optimize the transformed incremental logs before sending them to each target database, and then send the optimized incremental logs to the corresponding target databases.

[0098] Optimization processing refers to eliminating the reverse operation from the incremental synchronization start point to the first request, or eliminating the reverse operation from the incremental synchronization start point to the second request.

[0099] In one possible implementation, the second data sending module 24 is specifically used for:

[0100] Based on the pre-stored log parsing file of the source database, the incremental logs of each target database are parsed to obtain the table data and operation types of each target database.

[0101] Based on the pre-stored log files corresponding to the target databases, the table data corresponding to each target database, and the operation types of the table data, the transformed incremental logs for each target database are generated.

[0102] In one possible implementation, the compressed log carries a compressed log number. The compressed log is used to instruct the target database to extract the table data involved in the DML operation from the compressed log. If the query request received by the target database contains the table data involved in the DML operation, it determines that the compressed log matches the received query request and issues a first request containing the compressed log number.

[0103] Accordingly, the data selection module 23 is specifically used for:

[0104] The starting point for incremental synchronization is determined based on the compressed log number.

[0105] The incremental log corresponding to the first request is determined based on the starting point of the incremental synchronization.

[0106] Accordingly, the second data sending module 24 is specifically used for:

[0107] Based on the preset mapping relationship between the target database and the source database, the incremental log corresponding to the first request is transformed, and the transformed incremental log is sent to the corresponding target database.

[0108] In one possible implementation, the first data sending module 22 is also used to determine the compressed log number:

[0109] The first incremental log among the multiple incremental logs sent by the source database is parsed to obtain the starting point of incremental synchronization in the first incremental log.

[0110] A compressed log number is generated based on the starting point of incremental synchronization in the first incremental log entry.

[0111] In one possible implementation, the compressed log carries a compressed log number, which is also carried in the second request.

[0112] Data selection module 23 is specifically used for:

[0113] The starting point for incremental synchronization is determined based on the compressed log number.

[0114] The incremental log corresponding to the second request is determined based on the starting point of the incremental synchronization.

[0115] Accordingly, the second data sending module 24 is specifically used for:

[0116] Based on the preset mapping relationship between the target database and the source database, the incremental log corresponding to the second request is transformed, the transformed incremental log is compressed, and the compressed incremental log is sent to the corresponding target database.

[0117] See Figure 3 , Figure 3 This is a schematic block diagram of a terminal device provided according to an embodiment of the present invention. Figure 3 The terminal 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of the modules / units in the above-described device embodiments, such as... Figure 2 The functions of modules 21 to 25 are shown.

[0118] It should be understood that, in this embodiment of the invention, the processor 301 may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0119] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0120] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0121] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of the present invention can execute the implementation methods described in the first and second embodiments of the data incremental synchronization method provided in the embodiments of the present invention, or they can execute the implementation methods of the terminal described in the embodiments of the present invention, which will not be repeated here.

[0122] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. The computer program can also instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0123] The computer-readable storage medium can be an internal storage unit of the terminal in any of the foregoing embodiments, such as the terminal's hard disk or memory. The computer-readable storage medium can also be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the terminal. The computer-readable storage medium is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0124] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0125] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the terminals and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed terminals and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0127] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0128] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0129] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method of data incremental synchronization, the method comprising: The method comprises the following steps: receiving a plurality of incremental logs sent by a source database; extracting table data involved in DML operations from the plurality of incremental logs, and generating compressed logs based on the table data, and sending the compressed logs to each target database; wherein the compressed logs are used to instruct the target database to issue a first request when it is determined that the compressed logs match the query request received by the target database, or are used to instruct the target database to issue a second request after a preset time of receiving the compressed logs; receiving the first request or the second request sent by each target database, and determining the incremental log corresponding to each target database according to the received first request or second request; based on the preset mapping relationship between each target database and the source database, converting the incremental log corresponding to each target database, and sending the converted incremental log to each target database; the method of converting the incremental log corresponding to each target database based on the preset mapping relationship between each target database and the source database comprises: based on the pre-stored log analysis file of the source database, analyzing the incremental log corresponding to each target database to obtain the table data corresponding to each target database and the operation type of the table data; based on the pre-stored log generation file corresponding to each target database, the table data corresponding to each target database and the operation type of the table data, generating the converted incremental log corresponding to each target database; the condition for the target database to issue the first request is that the query request of the user contains the table data involved in the DML operation, that is, the data that has not been incrementally synchronized; before the target database issues the second request, if the user queries the related data, the target database immediately sends the first request for data synchronization.

2. The data delta synchronization method of claim 1, wherein, The compressed logs carry a compressed log number; the compressed logs are used to instruct the target database to extract the table data involved in the DML operation from the compressed logs, and if the query request received by the target database contains the table data involved in the DML operation, it is determined that the compressed logs match the query request received by the target database, and the first request containing the compressed log number is issued; correspondingly, the method of sending the converted incremental log to the corresponding target database when the first request is received comprises: determining the starting point of incremental synchronization according to the compressed log number; determining the incremental log corresponding to the first request according to the starting point of incremental synchronization; based on the preset mapping relationship between the corresponding target database and the source database, converting the incremental log corresponding to the first request, and sending the converted incremental log to the corresponding target database.

3. The data delta synchronization method of claim 2, wherein, The method for determining the compressed log number comprises: analyzing the first incremental log in the plurality of incremental logs sent by the source database to obtain the starting point of incremental synchronization in the first incremental log; generating the compressed log number according to the starting point of incremental synchronization in the first incremental log.

4. The data delta synchronization method of claim 1, wherein, The compressed logs carry a compressed log number, and the second request carries the compressed log number; the method of sending the converted incremental log to the corresponding target database when the second request is received comprises: Determine the starting point of the incremental synchronization according to the compressed log number; Determine the incremental log corresponding to the second request according to the starting point of the incremental synchronization; Convert the incremental log corresponding to the second request based on the preset mapping relationship between the corresponding target end database and the source end database, compress the converted incremental log, and send the compressed incremental log to the corresponding target end database.

5. The data delta synchronization method of claim 1, wherein, Before sending the converted incremental log to each target end database, further comprising: Optimizing the converted incremental log and sending the corresponding optimized incremental log to the corresponding target end database; The optimization processing refers to eliminating the reverse operation during the incremental synchronization starting point to the first request, or eliminating the reverse operation during the incremental synchronization starting point to the second request.

6. A data delta synchronization apparatus, characterized by comprising: Comprise: The data acquisition module is used for receiving the multiple incremental logs sent by the source end database; The first data sending module is used for extracting the table data involved in the DML operation from the multiple incremental logs, generating a compressed log based on the table data, and sending the compressed log to each target end database; wherein the compressed log is used to instruct the target end database to issue a first request when determining that the compressed log matches the query request received by the target end database, or to instruct the target end database to issue a second request after receiving the compressed log for a preset time; The condition for the target end database to issue the first request is that the user's query request for data contains the table data involved in the DML operation, that is, it involves data that has not been incrementally synchronized; Before the target end database does not issue the second request, if a user queries related data, the target end database immediately sends a first request for data synchronization; The data selection module is used for receiving the first request or the second request sent by each target end database, and determining the incremental log corresponding to each target end database according to the received first request or second request; The second data sending module is used for converting the incremental log corresponding to each target end database based on the preset mapping relationship between each target end database and the source end database, and sending the converted incremental log to each target end database; The conversion of the incremental log corresponding to each target end database based on the preset mapping relationship between each target end database and the source end database comprises: Based on the pre-stored log analysis file of the source end database, the incremental log corresponding to each target end database is parsed to obtain the table data corresponding to each target end database and the operation type of the table data; Based on the pre-stored log generation file corresponding to the target end database, the table data corresponding to each target end database and the operation type of the table data, the converted incremental log corresponding to each target end database is generated.

7. The data delta synchronization apparatus of claim 6, wherein, Further comprising: Data optimization module; The data optimization module is used for optimizing the converted incremental log before sending the converted incremental log to each target end database, and sending the corresponding optimized incremental log to the corresponding target end database; The optimization processing refers to eliminating the reverse operation from the incremental synchronization starting point to the first request period, or eliminating the reverse operation from the incremental synchronization starting point to the second request period.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program, when executed by the processor, implements the steps of the method according to any one of claims 1 to 5.

Images