Data incremental synchronization method, apparatus and terminal device

Abstract

The application provides a data incremental synchronization method and device and a terminal device. The method comprises the following steps: receiving a plurality of incremental logs sent by a source database; extracting table data involved in DML operations from the plurality of incremental logs, and generating compressed logs based on the table data, and sending the compressed logs to each target database; wherein the compressed logs are used to instruct the target database to send a first request when the target database determines that the compressed logs match a query request received by the target database, or to instruct the target database to send a second request after the target database receives the compressed logs for a preset time; receiving the first request or the second request sent by each target database, and sending corresponding incremental logs to the corresponding target database according to the received first request or second request. The application can improve the data synchronization efficiency.

Description

Technical Field

[0001] This invention belongs to the field of data synchronization technology, and more specifically, relates to a data incremental synchronization method, apparatus and terminal equipment. Background Technology

[0002] The development of information technology has led to an increasing number of data collection, storage, and processing tasks, and network-based data sharing has become an indispensable way to acquire data resources.

[0003] To better achieve data sharing, it is necessary to synchronize shared data. For example, to better achieve data sharing between various civil affairs systems, data synchronization is required between these systems. However, the amount of data involved in civil affairs systems is enormous, and synchronizing data across civil affairs systems nationwide would lead to a surge in bandwidth pressure and reduce the efficiency of incremental data synchronization. Summary of the Invention

[0004] The purpose of this invention is to provide a data incremental synchronization method, apparatus, and terminal device to improve the efficiency of data incremental synchronization.

[0005] A first aspect of the present invention provides a data incremental synchronization method, comprising:

[0006] Receive multiple incremental logs sent by the source database;

[0007] The table data involved in the DML operation is extracted from the multiple incremental logs, and compressed logs are generated based on the table data. The compressed logs are then sent to each target database. The compressed logs are used to instruct the target database to issue a first request when it determines that the compressed logs match the query requests it receives, or to instruct the target database to issue a second request after receiving the compressed logs for a preset time.

[0008] Receive a first request or a second request from each target database, and send the corresponding incremental log to the corresponding target database according to the received first request or second request.

[0009] In one possible implementation, the compressed log carries a compressed log number; the compressed log is used to instruct the target database to extract the table data involved in the DML operation from the compressed log. If the query request received by the target database contains the table data involved in the DML operation, it is determined that the compressed log matches the received query request, and a first request containing the compressed log number is issued.

[0010] Accordingly, based on the received first request, the corresponding incremental log is sent to the corresponding target database, including:

[0011] Determine the starting point for incremental synchronization based on the compressed log number;

[0012] The incremental log corresponding to the first request is determined based on the starting point of the incremental synchronization, and the incremental log corresponding to the first request is sent to the corresponding target database.

[0013] In one possible implementation, the method for determining the compressed log number is as follows:

[0014] The first incremental log among the multiple incremental logs sent by the source database is parsed to obtain the starting point of incremental synchronization in the first incremental log.

[0015] A compressed log number is generated based on the starting point of incremental synchronization in the first incremental log entry.

[0016] In one possible implementation, the compressed log carries a compressed log number, and the second request carries the compressed log number.

[0017] Accordingly, based on the received second request, the corresponding incremental logs are sent to the target database, including:

[0018] Determine the starting point for incremental synchronization based on the compressed log number;

[0019] The incremental log corresponding to the second request is determined based on the starting point of the incremental synchronization, and the incremental log corresponding to the second request is compressed and sent to the corresponding target database.

[0020] In one possible implementation, the step of extracting table data involved in DML operations from the multiple incremental logs and generating compressed logs based on the table data includes:

[0021] Extract all table data involved in the DML operations from the multiple incremental logs;

[0022] All table data is deduplicated, and compressed logs are generated based on the deduplicated table data.

[0023] In one possible implementation, after receiving multiple incremental logs from the source database, the incremental data synchronization method further includes:

[0024] The multiple incremental logs are merged to obtain the merged incremental log.

[0025] In one possible implementation, based on the received first request or second request, the corresponding incremental log is sent to the corresponding target database, including:

[0026] Based on the first request or the second request, the incremental log is optimized and sent to the corresponding target database;

[0027] The optimization process refers to eliminating the reverse operation from the incremental synchronization start point to the first request, or eliminating the reverse operation from the incremental synchronization start point to the second request.

[0028] A second aspect of the present invention provides a data incremental synchronization device, comprising:

[0029] The data acquisition module is used to receive multiple incremental logs sent by the source database;

[0030] The first data sending module is used to extract table data involved in DML operations from the multiple incremental logs, generate compressed logs based on the table data, and send the compressed logs to each target database. The compressed logs are used to instruct the target database to issue a first request when it determines that the compressed logs match the query requests it receives, or to instruct the target database to issue a second request after receiving the compressed logs for a preset time.

[0031] The second data sending module is used to receive a first request or a second request sent by each target database, and send the corresponding incremental log to the corresponding target database according to the received first request or second request.

[0032] A third aspect of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described data incremental synchronization method.

[0033] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described data incremental synchronization method.

[0034] The beneficial effects of the data incremental synchronization method, apparatus, and terminal device provided in this invention are as follows:

[0035] Unlike existing technologies that distribute incremental logs for incremental synchronization immediately upon receiving them, this invention performs incremental synchronization in two scenarios: basic periodic synchronization and temporary synchronization when users query relevant data. In practical applications, different incremental synchronization cycles can be set for each target database to reduce bandwidth pressure and improve the efficiency of incremental data synchronization. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating a data incremental synchronization method provided in an embodiment of the present invention;

[0038] Figure 2 This is a structural block diagram of a data incremental synchronization device provided in an embodiment of the present invention;

[0039] Figure 3 This is a schematic block diagram of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0040] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0041] To make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments will be described below in conjunction with the accompanying drawings.

[0042] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a data incremental synchronization method according to an embodiment of the present invention. The method includes:

[0043] S101: Receives multiple incremental logs sent by the source database.

[0044] S102: Extract table data involved in DML (Data Manipulation Language) operations from multiple incremental logs, generate compressed logs based on the table data, and send the compressed logs to each target database. The compressed logs are used to instruct the target database to issue a first request when it determines that the compressed logs match its received query requests, or to instruct the target database to issue a second request after a preset time has elapsed since receiving the compressed logs.

[0045] S103: Receive the first or second request sent by each target database, and send the corresponding incremental log to the corresponding target database according to the received first or second request.

[0046] In this embodiment, the execution subject of steps S101 to S103 is a terminal device, which acts as an intermediate server.

[0047] In this embodiment, the preset time is determined based on the table data involved in the DML operations in the compressed log. That is, the compressed log also instructs the target database to extract the table data involved in the DML operations from the compressed log, and determines the preset time based on the usage frequency of the table data involved in the DML operations. This usage frequency can be obtained from historical data statistics.

[0048] In this embodiment, the compressed log is the data obtained by compressing the log generated based on the table data. In this embodiment, while generating the compressed log based on the table data, a corresponding number for the compressed log is also generated, i.e., the compressed log number, and this compressed log number is sent to each target database.

[0049] In one possible implementation, the method for determining the compressed log number is as follows:

[0050] The first incremental log among the multiple incremental logs sent by the source database is parsed to obtain the starting point of incremental synchronization in the first incremental log.

[0051] A compressed log number is generated based on the starting point of incremental synchronization in the first incremental log entry.

[0052] In one possible implementation of this embodiment, after receiving multiple incremental logs sent by the source database, the data incremental synchronization method further includes:

[0053] Multiple incremental logs are merged to obtain the merged incremental log.

[0054] In this embodiment, during actual implementation, multiple incremental logs can be merged and stored in a contiguous storage space. After all target databases have synchronized data based on a certain incremental log, the incremental log is then saved to another storage space and deleted from the contiguous storage space.

[0055] Correspondingly, the extraction of table data involved in DML operations from multiple incremental logs can also be completed periodically. That is, the latest incremental logs can be retrieved from the contiguous storage space periodically, the table data involved in DML operations in the latest incremental logs can be extracted, compressed logs can be generated, and the compressed logs can be sent to the databases of each target end.

[0056] As can be seen from the above, unlike the existing technology that distributes incremental logs for incremental synchronization immediately after receiving incremental logs, the present invention performs incremental synchronization in two situations: one is basic periodic synchronization, and the other is temporary synchronization when users query relevant data. In practical applications, different incremental synchronization cycles can be set on each target database to reduce bandwidth pressure and improve the efficiency of incremental data synchronization.

[0057] In one possible implementation, the compressed log carries a compressed log number. The compressed log is used to instruct the target database to extract the table data involved in the DML operation from the compressed log. If the query request received by the target database contains the table data involved in the DML operation, it determines that the compressed log matches the received query request and issues a first request containing the compressed log number.

[0058] Accordingly, based on the received first request, the corresponding incremental logs are sent to the corresponding target database, including:

[0059] The starting point for incremental synchronization is determined based on the compressed log number.

[0060] The incremental log corresponding to the first request is determined based on the starting point of the incremental synchronization, and the incremental log corresponding to the first request is sent to the corresponding target database.

[0061] In other words, the condition for the target database to issue the first request is that the user's query request for data contains table data involved in the DML operation, that is, data that has not been incrementally synchronized.

[0062] In other words, if a user queries relevant data before the target database sends a second request, the target database will immediately send the first request to synchronize the data.

[0063] In one possible implementation, the compressed log carries a compressed log number, which is also carried in the second request.

[0064] Accordingly, based on the received second request, the corresponding incremental logs are sent to the target database, including:

[0065] The starting point for incremental synchronization is determined based on the compressed log number.

[0066] The incremental log corresponding to the second request is determined based on the starting point of the incremental synchronization, and the incremental log corresponding to the second request is compressed and sent to the corresponding target database.

[0067] In this embodiment, the incremental log is compressed to reduce the amount of data transmitted and improve data synchronization efficiency. Specifically, after the target database issues a first request or a second request and obtains the corresponding incremental log, it marks the compressed log corresponding to the first or second request to indicate that the compressed log has been used and the corresponding data has been synchronized.

[0068] In one possible implementation, table data involved in DML operations is extracted from multiple incremental logs, and compressed logs are generated based on the table data, including:

[0069] Extract all table data involved in the DML operations from multiple incremental logs.

[0070] All table data is deduplicated, and compressed logs are generated based on the deduplicated table data.

[0071] When extracting all table data involved in DML operations from multiple incremental logs, which may involve multiple operations on the same table, only one backup of the table data needs to be retained, as long as it indicates that the table data has been updated.

[0072] In one possible implementation, based on the received first or second request, corresponding incremental logs are sent to the corresponding target database, including:

[0073] Based on the first or second request, the incremental log is optimized and then sent to the corresponding target database.

[0074] Optimization processing refers to eliminating the reverse operation from the incremental synchronization start point to the first request, or eliminating the reverse operation from the incremental synchronization start point to the second request.

[0075] In this embodiment, for example, if data in a certain table is added or deleted sequentially during the period from the start of incremental synchronization to the time the target database issues its first request, then all addition and deletion operations in the incremental log will be deleted to achieve optimization. Of course, to ensure data traceability, even after optimizing the incremental log, a copy of the original incremental log will be retained.

[0076] Corresponding to the data incremental synchronization method in the above embodiment, Figure 2 This is a structural block diagram of a data incremental synchronization device provided according to an embodiment of the present invention. For ease of explanation, only the parts relevant to the embodiment of the present invention are shown. References Figure 2 The data incremental synchronization device 20 includes: a data acquisition module 21, a first data transmission module 22, and a second data transmission module 23.

[0077] Among them, the data acquisition module 21 is used to receive multiple incremental logs sent by the source database.

[0078] The first data sending module 22 is used to extract table data involved in DML operations from multiple incremental logs, generate compressed logs based on the table data, and send the compressed logs to each target database. The compressed logs are used to instruct the target database to issue a first request when it determines that the compressed logs match its received query requests, or to instruct the target database to issue a second request after receiving the compressed logs for a preset time.

[0079] The second data sending module 23 is used to receive the first request or the second request sent by each target database, and send the corresponding incremental log to the corresponding target database according to the received first request or the second request.

[0080] In one possible implementation, the compressed log carries a compressed log number. The compressed log is used to instruct the target database to extract the table data involved in the DML operation from the compressed log. If the query request received by the target database contains the table data involved in the DML operation, it determines that the compressed log matches the received query request and issues a first request containing the compressed log number.

[0081] The second data transmission module 23 is specifically used for:

[0082] The starting point for incremental synchronization is determined based on the compressed log number.

[0083] The incremental log corresponding to the first request is determined based on the starting point of the incremental synchronization, and the incremental log corresponding to the first request is sent to the corresponding target database.

[0084] In one possible implementation, the first data sending module 22 is also used to determine the compressed log number:

[0085] The first incremental log among the multiple incremental logs sent by the source database is parsed to obtain the starting point of incremental synchronization in the first incremental log.

[0086] A compressed log number is generated based on the starting point of incremental synchronization in the first incremental log entry.

[0087] In one possible implementation, the compressed log carries a compressed log number, which is also carried in the second request.

[0088] The second data transmission module 23 is specifically used for:

[0089] The starting point for incremental synchronization is determined based on the compressed log number.

[0090] The incremental log corresponding to the second request is determined based on the starting point of the incremental synchronization, and the incremental log corresponding to the second request is compressed and sent to the corresponding target database.

[0091] In one possible implementation, the first data sending module 22 is specifically used for:

[0092] Extract all table data involved in the DML operations from multiple incremental logs.

[0093] All table data is deduplicated, and compressed logs are generated based on the deduplicated table data.

[0094] In one possible implementation, after receiving multiple incremental logs from the source database, the data acquisition module 21 is further used to:

[0095] Multiple incremental logs are merged to obtain the merged incremental log.

[0096] In one possible implementation, the second data sending module 23 is specifically used for:

[0097] Based on the first or second request, the incremental log is optimized and then sent to the corresponding target database.

[0098] Optimization processing refers to eliminating the reverse operation from the incremental synchronization start point to the first request, or eliminating the reverse operation from the incremental synchronization start point to the second request.

[0099] See Figure 3 , Figure 3 This is a schematic block diagram of a terminal device provided according to an embodiment of the present invention. Figure 3 The terminal 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of the modules / units in the above-described device embodiments, such as... Figure 2 The functions of modules 21 to 23 are shown.

[0100] It should be understood that, in this embodiment of the invention, the processor 301 may be a Central Processing Unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0101] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0102] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0103] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of the present invention can execute the implementation methods described in the first and second embodiments of the data incremental synchronization method provided in the embodiments of the present invention, or they can execute the implementation methods of the terminal described in the embodiments of the present invention, which will not be repeated here.

[0104] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. The computer program can also instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0105] The computer-readable storage medium can be an internal storage unit of the terminal in any of the foregoing embodiments, such as the terminal's hard disk or memory. The computer-readable storage medium can also be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the terminal. The computer-readable storage medium is used to store computer programs and other programs and data required by the terminal. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0106] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0107] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the terminals and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed terminals and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0109] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0110] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0111] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data incremental synchronization method, characterized in that, include: Receive multiple incremental logs sent by the source database; The table data involved in the DML operation is extracted from the multiple incremental logs, and compressed logs are generated based on the table data. The compressed logs are then sent to each target database. The compressed logs are used to instruct the target database to issue a first request when it determines that the compressed logs match the query requests it receives, or to instruct the target database to issue a second request after receiving the compressed logs for a preset time. Receive a first request or a second request from each target database, and send the corresponding incremental log to the corresponding target database according to the received first request or second request; The compressed log carries a compressed log number; the compressed log is used to instruct the target database to extract the table data involved in the DML operation from the compressed log; the first request carries the compressed log number; the second request carries the compressed log number; The method for determining the compressed log number is as follows: parse the first incremental log among the multiple incremental logs sent by the source database to obtain the starting point of incremental synchronization in the first incremental log; generate the compressed log number based on the starting point of incremental synchronization in the first incremental log.

2. The data incremental synchronization method as described in claim 1, characterized in that, If the query request received by the target database contains table data involved in the DML operation, it is determined that the compressed log matches the received query request, and a first request containing the compressed log number is issued. Accordingly, based on the received first request, the corresponding incremental log is sent to the corresponding target database, including: Determine the starting point for incremental synchronization based on the compressed log number; The incremental log corresponding to the first request is determined based on the starting point of the incremental synchronization, and the incremental log corresponding to the first request is sent to the corresponding target database.

3. The data incremental synchronization method as described in claim 1, characterized in that, Based on the received second request, the corresponding incremental logs are sent to the corresponding target database, including: Determine the starting point for incremental synchronization based on the compressed log number; The incremental log corresponding to the second request is determined based on the starting point of the incremental synchronization, and the incremental log corresponding to the second request is compressed and sent to the corresponding target database.

4. The data incremental synchronization method as described in claim 1, characterized in that, The step of extracting table data involved in DML operations from the multiple incremental logs and generating compressed logs based on the table data includes: Extract all table data involved in the DML operations from the multiple incremental logs; All table data is deduplicated, and compressed logs are generated based on the deduplicated table data.

5. The data incremental synchronization method as described in any one of claims 1 to 4, characterized in that, After receiving multiple incremental logs from the source database, it also includes: The multiple incremental logs are merged to obtain the merged incremental log.

6. The data incremental synchronization method as described in claim 5, characterized in that, Based on the received first request or second request, send the corresponding incremental log to the corresponding target database, including: Based on the first request or the second request, the incremental log is optimized and sent to the corresponding target database; The optimization process refers to eliminating the reverse operation from the incremental synchronization start point to the first request, or eliminating the reverse operation from the incremental synchronization start point to the second request.

7. A data incremental synchronization device, characterized in that, include: The data acquisition module is used to receive multiple incremental logs sent by the source database; The first data sending module is used to extract table data involved in DML operations from the multiple incremental logs, generate compressed logs based on the table data, and send the compressed logs to each target database. The compressed logs are used to instruct the target database to issue a first request when it determines that the compressed logs match the query requests it receives, or to instruct the target database to issue a second request after receiving the compressed logs for a preset time. The second data sending module is used to receive the first request or the second request sent by each target database, and send the corresponding incremental log to the corresponding target database according to the received first request or the second request. The compressed log carries a compressed log number; the compressed log is used to instruct the target database to extract the table data involved in the DML operation from the compressed log; the first request carries the compressed log number; the second request carries the compressed log number; The first data sending module is also used to determine the compressed log number: parse the first incremental log among the multiple incremental logs sent by the source database to obtain the starting point of incremental synchronization in the first incremental log; and generate a compressed log number based on the starting point of incremental synchronization in the first incremental log.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images