Database log synchronization system, method, storage medium and electronic device

Abstract

The present application relates to a database log synchronization system, method, storage medium and electronic device. The system includes: a source database, which is used to execute SQL requests received from a business end and generate log data; the source database includes a first replication component, which is used to determine whether the log data needs to be replicated according to the replication rules of the target database, and if it is determined that the log data needs to be replicated, the log data is sent to the second replication component of the target database, and the replication rules include the tables and libraries that need to be replicated; the target database is used to parse and replay the log data when the log data is received through the second replication component to perform data replication, and record the synchronization log point according to the transaction identifier in the log data, and the synchronization log point is used for continuity verification. The present application solves the technical problem that traditional synchronization tools are difficult to meet the requirements of efficient, real-time and secure data filtering.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a database log synchronization system, method, storage medium and electronic device. Background Art

[0002] With the rapid development of financial technology, the financial industry has an increasing demand for database synchronization, especially in key scenarios such as disaster recovery databases, emergency databases or static databases, which require real-time, consistency and security of data. In order to ensure high availability and disaster recovery capabilities, many financial institutions synchronize production database clusters to another set of clusters. However, with the diversification of business needs, traditional data synchronization tools face many challenges. First, financial institutions often need to filter data synchronization. For example, a securities company hopes to synchronize only key data in the production cluster and filter out irrelevant databases and tables to avoid bandwidth waste and synchronization delay. Some core banking systems also need to synchronize static data at specific times, but some tables are continuously written, affecting the synchronization status and causing resource waste. Although traditional synchronization tools support library and table filtering, data security and consistency are difficult to guarantee due to the multi-step processing of external tools. Secondly, ordinary synchronization tools usually use log collection, parsing and intermediate data format generation, which cannot achieve effective filtering of source data, resulting in resource waste and performance bottlenecks. These tools cannot meet the high requirements of the financial industry for data synchronization, especially when it comes to data consistency, real-time and performance, and cannot provide sufficient guarantees. Summary of the invention

[0003] The present application provides a database log synchronization system, method, storage medium and electronic device to solve the technical problem that traditional synchronization tools are difficult to meet the requirements of efficient, real-time and secure data filtering.

[0004] In a first aspect, the present application provides a database log synchronization system, comprising: a source database, for executing an SQL request received from a business end and generating log data; the source database comprises a first replication component, the first replication component is used to determine whether the log data needs to be replicated according to a replication rule of a target database, and when it is determined that the log data needs to be replicated, the log data is sent to a second replication component of the target database, wherein one target database corresponds to one replication rule, and the replication rule includes tables and libraries that need to be replicated; the target database is used to parse and replay the log data when the log data is received through the second replication component to perform data replication, and record a synchronization log point according to a transaction identifier in the log data, wherein the synchronization log point is used for continuity verification.

[0005] As an optional example, the first replication component is specifically used to: determine the library or table corresponding to the SQL request; if the library or table corresponding to the SQL request exists in the replication rule, determine that the log data needs to be replicated; if the library or table corresponding to the SQL request does not exist in the replication rule, determine that the log data does not need to be replicated.

[0006] As an optional example, the first replication component is also used to: when it is determined that the log data does not need to be replicated, filter the log data so that the filtered log data only contains the transaction identifier of the SQL request; send the filtered log data to the second replication component so that the target database receives the filtered log data through the second replication component, and record the synchronization log point according to the transaction identifier in the filtered log data.

[0007] As an optional example, the first replication component is also used to: when the log data of multiple consecutive SQL requests are determined not to need to be replicated, merge the filtered log data of the multiple SQL requests to obtain merged log data; send the merged log data to the second replication component, so that the target database receives the merged log data through the second replication component, and records the synchronization log point according to the last transaction identifier in the merged log data.

[0008] As an optional example, the second replication component is also used to continue to obtain log data from the first replication component according to the synchronization log point after the first replication component or the second replication component fails and recovers.

[0009] As an optional example, the first replication component is further used to send the log data to a standby database of the source database, so that the standby database replicates the data by analyzing and replaying the log data.

[0010] In a second aspect, the present application provides a method for synchronizing a database log, which is applied to a synchronization system for a database log, including: executing an SQL request received from a business end through a source database and generating log data; judging whether the log data needs to be replicated according to a replication rule of a target database through a first replication component of the source database, wherein one target database corresponds to one replication rule, and the replication rule includes tables and libraries that need to be replicated; when it is determined that the log data needs to be replicated, sending the log data to a second replication component of the target database, so that the target database receives the log data through the second replication component, performing data replication by analyzing and replaying the log data, and recording a synchronization log point according to a transaction identifier in the log data, wherein the synchronization log point is used for continuity verification.

[0011] In a third aspect, the present application provides a database log synchronization device, which is applied to a database log synchronization system, including: a generation module, which is used to execute an SQL request received from a business end through a source database and generate log data; a judgment module, which is used to judge whether the above-mentioned log data needs to be replicated according to the replication rule of the target database through the first replication component of the above-mentioned source database, wherein one target database corresponds to one replication rule, and the above-mentioned replication rule includes tables and libraries that need to be replicated; a synchronization module, which is used to send the above-mentioned log data to the second replication component of the above-mentioned target database when it is determined that the above-mentioned log data needs to be replicated, so that the above-mentioned target database receives the above-mentioned log data through the above-mentioned second replication component, replicates the data by analyzing and replaying the above-mentioned log data, and records the synchronization log point according to the transaction identifier in the above-mentioned log data, wherein the above-mentioned synchronization log point is used for continuity verification.

[0012] In a fourth aspect, the present application provides a storage medium storing a computer program, wherein the computer program executes the above-mentioned database log synchronization method when executed by a processor.

[0013] In a fifth aspect, the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-mentioned database log synchronization method through the above-mentioned computer program.

[0014] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0015] The present application adopts a source database for executing SQL requests received from a business end and generating log data; the source database includes a first replication component, the first replication component is used to determine whether the log data needs to be replicated according to the replication rule of the target database, and when it is determined that the log data needs to be replicated, the log data is sent to the second replication component of the target database, wherein one target database corresponds to one replication rule, and the replication rule includes tables and libraries that need to be replicated; the target database is used to parse and replay the log data when the log data is received through the second replication component to perform data replication, and record synchronization log points according to the transaction identifier in the log data, wherein the synchronization log points are used for a continuity verification system, because in the system, efficient data synchronization is achieved through the replication components between the source database and the target database, the source database receives the SQL request of the business end and generates log data, the first replication component of the source database determines which log data needs to be synchronized according to the replication rule of the target database, and the qualified log data is sent to the second replication component of the target database, which is responsible for parsing and replaying the log data to the target database to ensure data synchronization. At the same time, the target database records synchronization log points to ensure data consistency and continuity, thereby achieving efficient, real-time and secure custom data replication while reducing bandwidth and storage costs, thereby solving the technical problem that traditional synchronization tools are difficult to meet efficient, real-time and secure data filtering requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0019] Figure 1 is a schematic diagram of the structure of an optional database log synchronization system according to an embodiment of the present application;

[0020] Figure 2is a flowchart of an optional database log synchronization method according to an embodiment of the present application;

[0021] Figure 3 is a schematic structural diagram of an optional database log synchronization device according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0024] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0025] According to a first aspect of an embodiment of the present application, a database log synchronization system is provided. Optionally, as Figure 1 As shown, the above system includes:

[0026] The source database 102 is used to execute the SQL request received from the business end and generate log data;

[0027] The source database 102 includes a first replication component 1022, which is used to determine whether the log data needs to be replicated according to the replication rule of the target database 104, and send the log data to the second replication component 1042 of the target database 104 if it is determined that the log data needs to be replicated, wherein one target database corresponds to one replication rule, and the replication rule includes tables and libraries that need to be replicated;

[0028] The target database 104 is used to parse and replay the log data when receiving the log data through the second replication component 1042 to perform data replication, and record the synchronization log point according to the transaction identifier in the log data, wherein the synchronization log point is used for continuity verification.

[0029] Optionally, in this embodiment, a data replication process between a source database and a target database is involved, and the specific operation process includes the generation, transmission, parsing and playback of log data. Its key components include a first replication component of the source database, a second replication component of the target database, and a mechanism for defining data replication rules. The target database is a complete database, which is set for different purposes according to the usage scenario, and the typical scenario is a disaster recovery database and a static database. The main goal of the system is to efficiently and in real time synchronize the data in the source database to the target database, while ensuring the consistency of the data and the traceability of the synchronization status. The source database performs data operations such as inserting, updating or deleting according to the SQL request received by the business end, and these SQL operations will be converted into database logs to record the changes within the database. Whenever the data changes in the source database, the source database will generate corresponding log data, which contains detailed information and transaction identifiers of the data change. The first replication component of the source database is responsible for determining which log data needs to be replicated according to the replication rules defined by the target database. The replication rules can specify which tables and which libraries need to be synchronized to the target database. When processing log data, the first replication component will filter out the log data that needs to be replicated according to the rules, and ensure that unnecessary data (such as data in irrelevant tables) will not be synchronized, thereby improving performance and reducing bandwidth waste. After the target database receives the log data sent by the source database, the second replication component is responsible for parsing these logs and playing them back to the target database. The playback process is to accurately synchronize the data changes in the source database to the target database. The transaction identifiers in the log data are used to record synchronization log points in the target database. These points help to determine the continuity of data synchronization and ensure the consistency of the target database with the source database. Even if there are network problems or interruptions, the synchronization process can be resumed from the interruption point.

[0030] Optionally, in this embodiment, by defining precise replication rules, only critical data is replicated, synchronization of irrelevant data is avoided, synchronization efficiency is improved, and bandwidth and storage costs are reduced. The replication component of the source database filters log data according to the replication rules of the target database, reducing the processing and transmission of irrelevant data and optimizing resource utilization. The playback of log data and the recording of synchronization log points ensure the data consistency and accuracy of the synchronization status between the source database and the target database. Through the continuous recording of log points and transaction identifiers, the system can quickly restore data synchronization when an exception occurs, ensuring the stability and fault tolerance of the system. Customized replication rules and optimized data transmission paths effectively reduce data synchronization delays and improve overall system performance.

[0031] As an optional example, the first replication component is specifically used for:

[0032] Determine the library or table corresponding to the SQL request;

[0033] When the database or table corresponding to the SQL request exists in the replication rule, it is determined that the log data needs to be replicated;

[0034] When the database or table corresponding to the SQL request does not exist in the replication rule, it is determined that the log data does not need to be replicated.

[0035] Optionally, in this embodiment, the main function of the first replication component is to process and judge the SQL request in the source database according to the replication rule of the target database to determine which log data needs to be synchronized to the target database. Not only does it improve the synchronization efficiency, it also avoids the synchronization of irrelevant data and optimizes the utilization of resources. The replication component sets whitelists and blacklists respectively by configuring replication rules such as libraries and tables. When the data synchronization blacklist of the library or table is configured, the data involved in the library and table in the blacklist will be excluded from the synchronization scope, which means that the log data generated by the operation (such as insert, update, delete) of the library or table in the blacklist will not be sent to the replication component of the target database. Through the blacklist mechanism, the synchronization of irrelevant data can be effectively avoided, bandwidth occupation and storage resource waste can be reduced. For example, some log tables or historical data tables that do not require disaster recovery may not need to be synchronized to the target database. The libraries and tables in the blacklist do not participate in the data replication of the target cluster, nor will they affect the master node election and high availability mechanism of the source cluster. The data that does not need to be synchronized in the target cluster will not affect the normal operation of the master node, maintaining the high availability of the source cluster and the master node election logic. In contrast to the blacklist, the whitelist refers to synchronizing only the data of the libraries or tables specified in the whitelist. Only these specified libraries or tables are synchronized in the target database, and the data of other libraries and tables will be ignored. By using the whitelist, the system can accurately control which data needs to be synchronized, ensure the targeted replication of data, and further improve the efficiency and controllability of synchronization.

[0036] Optionally, in this embodiment, when the business end initiates an SQL request, the first replication component first identifies the database (library) or table involved in the request. By parsing the SQL statement, the component can accurately find the target object of the operation. If the library or table corresponding to the SQL request is a whitelist object in the replication rule of the target database, the first replication component will confirm that these log data need to be replicated. At this time, the log data generated by the source database will be sent to the target database for synchronization. If the library or table corresponding to the SQL request is not in the replication rule of the target database, or is in the blacklist, the first replication component will determine that these log data do not need to be replicated. At this time, these log data will be ignored or marked as not requiring synchronization to avoid waste of irrelevant data.

[0037] As an optional example, the first replication component is further configured to:

[0038] When it is determined that the log data does not need to be copied, the log data is filtered so that the filtered log data only contains the transaction identifier of the SQL request;

[0039] The filtered log data is sent to the second replication component, so that the target database receives the filtered log data through the second replication component, and records the synchronization log point according to the transaction identifier in the filtered log data.

[0040] Optionally, in this embodiment, in addition to determining which log data needs to be replicated, the first replication component also undertakes the task of filtering the data that does not need to be replicated and sending the filtered log data to the target database. When the first replication component determines that some log data does not need to be replicated, it will filter these log data. Specifically, the filtering operation is to remove the actual change content (such as data insertion, update, deletion, etc.) in the log data, and only retain the transaction identification information related to the transaction. The filtered log data only contains the transaction identification (such as transaction ID) of the SQL request. This method can ensure that the target database can still identify and track the execution status of the transaction without actually synchronizing the data change content. The filtered log data will be sent to the second replication component of the target database. After the target database receives these logs, it will record the synchronization log point according to the transaction identification in the log. The target database records the synchronization log point according to the transaction identification to ensure the continuity and consistency of data synchronization. These points are used to verify the progress of data synchronization, ensure that the data synchronization status of the source and target ends is consistent, and even in the event of network interruption or failure, the synchronization process can be restored from the log point.

[0041] As an optional example, the first replication component is further configured to:

[0042] When the log data of multiple consecutive SQL requests are determined not to need to be copied, the filtered log data of the multiple SQL requests are merged to obtain merged log data;

[0043] The merged log data is sent to the second replication component, so that the target database receives the merged log data through the second replication component, and records the synchronization log point according to the last transaction identifier in the merged log data.

[0044] Optionally, in this embodiment, the further optimization function of the first replication component in log data processing includes: when multiple continuous SQL request log data are determined to be unnecessary to be replicated, these log data are merged to reduce the processing and transmission volume of log data. When the log data generated by multiple continuous SQL requests are determined to be unnecessary to be replicated, the first replication component will filter these log data, retain only the transaction identifier, and merge these filtered log data into a single log record. The merged log data will contain all relevant transaction identifiers, but will not contain specific change data. The purpose of merging logs is to reduce unnecessary repeated transmissions, reduce network load and storage pressure, and maintain the effectiveness of the synchronization process. The first replication component sends the merged log data to the second replication component of the target database. After receiving these log data, the second replication component will record the synchronization log point according to the last transaction identifier in the log, and the target database will record the synchronization log point according to the last transaction identifier in the merged log data. These points represent the latest status of the current synchronization, ensuring the data synchronization continuity of the source database and the target database. Even if there are multiple transactions that do not need to be replicated, the merged log can still provide the target database with information on the synchronization progress.

[0045] As an optional example, the second replication component is also used to continue to obtain log data from the first replication component according to the synchronization log point after the first replication component or the second replication component has an exception and recovers.

[0046] Optionally, in this embodiment, the role of the second replication component in the entire data synchronization process is to receive and parse the log data transmitted from the first replication component to complete the synchronization of the target database. When the first replication component or the second replication component is abnormal and recovered, the second replication component also assumes the function of ensuring that the synchronization process continues. In the normal synchronization process, the target database will record the synchronization log point according to the transaction identifier in the log data, and the synchronization log point is a sign of the synchronization progress of the target database and the source database. If the first replication component or the second replication component has an abnormality during the synchronization process (such as network interruption, service downtime, etc.), the synchronization process will be interrupted, and the target database may not be able to continue to synchronize the newly generated log data. Once the first replication component or the second replication component resumes work, the second replication component will re-pull the log data from the first replication component according to the previously recorded synchronization log point, and this log point indicates the last successfully synchronized transaction position. The second replication component will start from the recorded synchronization log point and continue to pull log data from the source database to ensure that the synchronization process will not lose any data and can be successfully restored. This process ensures data consistency between the target database and the source database. By recording and using synchronization log points, the second replication component can ensure that even if an exception occurs, it can accurately resume synchronization from the interruption point, without missing any critical data or repeating the synchronization of data that has already been synchronized. Log points serve as important "timestamps" in the synchronization process, which can help the second replication component accurately restore the synchronization progress and avoid data loss or inconsistency.

[0047] As an optional example, the first replication component is further configured to send the log data to a standby database of the source database, so that the standby database performs data replication by parsing and replaying the log data.

[0048] Optionally, in this embodiment, the function of the first replication component is not limited to data synchronization to the target database, it can also send log data to the standby database (such as a replica database, a disaster recovery database, etc.) of the source database to achieve synchronization and data replication of the standby database. When the source database generates log data, the first replication component will send the same log data to the standby database of the source database in addition to sending the log data to be synchronized to the second replication component of the target database. The standby database can be a replica database used for disaster recovery, data recovery or load balancing. After receiving the log data, the standby database will apply the operations recorded in the log (such as insert, update, delete, etc.) to the table of the standby database through log parsing and playback to achieve data replication. In this way, the standby database maintains data consistency with the source database. Whether it is a business request of the main database or log data triggered by the replication rule, the standby database will keep in sync with the source database. The standby database is usually used as a disaster recovery library. It can quickly take over and provide data recovery services when the main database fails. By sending log data to the standby database, it is ensured that the standby database can obtain data consistent with the main database at any time. The real-time synchronization and data replication mechanism of the standby database makes the system more fault-tolerant and available. When the primary database fails, the standby database can recover quickly and continue to provide services, reducing business interruption time.

[0049] According to a second aspect of an embodiment of the present application, a method for synchronizing a database log is provided, which is applied to a synchronization system for a database log. Optionally, as follows: Figure 2 As shown, the above method includes:

[0050] S202, executing the SQL request received from the business end through the source database and generating log data;

[0051] S204, judging whether the log data needs to be replicated according to the replication rule of the target database by using the first replication component of the source database, wherein one target database corresponds to one replication rule, and the replication rule includes tables and libraries that need to be replicated;

[0052] S206, when it is determined that the log data needs to be replicated, the log data is sent to the second replication component of the target database, so that the target database receives the log data through the second replication component, replicates the data by parsing and replaying the log data, and records the synchronization log point according to the transaction identifier in the log data, wherein the synchronization log point is used for continuity verification.

[0053] Optionally, in this embodiment, a data replication process between a source database and a target database is involved, and the specific operation process includes the generation, transmission, parsing and playback of log data. Its key components include a first replication component of the source database, a second replication component of the target database, and a mechanism for defining data replication rules. The target database is a complete database, which is set for different purposes according to the usage scenario, and the typical scenario is a disaster recovery database and a static database. The main goal of the system is to efficiently and in real time synchronize the data in the source database to the target database, while ensuring the consistency of the data and the traceability of the synchronization status. The source database performs data operations such as inserting, updating or deleting according to the SQL request received by the business end, and these SQL operations will be converted into database logs to record the changes within the database. Whenever the data changes in the source database, the source database will generate corresponding log data, which contains detailed information and transaction identifiers of the data change. The first replication component of the source database is responsible for determining which log data needs to be replicated according to the replication rules defined by the target database. The replication rules can specify which tables and which libraries need to be synchronized to the target database. When processing log data, the first replication component will filter out the log data that needs to be replicated according to the rules, and ensure that unnecessary data (such as data in irrelevant tables) will not be synchronized, thereby improving performance and reducing bandwidth waste. After the target database receives the log data sent by the source database, the second replication component is responsible for parsing these logs and playing them back to the target database. The playback process is to accurately synchronize the data changes in the source database to the target database. The transaction identifiers in the log data are used to record synchronization log points in the target database. These points help to determine the continuity of data synchronization and ensure the consistency of the target database with the source database. Even if there are network problems or interruptions, the synchronization process can be resumed from the interruption point.

[0054] Optionally, in this embodiment, by defining precise replication rules, only critical data is replicated, synchronization of irrelevant data is avoided, synchronization efficiency is improved, and bandwidth and storage costs are reduced. The replication component of the source database filters log data according to the replication rules of the target database, reducing the processing and transmission of irrelevant data and optimizing resource utilization. The playback of log data and the recording of synchronization log points ensure the data consistency and accuracy of the synchronization status between the source database and the target database. Through the continuous recording of log points and transaction identifiers, the system can quickly restore data synchronization when an exception occurs, ensuring the stability and fault tolerance of the system. Customized replication rules and optimized data transmission paths effectively reduce data synchronization delays and improve overall system performance.

[0055] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0056] For other examples of this embodiment, please refer to the above examples and will not be repeated here.

[0057] According to a third aspect of the embodiment of the present application, a database log synchronization system device is also provided, which is applied to a database log synchronization system, such as Figure 3 As shown, including:

[0058] A generating module 302 is used to execute the SQL request received from the business end through the source database and generate log data;

[0059] The judgment module 304 is used to judge whether the log data needs to be replicated according to the replication rule of the target database through the first replication component of the source database, wherein one target database corresponds to one replication rule, and the replication rule includes the tables and libraries that need to be replicated;

[0060] The synchronization module 306 is used to send the log data to the second replication component of the target database when it is determined that the log data needs to be replicated, so that the target database receives the log data through the second replication component, replicates the data by parsing and replaying the log data, and records the synchronization log point according to the transaction identifier in the log data, wherein the synchronization log point is used for continuity verification.

[0061] It should be noted that the generation module 302 in this embodiment can be used to execute step S202 in the embodiment of the present application, the judgment module 304 in this embodiment can be used to execute step S204 in the embodiment of the present application, and the synchronization module 306 in this embodiment can be used to execute step S206 in the embodiment of the present application.

[0062] For other examples of this embodiment, please refer to the above examples and will not be repeated here.

[0063] Figure 4 is a schematic diagram of an optional electronic device according to an embodiment of the present application, such as Figure 4 As shown, it includes a processor 402, a communication interface 404, a memory 406 and a communication bus 408, wherein the processor 402, the communication interface 404 and the memory 406 communicate with each other through the communication bus 408, wherein,

[0064] Memory 406, used to store computer programs;

[0065] The processor 402 is used to implement the following steps when executing the computer program stored in the memory 406:

[0066] Execute the SQL request received from the business end through the source database and generate log data;

[0067] Determining whether the log data needs to be replicated according to the replication rule of the target database by using the first replication component of the source database, wherein one target database corresponds to one replication rule, and the replication rule includes tables and libraries that need to be replicated;

[0068] When it is determined that the log data needs to be replicated, the log data is sent to the second replication component of the target database, so that the target database receives the log data through the second replication component, and the data is replicated by parsing and replaying the log data, and the synchronization log point is recorded according to the transaction identifier in the log data, wherein the synchronization log point is used for continuity verification.

[0069] Optionally, in this embodiment, the communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industrial Standard Architecture) bus. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The communication interface is used for communication between the above electronic device and other devices.

[0070] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0071] As an example, the memory 406 may include but is not limited to the generation module 302, the judgment module 304 and the synchronization module 306 in the synchronization device for database logs. In addition, it may also include but is not limited to other module units in the synchronization device for database logs, which will not be described in detail in this example.

[0072] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0073] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0074] It can be understood by those skilled in the art that Figure 4 The structure shown is for illustration only. The device for implementing the above-mentioned database log synchronization method may be a terminal device, which may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Figure 4 The structure of the electronic device is not limited. For example, the electronic device may also include Figure 4 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 4 Different configurations shown.

[0075] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0076] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program executes the steps in the above-mentioned database log synchronization method when executed by a processor.

[0077] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0078] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0079] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0080] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0081] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0082] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0083] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0084] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A database log synchronization system, characterized in that: include: The source database is used to execute SQL requests received from the business end and generate log data; The source database includes a first replication component, the first replication component is used to determine whether the log data needs to be replicated according to the replication rule of the target database, and if it is determined that the log data needs to be replicated, send the log data to a second replication component of the target database, wherein one target database corresponds to one replication rule, and the replication rule includes tables and libraries that need to be replicated; The target database is used to parse and replay the log data when the log data is received through the second replication component to perform data replication, and record synchronization log points according to transaction identifiers in the log data, wherein the synchronization log points are used for continuity verification.

2. The system according to claim 1, characterized in that The first replication component is specifically used for: Determine the library or table corresponding to the SQL request; In a case where the library or table corresponding to the SQL request exists in the replication rule, determining that the log data needs to be replicated; When the library or table corresponding to the SQL request does not exist in the replication rule, it is determined that the log data does not need to be replicated.

3. The system according to claim 1, characterized in that The first replication component is also used for: When it is determined that the log data does not need to be copied, filtering the log data so that the filtered log data only contains the transaction identifier of the SQL request; The filtered log data is sent to the second replication component, so that the target database receives the filtered log data through the second replication component, and records the synchronization log point according to the transaction identifier in the filtered log data.

4. The system according to claim 3, characterized in that The first replication component is also used for: When the log data of multiple consecutive SQL requests are all determined not to need to be copied, merging the filtered log data of the multiple SQL requests to obtain merged log data; The merged log data is sent to the second replication component, so that the target database receives the merged log data through the second replication component, and records the synchronization log point according to the last transaction identifier in the merged log data.

5. The system according to claim 1, characterized in that The second replication component is also used to continue to obtain log data from the first replication component according to the synchronization log point after the first replication component or the second replication component has an exception and recovers.

6. The system according to claim 1, characterized in that The first replication component is further configured to send the log data to a standby database of the source database, so that the standby database replicates the data by parsing and replaying the log data.

7. A method for synchronizing database logs, characterized in that: A synchronization system applied to database logs, including: Execute the SQL request received from the business end through the source database and generate log data; Determining whether the log data needs to be replicated according to a replication rule of a target database by using a first replication component of the source database, wherein one target database corresponds to one replication rule, and the replication rule includes tables and libraries that need to be replicated; When it is determined that the log data needs to be replicated, the log data is sent to the second replication component of the target database, so that the target database receives the log data through the second replication component, replicates the data by parsing and replaying the log data, and records the synchronization log point according to the transaction identifier in the log data, wherein the synchronization log point is used for continuity verification.

8. A database log synchronization device, characterized in that: A synchronization system applied to database logs, including: A generation module is used to execute SQL requests received from the business end through the source database and generate log data; A judgment module, configured to judge whether the log data needs to be replicated according to a replication rule of a target database through a first replication component of the source database, wherein one target database corresponds to one replication rule, and the replication rule includes tables and libraries that need to be replicated; A synchronization module is used to send the log data to the second replication component of the target database when it is determined that the log data needs to be replicated, so that the target database receives the log data through the second replication component, replicates the data by parsing and replaying the log data, and records the synchronization log point according to the transaction identifier in the log data, wherein the synchronization log point is used for continuity verification.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method described in claim 7 is performed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method of claim 7 through the computer program.

Images