Database data synchronization method and device, storage medium and product

Abstract

The invention provides a data synchronization method and device of a database, a storage medium and a product. The data synchronization method of the database comprises the steps of obtaining a table mapping log event of a source end database; determining whether a data table recorded by the table mapping log event needs to be filtered or not according to preset configuration parameters, and if yes, updating data table information recorded by the table mapping log event in a filtering buffer so as to selectively read and analyze a row record log event of the source end database according to the information recorded by the filtering buffer; wherein the configuration parameters are used for identifying the data table needing to be synchronized and the database to which the table belongs, or the data table needing to be filtered and the database to which the table belongs. The data table information needing to be filtered is stored in the filtering buffer, the row record log event of the source end database can be selectively read and analyzed according to the information recorded by the filtering buffer, and resource consumption of a database system is reduced.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a database data synchronization method, device, storage medium and product. Background Art

[0002] During the operation of a database system, data synchronization between two databases is a common work requirement. The database that provides data is usually called the source database, and the database that receives data is usually called the target database. In the process of data synchronization between two databases, the stock data is first initialized and loaded, that is, the stock data of the source database is synchronized to the target database as the basis for subsequent data synchronization, and then the incremental data is synchronized based on the basis. For the synchronization of incremental data, it is necessary to obtain the incremental data by analyzing the logs of the source database.

[0003] In the actual use of the database system, according to the user's business needs, sometimes part of the data in the source database does not need to be synchronized to the target database, so the data will be filtered. Specifically, after parsing the logs of the source database into synchronized data that can be synchronized to the target database, the data that does not need to be synchronized is deleted according to the filtering conditions, and the remaining data is synchronized to the target database. However, because parsing the logs of the source database also consumes a lot of system resources, parsing operations are also performed on data that does not need to be synchronized, resulting in a waste of system resources. Summary of the invention

[0004] An object of the present invention is to provide a database data synchronization method, device, storage medium and product that can solve any of the above problems.

[0005] In particular, the present invention provides a method for synchronizing data in a database, comprising:

[0006] Get the table mapping log events of the source database;

[0007] Determine whether the data table of the table mapping log event record needs to be filtered according to the preset configuration parameters, and if so, update the data table information of the table mapping log event record in the filter buffer to selectively read and parse the log event of the source database according to the information recorded in the filter buffer;

[0008] The configuration parameters are used to identify the data tables that need to be synchronized and the databases to which the tables belong, or the data tables that need to be filtered and the databases to which the tables belong.

[0009] Optionally, after determining that the data table of the table mapping log event record needs to be filtered, the method includes: stopping reading the table mapping log event and generating an event file to be parsed with a filtering mark;

[0010] After determining that the data table of the table mapping log event record does not need to be filtered, it includes: reading the table mapping log event and generating a complete event file to be parsed, and updating the data table information of the table mapping log event record in a whitelist buffer, wherein the whitelist buffer is used to store information of the data table that needs to be synchronized.

[0011] Optionally, before the step of determining whether the data table of the table mapping log event record needs to be filtered according to the pre-configured configuration parameters, the step includes:

[0012] Detect whether the data table of the table mapping log event record exists in the filtering buffer, and if so, execute the step of stopping reading the table mapping log event and generating a to-be-parsed event file with a filtering mark.

[0013] Optionally, the step of detecting whether the data table of the table mapping log event record exists in the filtering buffer includes:

[0014] If not, detect whether the data table of the table mapping log event record exists in the whitelist buffer. If so, execute the step of reading the table mapping log event and generating a complete event file to be parsed. If not, execute the step of determining whether the data table of the table mapping log event record needs to be filtered according to the pre-configured configuration parameters.

[0015] Optionally, the database data synchronization method includes:

[0016] Get the row record log event of the source database;

[0017] Detect whether the data table of the row record log event record exists in the filter buffer. If so, stop reading the row record log event and generate an event file to be parsed with a filter mark. If not, read the row record log event and generate a complete event file to be parsed.

[0018] Optionally, the database data synchronization method includes:

[0019] Get the generated event file to be parsed;

[0020] Check whether the event file to be parsed has a filtering mark, if so, skip the event to be parsed, if not, parse the event file to be parsed.

[0021] Optionally, the step of updating the data table information of the table mapping log event record in the filter buffer includes:

[0022] The table number and table name of the data table of the table mapping log event record and the database name are stored in the filtering buffer in the form of key-value pairs.

[0023] According to another aspect of the present invention, a computer device is also provided, comprising a memory, a processor, and a computer executable program stored in the memory and running on the processor, and when the processor executes the computer executable program, the database data synchronization method according to any one of the above items is implemented.

[0024] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer-executable program is stored. When the computer-executable program is executed by a processor, the method for synchronizing database data according to any one of the above items is implemented.

[0025] According to another aspect of the present invention, a computer program product is provided, including a computer executable program, and when the computer executable program is executed by a processor, the database data synchronization method according to any one of the above items is implemented.

[0026] The data synchronization method, device, storage medium and product of the database of the present invention provide external configuration parameters and configure the filtering buffer. After obtaining the table mapping log event of the source database, it is determined whether the data table recorded in the table mapping log event needs to be filtered according to the preset configuration parameters. If filtering is required, the data table information recorded in the table mapping log event is updated in the filtering buffer. Because there must be a corresponding table mapping log event before each row record log event, when the subsequent log event is obtained, the corresponding table mapping log event must have been detected. In this way, in the subsequent synchronization work, after obtaining whether it is a table mapping log event or a row record log event, it can be determined whether the row record log event needs to be filtered according to the information recorded in the filtering buffer. If it needs to be filtered, the log event can be read and parsed, that is, the log event of the source database can be selectively read and parsed. Therefore, it is not necessary to read and parse all log events completely, which reduces the resource consumption of the database system and helps to improve the synchronization efficiency of the database system. In addition, by setting the configuration parameters to be synchronized or not synchronized, the user can flexibly set the configuration parameters according to actual needs, reducing the occurrence of subsequent configuration modifications.

[0027] Based on the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will become more aware of the above and other objects, advantages and features of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Hereinafter, some specific embodiments of the present invention will be described in detail in an exemplary and non-limiting manner with reference to the accompanying drawings. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. In the accompanying drawings:

[0029] Figure 1 is a schematic flow chart of a method for synchronizing database data according to an embodiment of the present invention;

[0030] Figure 2 is a schematic flow chart of a method for synchronizing database data according to another embodiment of the present invention;

[0031] Figure 3 is a schematic flow chart of a method for synchronizing database data according to yet another embodiment of the present invention;

[0032] Figure 4 is a schematic flow chart of a method for synchronizing database data according to yet another embodiment of the present invention;

[0033] Figure 5 is a schematic flow chart of a method for synchronizing database data according to yet another embodiment of the present invention;

[0034] Figure 6 is a schematic diagram of a computer device according to an embodiment of the present invention;

[0035] Figure 7 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention;

[0036] Figure 8 is a schematic diagram of a computer program product according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] It should be understood by those skilled in the art that the embodiments described below are only some embodiments of the present invention, rather than all embodiments of the present invention, and these embodiments are intended to explain the technical principles of the present invention, rather than to limit the protection scope of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should still fall within the protection scope of the present invention.

[0038] It should be noted that the logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or equipment (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or equipment and execute instructions), or used in combination with these instruction execution systems, devices or equipment.

[0039] The flow chart provided by the present invention is not intended to indicate that the operation of the method will be performed in any particular order, or all operations of the method are included in all every case. In addition, the method may include additional operations. Within the scope of the technical thinking provided by the present embodiment method, additional changes can be made to the above method.

[0040] like Figure 1 As shown, in one embodiment, the data synchronization method of the database generally includes:

[0041] Step S101, obtain the table mapping log event of the source database. Specifically, for the binlog (Binary Log) of the database, historical information is recorded in the form of an event (in the database field, it can be called event), so it can be said that each log records an event. Among them, the table mapping log event (in the database field, it can be called TABLE_MAP_LOG_EVENT) and the row record log event (in the database field, it can be called ROWS_LOG_EVENT) are two types of log events. The row record log event is a log event for performing specific operations on the data table, including an insert operation log event (in the database field, it can be called WRITE_ROWS_LOG_EVENT), an update operation log event (in the database field, it can be called UPDATE_ROWS_LOG_EVENT) and a delete operation log event (in the database field, it can be called DELETE_ROWS_LOG_EVENT).

[0042] Before recording the log event of a specific operation, the database will write a table mapping log event in the binlog. The table mapping log event records the definition information of the data table to be modified by the specific operation, including the table name, the name of the database to which it belongs, and a table number (i.e., table_id) converted from the definition information. The table mapping log event is used to provide the table definition information for subsequent row record log events, and subsequent row record log events only need to reference the table number and can be parsed without recording the table information, reducing storage usage.

[0043] Therefore, for a transaction, a table mapping log event will be generated in the binlog for each table corresponding to all specific operations in the transaction.

[0044] Furthermore, in binlog, the recorded event type is identified by a type code (in the database field, it can be called type_code). Therefore, for this step, a log file of a source database is obtained, and the event type recorded in the log file is determined according to the type code, and it is determined that the log file records a table mapping log event, that is, the table mapping log event of the source database is obtained.

[0045] Step S102, determine whether the data table of the table mapping log event record needs to be filtered according to the preset configuration parameters, if yes, execute step S103, if not, execute step S104. The configuration parameters are used to identify the data table to be synchronized and the database to which the table belongs, or the data table to be filtered and the database to which the table belongs.

[0046] Specifically, the data synchronization software provides external configuration parameters, which have two configuration modes: one for users to fill in the configuration of data tables that need to be synchronized, and the other for users to fill in the configuration of data tables that do not need to be synchronized, that is, data tables that need to be filtered. Users can set configuration parameters as needed. For example, if there are fewer data tables that need to be synchronized, fill in the configuration parameters that need to be synchronized; if there are fewer data tables that do not need to be synchronized, fill in the configuration parameters that do not need to be synchronized.

[0047] Furthermore, configuration parameters are used to identify data tables and the databases to which the tables belong. For example, parameter configuration can be performed directly in the format of "database name.table name", or can be identified using pre-set wildcard symbols. For example, "*" represents all data tables, and "database name.*" means that all data tables under the database need to be synchronized, or none of them need to be synchronized.

[0048] After obtaining a table mapping log event, first read the table identification information of the table mapping log event, which at least includes the table name and the name of the database to which it belongs. Then, by matching the table name and database name with the configuration parameters, it can be determined whether the data table recorded in the current table mapping log event needs to be filtered.

[0049] Exemplarily, for example, the configuration parameters set the data tables and databases to which the tables belong that do not need to be synchronized. If the table identification information of the current table mapping log event record can successfully match the configuration parameters, it means that the data table of the current table mapping log event record needs to be filtered. If it cannot successfully match the configuration parameters, it means that the data table of the current table mapping log event record needs to be synchronized.

[0050] For example, the configuration parameters set the data table that needs to be synchronized and the database to which the table belongs. If the table identification information of the current table mapping log event record can successfully match the configuration parameters, it means that the data table of the current table mapping log event record needs to be synchronized. If it cannot successfully match the configuration parameters, it means that the data table of the current table mapping log event record needs to be filtered.

[0051] Step S103, updating the data table information of the table mapping log event record in the filter buffer. Specifically, the data table information includes at least the data table name and the name of the database to which it belongs. The filter buffer is a pre-set storage space for storing information of the data table that needs to be filtered. If the data table of the table mapping log event record needs to be filtered, the data table information of the table mapping log event record is added to the filter buffer. Therefore, the log events of the source database can be selectively read and parsed according to the information recorded in the filter buffer, that is, the log events corresponding to the data table in the filter buffer are not read and parsed.

[0052] Furthermore, this step includes: storing the table number and table name of the data table of the table mapping log event record and the database name in the form of a key-value pair in the filter buffer. Specifically, a key-value pair is a set of data that corresponds one to one (in the database field, it can be called a key-value), that is, a key-value pair consisting of a table number on one side and a table name and a database name on the other side. Through the above settings, the search efficiency of the information stored in the filter buffer can be improved, thereby improving the synchronization efficiency.

[0053] In addition, when reading the table identification information of the read table mapping log event, the table identification information also includes the table number.

[0054] Step S104, updating the data table information of the table mapping log event record in the whitelist buffer. If the data table of the table mapping log event record needs to be synchronized, the data table information of the table mapping log event record is added to the whitelist buffer. In the whitelist buffer, only the table name and database name are stored, and they are stored in the form of a linked list, that is, a corresponding set of table names and database names can be regarded as a node of the linked list.

[0055] In the scheme of this embodiment, by providing external configuration parameters and configuring the filter buffer, after obtaining the table mapping log event of the source database, it is determined whether the data table recorded by the table mapping log event needs to be filtered according to the preset configuration parameters. If filtering is required, the data table information recorded by the table mapping log event is updated in the filter buffer. Because there must be a corresponding table mapping log event before each row record log event, when the row record log event is obtained, its corresponding table mapping log event must have been detected. In this way, in the subsequent synchronization work, after obtaining whether it is a table mapping log event or a row record log event, it can be determined whether the log event needs to be filtered according to the information recorded in the filter buffer. If it needs to be filtered, the log event can be read and parsed without reading and parsing, that is, the log event of the source database is selectively read and parsed. Therefore, there is no need to read and parse all log events completely, which reduces the resource consumption of the database system and helps to improve the synchronization efficiency of the database system.

[0056] In addition, by setting the configuration parameters to require synchronization or not, the user can flexibly set the configuration parameters according to actual needs, thereby reducing the occurrence of subsequent configuration modifications.

[0057] Furthermore, by setting configuration parameters, the data table information is added to the filter buffer after the table mapping log event is subsequently obtained, so that the table number can be obtained and added to the filter buffer together. By storing the table number and table name and database name of the data table recorded in the table mapping log event in the form of key-value pairs in the filter buffer, when the filter buffer is used to determine whether the log event needs to be filtered, the information stored in the filter buffer does not need to be traversed through all the contents of the filter memory, thereby improving the efficiency of searching the information stored in the filter buffer, and further helping to improve the efficiency of data synchronization.

[0058] Furthermore, by setting up a whitelist cache to store information about data tables that need to be synchronized, when subsequently determining whether table mapping log events need to be filtered, the filter cache and the whitelist cache can be used to make a judgment first. Compared with only the filter cache, the frequency of connecting to external configuration parameters can be reduced, thereby reducing resource consumption.

[0059] like Figure 2 As shown, in one embodiment, the data synchronization method of the database generally includes:

[0060] Step S201, obtaining table mapping log events of the source database.

[0061] Step S202, determining whether the data table of the table mapping log event record needs to be filtered according to the preset configuration parameters, if so, executing step S203, if not, executing step S205.

[0062] Step S203: Update the data table information of the table mapping log event record in the filter buffer.

[0063] Step S204, stop reading the table mapping log event and generate an event file to be parsed with a filter mark. Specifically, after determining that the data table recorded in the table mapping log event needs to be filtered, only the table identification information of the table mapping log event is read, and the subsequent content is no longer read, and then a file to be parsed with a filter mark is generated. In other words, the content in the event file to be parsed with a filter mark is not complete.

[0064] It should be noted that this step and step S203 are not limited in sequence.

[0065] Step S205: Update the data table information of the table mapping log event record in the whitelist buffer.

[0066] Step S206, read the table mapping log event and generate a complete event file to be parsed. Specifically, after determining that the data table recorded in the table mapping log event does not need to be filtered, continue to read the remaining content of the table mapping log event until all reading is completed to generate a complete event file to be parsed.

[0067] In the solution of this embodiment, after determining that the data table of the table mapping log event record needs to be filtered, the reading of the table mapping log event is stopped and an event file to be parsed with a filter mark is generated. That is to say, even if the table mapping log event needs to be filtered, an event file to be parsed is still formed. Because it carries the filter mark, it can be determined not to be parsed based on the filter mark later. On the basis of reducing system resource consumption, the integrity of the number of events in a transaction is guaranteed, and the occurrence of synchronization failures is reduced.

[0068] like Figure 3 As shown, in one embodiment, the data synchronization method of the database generally includes:

[0069] Step S301, obtaining table mapping log events of the source database.

[0070] Step S302, detect whether the data table recorded by the table mapping log event exists in the filter buffer, if so, execute step S306, if not, execute step S303. Specifically, because the same table may be operated in different transactions, different transactions may generate table mapping log events for the same table. Therefore, after obtaining the table mapping log event of the source database, first detect whether the data table recorded by the table mapping log event exists in the filter buffer, that is, find out whether there is a table number of the current table mapping log event in the information recorded in the filter buffer.

[0071] Step S303, check whether the data table of the table mapping log event record exists in the whitelist buffer, if yes, execute step S308, if not, execute step S304. If it does not exist in the filter buffer, then check whether the data table of the table mapping log event record exists in the whitelist buffer, that is, find out whether the table name and database name of the current table mapping log event exist in the information recorded in the whitelist buffer.

[0072] Step S304, determine whether the data table of the table mapping log event record needs to be filtered according to the preset configuration parameters, if yes, execute step S305, if no, execute step S307. If it does not exist in the filter buffer or the whitelist buffer, it means that the data table of the table mapping log event record is read for the first time, then determine whether the data table of the table mapping log event record needs to be filtered according to the configuration parameters.

[0073] Step S305: Update the data table information of the table mapping log event record in the filtering buffer. If it is determined that the data table of the table mapping log event record needs to be filtered, the data table information is added to the filtering buffer.

[0074] Step S306, stop reading the table mapping log event and generate the event file to be parsed with the filter mark. If the data table recorded by the table mapping log event exists in the filter buffer, or it is determined through the configuration parameters that the data table recorded by the table mapping log event needs to be filtered, stop reading the table mapping log event and generate the event file to be parsed with the filter mark.

[0075] Step S307, updating the data table information of the table mapping log event record in the white list buffer. If it is determined that the data table of the table mapping log event record does not need to be filtered, the data table information is added to the white list buffer.

[0076] Step S308, read the table mapping log event and generate a complete event file to be parsed. Specifically, if the data table recorded by the table mapping log event exists in the whitelist buffer, or after determining through configuration parameters that the data table recorded by the table mapping log event does not need to be filtered, continue to read the remaining content of the table mapping log event until all reading is completed to generate a complete event file to be parsed.

[0077] In the scheme of this embodiment, by setting a filter cache and a whitelist cache, after obtaining the table mapping log event of the source database, the filter cache and the whitelist cache can be used to determine whether the data table needs to be filtered. If there is no corresponding information in the filter cache and the whitelist cache, then compare it with the configuration parameters. That is to say, first compare from the internal cache. If there is no internal cache, then compare with the external configuration parameters to avoid frequent connection to the external configuration parameters, thereby helping to reduce resource consumption.

[0078] Moreover, because there is a whitelist cache, even if the filter cache cannot be found, it can be searched in the whitelist cache instead of directly connecting to the external configuration parameters. Compared with only the filter cache, the frequency of connecting to the external configuration parameters can be further reduced, thereby reducing resource consumption.

[0079] Furthermore, because the detection of table mapping log events and row record log events both utilizes the filter cache, and the table mapping log event first searches the filter cache, by making the filter cache store information in the form of key-value pairs, the information stored in the filter cache does not need to traverse all the contents of the filter memory to search, thereby improving the search efficiency of the information stored in the filter cache, that is, improving the efficiency of detecting whether the log event needs to be filtered, which in turn helps to improve data synchronization efficiency.

[0080] like Figure 4 As shown, in one embodiment, the data synchronization method of the database generally includes:

[0081] Step S401, obtaining a row record log event of a source database. Specifically, obtaining a log file of a source database, determining the event type recorded in the log file according to the type code, and determining that the log file records a row record log event, that is, obtaining a row record log event of the source database.

[0082] Step S402, detect whether the data table recorded in the row record log event exists in the filter buffer, if so, execute step S403, if not, execute step S404. Because each row record log event must be preceded by a corresponding table mapping log event, when the row record log event is obtained, its corresponding table mapping log event must have been detected, and only by detecting whether the data table recorded in the row record log event exists in the filter buffer, it can be determined whether the row record log event needs to be filtered. Specifically, it is to read the table identification information of the row record log event, that is, the table number, table name and database name are compared with the information stored in the filter buffer.

[0083] Step S403, stop reading the row record log event and generate an event file to be parsed with a filter mark. Specifically, after determining that the data table of the row record log event record needs to be filtered, only the table identification information of the row record log event is read, and the subsequent content is no longer read, and then a file to be parsed with a filter mark is generated. In other words, the content in the event file to be parsed with a filter mark is not complete.

[0084] Step S404, read the row record log event and generate a complete event file to be parsed. Specifically, after determining that the data table of the row record log event record does not need to be filtered, continue to read the remaining content of the row record log event until all reading is completed to generate a complete event file to be parsed.

[0085] In the scheme of this embodiment, after determining that the data table of the row record log event record needs to be filtered, the reading of the row record log event is stopped and an event file to be parsed with a filter mark is generated. That is to say, even if the row record log event needs to be filtered, an event file to be parsed is still formed. Because it carries the filter mark, it can be determined not to be parsed based on the filter mark later. On the basis of reducing system resource consumption, the integrity of the number of events in a transaction is guaranteed, and the occurrence of synchronization failures is reduced.

[0086] like Figure 5 As shown, in one embodiment, the data synchronization method of the database generally includes:

[0087] Step S501, obtaining the generated event file to be parsed, specifically, obtaining the event file to be parsed read from the log of the source database.

[0088] Step S502, detect whether the event file to be parsed has a filter mark, if yes, execute step S503, if no, execute step S504. Specifically, the event file to be parsed with a filter mark does not need to be parsed, and the content is incomplete and cannot be parsed, so it is detected whether the event file to be parsed has a filter mark.

[0089] Step S503: skip the event file to be parsed. Specifically, the parsing of the event file to be parsed is stopped and the next event file to be parsed is obtained.

[0090] Step S504: parsing the event file to be parsed. Specifically, the event file to be parsed is parsed to obtain data that can be synchronized to the target database.

[0091] This embodiment also provides a computer device and a computer-readable storage medium. Figure 6 is a schematic diagram of a computer device 10 according to one embodiment of the present invention. Figure 7 is a schematic diagram of a computer-readable storage medium 20 according to one embodiment of the present invention.

[0092] The computer device 10 may include a memory 110, a processor 120, and a computer executable program 11 stored in the memory 110 and running on the processor 120, and the processor 120 implements the database data synchronization method of any of the above embodiments when executing the computer executable program 11.

[0093] The computer-readable storage medium 20 stores a computer-executable program 11 thereon. When the computer-executable program 11 is executed by a processor, the method for synchronizing database data in any of the above-mentioned embodiments can be implemented.

[0094] This embodiment also provides a computer program product. Figure 8 1 is a schematic diagram of a computer program product 30 according to an embodiment of the present invention. The computer program product 30 includes a computer executable program 11, and when the computer executable program 11 is executed by a processor 120, it can implement any of the above-mentioned database data synchronization methods.

[0095] Specifically, the computer executable program 11 for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, computer instructions, computer-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages.

[0096] For the purpose of the description of the present embodiment, the computer-readable storage medium 20 can be any device that can contain, store, communicate, propagate or transmit a program for use with an instruction execution system, device or equipment or in conjunction with these instruction execution systems, devices or equipment. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion (electronic device) with one or more wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium 20 can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting or processing in other suitable ways as necessary, and then stored in a computer memory.

[0097] It should be understood that each part of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system.

[0098] The computer device 10 may be, for example, a server, a desktop computer, a notebook computer, a tablet computer, or a smart phone. In some examples, the computer device 10 may be a cloud computing node. The computer device 10 may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, a program module may include routines, programs, target programs, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. The computer device 10 may be implemented in a distributed cloud computing environment where remote processing devices linked through a communication network perform tasks. In a distributed cloud computing environment, program modules may be located on a local or remote computing system storage medium including a storage device.

[0099] The computer device 10 may include a processor 120 adapted to execute stored instructions, and a memory 110 providing temporary storage space for the operation of the instructions during operation. The processor 120 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 110 may include a random access memory (RAM), a read-only memory, a flash memory, or any other suitable storage system.

[0100] The processor 120 may be connected to an I / O interface (input / output interface) suitable for connecting the computer device 10 to one or more I / O devices (input / output devices) via a system interconnect (e.g., PCI, PCI-Express, etc.). The I / O devices may include, for example, a keyboard and a pointing device, wherein the pointing device may include a touch pad or a touch screen, etc. The I / O devices may be built-in components of the computer device 10, or may be devices externally connected to the computing device.

[0101] Processor 120 can also be linked to a display interface suitable for connecting computer device 10 to a display device through a system interconnection. Display device can include a display screen as a built-in component of computer device 10. Display device can also include a computer monitor, a television or a projector, etc., which are externally connected to computer device 10. In addition, a network interface controller (NIC) can be suitable for connecting computer device 10 to a network through a system interconnection. In some embodiments, NIC can use any suitable interface or protocol (such as Internet Small Computer System Interface, etc.) to transmit data. The network can be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN) or the Internet, etc. Remote devices can be connected to computing devices through a network.

[0102] At this point, those skilled in the art should recognize that, although multiple exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications that conform to the principles of the present invention can still be directly determined or derived based on the content disclosed in the present invention without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention should be understood and recognized as covering all these other variations or modifications.

Claims

1. A method for synchronizing data in a database, comprising: Get the table mapping log events of the source database; Determine whether the data table of the table mapping log event record needs to be filtered according to the preset configuration parameters, and if so, update the data table information of the table mapping log event record in the filter buffer to selectively read and parse the log event of the source database according to the information recorded in the filter buffer; The configuration parameters are used to identify the data tables that need to be synchronized and the databases to which the tables belong, or the data tables that need to be filtered and the databases to which the tables belong.

2. The method for synchronizing database data according to claim 1, wherein: After determining that the data table of the table mapping log event record needs to be filtered, the method includes: stopping reading the table mapping log event and generating an event file to be parsed with a filtering mark; After determining that the data table of the table mapping log event record does not need to be filtered, it includes: reading the table mapping log event and generating a complete event file to be parsed, and updating the data table information of the table mapping log event record in a whitelist buffer, wherein the whitelist buffer is used to store information of the data table that needs to be synchronized.

3. The method for synchronizing database data according to claim 2, wherein: The step of determining whether the data table of the table mapping log event record needs to be filtered according to the pre-configured configuration parameters includes: Detect whether the data table of the table mapping log event record exists in the filtering buffer, and if so, execute the step of stopping reading the table mapping log event and generating a to-be-parsed event file with a filtering mark.

4. The method for synchronizing database data according to claim 3, wherein: The step of detecting whether the data table of the table mapping log event record exists in the filtering buffer includes: If not, detect whether the data table of the table mapping log event record exists in the whitelist buffer. If so, execute the step of reading the table mapping log event and generating a complete event file to be parsed. If not, execute the step of determining whether the data table of the table mapping log event record needs to be filtered according to the pre-configured configuration parameters.

5. The method for synchronizing database data according to claim 2, wherein: The data synchronization methods of the database include: Get the row record log event of the source database; Detect whether the data table of the row record log event record exists in the filter buffer. If so, stop reading the row record log event and generate an event file to be parsed with a filter mark. If not, read the row record log event and generate a complete event file to be parsed.

6. The method for synchronizing database data according to claim 5, wherein: The data synchronization methods of the database include: Get the generated event file to be parsed; Check whether the event file to be parsed has a filtering mark, if so, skip the event to be parsed, if not, parse the event file to be parsed.

7. The method for synchronizing database data according to claim 1, wherein: The step of updating the data table information of the table mapping log event record in the filter buffer comprises: The table number and table name of the data table of the table mapping log event record and the database name are stored in the filtering buffer in the form of key-value pairs.

8. A computer device, comprising a memory, a processor, and a computer executable program stored in the memory and running on the processor, wherein the processor implements the database data synchronization method according to any one of claims 1 to 7 when executing the computer executable program.

9. A computer-readable storage medium having a computer-executable program stored thereon, wherein the computer-executable program, when executed by a processor, implements the database data synchronization method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer executable program, wherein when the computer executable program is executed by a processor, the database data synchronization method according to any one of claims 1 to 7 is implemented.

Images