Database data synchronization method and device, equipment, medium and product

Abstract

The invention provides a data synchronization method and device of a database, equipment, a medium and a product, a second synchronization program of a destination end database can obtain a log file and parse the log file to obtain a first statement, then the first statement is stored in a cache, and under the condition that feature information of the cache meets a preset condition, the statement in the cache is executed, so that the data synchronization efficiency is improved. And storing the data of the statement operation in the cache into a destination end database. Therefore, in a data synchronization scene that small transactions are dense or a single transaction corresponds to a plurality of data tables and the like, the destination end database can reduce frequent data writing, so that the synchronization performance of the destination end database during data synchronization is improved.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a method, device, equipment, medium and product for synchronizing data in a database. Background Art

[0002] According to business needs, data synchronization operations are sometimes required between databases to ensure the consistency of data stored between the two databases. During data synchronization, the first synchronization program of the source database can send the log file of the data to be synchronized to the second synchronization program of the target database. After receiving the log file, the second synchronization program of the target database parses the log file to obtain the statement, and then stores the data operated by the statement in the database by executing the statement.

[0003] In the prior art, continuous statements generally operate on data in the same data table. The second synchronization program of the destination database stores multiple consecutive statements obtained by parsing the log file in the cache. When a statement different from the data table operated by the statement in the cache is subsequently parsed, the data stored in the cache is executed to store the data operated by the statement in the cache in the destination database, thereby reducing frequent data writing to the destination database.

[0004] However, in scenarios where small transactions are dense or a single transaction corresponds to multiple data tables, the second synchronization program of the destination database will frequently execute statements in the cache, thereby increasing the number of writes to the destination database and affecting the data synchronization performance of the destination database. Summary of the invention

[0005] The present application provides a database data synchronization method, device, equipment, medium and product, which are aimed at reducing the frequent writing of data to the destination database in scenarios such as small transactions are dense or a single transaction corresponds to multiple data tables, so as to improve the data synchronization performance of the database.

[0006] A first aspect of the present application provides a database data synchronization method, comprising: obtaining a log file from a source database; determining a first statement based on the log file; storing the first statement in a cache, wherein the cache table supports storing multiple statements from different transactions, and multiple statements for operating different data tables; when the first statement is stored in the cache, characteristic information of the cache meets preset conditions, executing the statement stored in the cache to store data operated by the statement in the cache in the destination database.

[0007] In an embodiment of the first aspect of the present application, storing the first statement into a cache includes: determining a first data table used by the first statement to operate; determining a first cache table in the cache based on the first data table and a mapping relationship, wherein the mapping relationship includes multiple data tables and a corresponding relationship between each data table and a cache table; storing the first statement into the first cache table.

[0008] In an embodiment of the first aspect of the present application, the preset condition includes at least one of the following: the number of transactions from which the multiple statements stored in the first cache table respectively come is greater than a first preset value; the time difference between the time when the first cache table stores the first statement and the time when the first statement is stored is greater than a second preset value; the number of cache tables stored in the cache is greater than a third preset value.

[0009] In an embodiment of the first aspect of the present application, after determining the first data table used to operate by the first statement, it also includes: when there is no corresponding cache table for the first data table, creating a first cache table corresponding to the first data table in the cache, and adding the correspondence between the first data table and the first cache table to the mapping relationship.

[0010] In an embodiment of the first aspect of the present application, characteristic information of the first cache table is also stored in the first cache table, and the characteristic information of the first cache table includes the number of transactions from which the statements stored in the first cache table come, the number of data tables used for operations, and the time when the first statement is stored; after storing the first statement in the cache, it also includes: judging whether the characteristic information of the cache meets the preset conditions based on the characteristic information of each cache table in the cache.

[0011] In an embodiment of the first aspect of the present application, after executing the statement stored in the cache, it also includes: deleting the statement stored in the first cache table, and modifying the characteristic information of the first cache table.

[0012] The second aspect of the present application provides a database data synchronization device, including: an acquisition module, used to acquire a log file from a source database; a determination module, used to determine a first statement based on the log file; a cache module, used to store the first statement in a cache, wherein the cache table supports storing multiple statements from different transactions, and multiple statements for operating different data tables; a synchronization module, used to execute the statement stored in the cache when the first statement is stored in the cache and the characteristic information of the cache meets a preset condition, so as to store the data operated by the statement in the cache in the destination database.

[0013] The third aspect of the present application provides an electronic device, comprising: a memory and a processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor executes the method described in any one of the first aspect of the present application.

[0014] A fourth aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, which, when executed, implement the method described in any one of the first aspects of the present application.

[0015] A fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, implements the method described in any one of the first aspects of the present application.

[0016] The data synchronization method, device, equipment, medium and product of the database provided by the present application, after the second synchronization program of the destination database obtains the log file and parses the first statement, the first statement is stored in the cache, and the cache of the destination database supports the storage of statements from different things and statements for operating different data tables. In other words, even if the destination database continuously receives statements from different things and statements for operating different data tables, it will not immediately execute the statements already stored in the cache as in the prior art, but will store all these statements. Ultimately, the destination database will only execute the statements in the cache if the characteristic information of its cache meets the preset conditions, so as to store the data operated by the statements in the cache in the destination database to achieve data synchronization. Therefore, in scenarios such as dense small transactions or a single transaction corresponding to multiple data tables, the second synchronization program can store more statements in the cache first, and after the cache meets the preset conditions, the statements in the cache are merged and executed again, thereby reducing the frequent writing of data to the destination database, thereby improving the synchronization performance of the destination database when performing data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0018] Figure 1 A schematic diagram of a scenario for data synchronization of a database used in this application;

[0019] Figure 2 A schematic diagram of a method for caching statements by a destination database in the prior art;

[0020] Figure 3 A schematic diagram of a flow chart of an embodiment of a method for synchronizing database data provided by the present application;

[0021] Figure 4 A schematic diagram of the cache of the destination database provided by this application;

[0022] Figure 5 A schematic diagram of the cache table of the destination database provided for this application;

[0023] Figure 6 A flowchart of another embodiment of the method for synchronizing database data provided by the present application;

[0024] Figure 7 A schematic diagram of the structure of a data synchronization device for a database provided by the present application;

[0025] Figure 8 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Figure 1 A schematic diagram of a scenario for data synchronization of a database used in this application, such as Figure 1 As shown in the figure, according to the needs of some database services, data synchronization operations are sometimes required between databases to ensure the consistency of data stored in the two databases. Figure 1In the example shown, the database providing data is recorded as the source database 10, and the database receiving data is recorded as the destination database 20. In some synchronization technologies, a first synchronization program 101 of the source end is run on the source database 10, and a second synchronization program 201 of the destination end is run on the destination database 20, so that data is synchronized between the source database 10 and the destination database 20 through the first synchronization program 101 and the second synchronization program 201.

[0029] In one embodiment, the first synchronization program 101 may be run on the source database 10 , or the first synchronization program 101 may also be run on an electronic device connected to the source database 10 .

[0030] In one embodiment, the second synchronization program 201 may be executed on the destination database 20 , or the second synchronization program 201 may also be executed on an electronic device connected to the destination database 20 .

[0031] In a specific synchronization method, data synchronization between the source database 10 and the target database 20 can be divided into the following three stages.

[0032] The first stage: the first synchronization program 101 and the second synchronization program 201 initialize the stock data to obtain the basic point of data synchronization between the source database 10 and the target database 20.

[0033] The second stage: the first synchronization program 101 and the second synchronization program 201 perform incremental data synchronization based on the synchronization base point established by the initialization data loading.

[0034] The third stage: the first synchronization program 101 and the second synchronization program 201 regularly compare and verify the source data and the target data of the data synchronization to confirm that no data is lost during the data synchronization process.

[0035] Among them, the second stage and the third stage will be in a long-term parallel state. In the second stage, when data synchronization is performed between the source database 10 and the destination database 20, the first synchronization program 101 of the source database 10 parses the source database 10 online logs, archived logs and other log files to obtain the addition, deletion and change of data, and then converts these changes into a specific message format inside the synchronization software in units of transactions, and then sends them to the second synchronization program 201 of the destination database 20 through the private transmission protocol of the data synchronization software. Subsequently, the destination database 20 restores the acquired log files into statements supported by the second synchronization program 201 through the second synchronization program 201, and executes them on the second synchronization program 201, and finally realizes the storage of the data of the source database 10 in the destination database 20. The statements supported by the second synchronization program 201 can specifically be SQL statements.

[0036] In some technologies, the second synchronization program 201 reduces the number of writes to the destination database 20 by caching statements to improve the efficiency of data storage in the destination database 20 during data synchronization. Among them, based on the fact that the statements in the log file generally operate on the data of the same data table, the second synchronization program 201 does not immediately execute the multiple statements obtained by parsing the log file, but first stores the statements in the cache separately. Later, when the statement parsed obtains a statement that operates on a different data table than the statement in the cache, the second synchronization program 201 executes the data stored in the cache again to store the data operated by the statement in the cache into the destination database 20, and stores the newly parsed statement in the cache again.

[0037] For example, Figure 2 FIG. 1 is a schematic diagram of a method for caching statements by a destination database in the prior art. Figure 2 As shown, when the statements acquired by the second synchronization program 201 are all used to operate table A, the second synchronization program 201 stores all the statements in the cache. At this time, the statements stored in the cache of the second synchronization program 201 are all used to operate table A, which are recorded as statement 1, statement 2, statement 3... Assuming that the latest statement acquired by the second synchronization program 201 is used to operate table B, the latest statement is different from the table operated by the statement in the cache. Therefore, the second synchronization program 201 first executes all statements 1, statement 2, statement 3... stored in the cache to store the data operated by each statement in the destination database 20. Subsequently, the second synchronization program 201 deletes all the statements in the cache used to operate table A, and stores the latest statement acquired for operating table B in the cache.

[0038] It is understandable that when the second synchronization program 201 stores each statement in the cache, it will compare the statement to be cached with the statement already stored in the cache, and when it is determined that the tables operated are different, the second synchronization program 201 will execute the statement in the cache. The different tables operated between the statements mentioned here include switching of statement table names or switching of INSERT / UPDATE / DELETE of tables.

[0039] In some technologies, the statements provided in the embodiments of the present application may specifically be a data manipulation language (DML), where DML is used for database operations. In the standard SQL language, the three instructions INSERT, UPDATE, and DELETE are the core, representing insertion, update, and deletion, respectively. DML allows users to insert, update, delete, and retrieve data stored in the database without affecting the database schema or the definition of database objects.

[0040] In the above-mentioned prior art, although the second synchronization program 201 achieves the purpose of reducing frequent writing to the destination database 20 by caching statements, in scenarios where small transactions are dense or a single transaction corresponds to multiple data tables, when the statements continuously obtained by the second synchronization program 201 may correspond to different tables or different transactions, the second synchronization program 201 will still frequently execute the statements in the cache, making it impossible to achieve the purpose of setting the cache, resulting in an increase in the number of data writes to the destination database 20, affecting the data synchronization performance of the destination database 20.

[0041] For example, when the log file received by the second synchronization program 201 is parsed and the statements obtained include:

[0042] Transaction 1:

[0043] SQL 0INSERT TABLE A

[0044] SQL 1 INSERT TABLE B

[0045] SQL 2 INSERT TABLE C

[0046] SQL 3 INSERT TABLE D

[0047] Transaction 2:

[0048] SQL 0INSERT TABLE A

[0049] SQL 1 INSERT TABLE B

[0050] SQL 2 INSERT TABLE C

[0051] SQL 3 INSERT TABLE D

[0052] Transaction 3:

[0053] SQL 0INSERT TABLE A

[0054] SQL 1 INSERT TABLE B

[0055] SQL 2 INSERT TABLE C

[0056] SQL 3 INSERT TABLE D

[0057] It can be seen that when the second synchronization program 201 obtains each of the above statements, since each statement is used to operate different data tables, the second synchronization program 201 needs to execute the previous statement stored in the cache and delete the previous statement before storing the current statement in the cache. In this small transaction scenario, the cache set by the second synchronization program 201 does not achieve the purpose of reducing the number of database writes.

[0058] Based on this, the present application provides a method, device, equipment, medium and product for synchronizing data of a database, which is aimed at reducing the frequent writing of data to the destination database 20 in scenarios such as dense small transactions or a single transaction corresponding to multiple data tables, so as to improve the data synchronization performance of the database. The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0059] Figure 3 This is a flow chart of an embodiment of a method for synchronizing database data provided by the present application. Figure 3 The method shown can be applied in Figure 1 In the scenario shown, the second synchronization program 201 of the destination database 20 is executed. Specifically, Figure 3 The data synchronization method of the database shown includes:

[0060] S101 : the second synchronization program 201 obtains a log file from the source database 10 .

[0061] Specifically, during data synchronization, the first synchronization program 101 of the source database 10 can send a log file to the second synchronization program 201 of the target database 20. In S101, the second synchronization program 201 receives the log file sent by the first synchronization program 101.

[0062] S102: The second synchronization program 201 determines a first statement according to the log file obtained in S101.

[0063] Specifically, the second synchronization program 201 parses the log file to obtain at least one statement, and performs subsequent processing on the at least one statement respectively. The first statement mentioned here is the statement currently processed by the second synchronization program 201.

[0064] S103: The second synchronization program 201 stores the first statement in a cache. In the embodiment of the present application, the cache of the second synchronization program 201 supports storing statements from multiple different transactions and multiple statements for operating different data tables.

[0065] Specifically, Figure 4A schematic diagram of the cache of the destination database provided for this application, such as Figure 4 As shown, in an embodiment of the present application, the cache of the second synchronization program 201 can store statements from different things and store statements for operating through data tables through different cache tables, and each cache table can be a storage space in the cache. For example, the second synchronization program 201 stores statements 11, 12, 13, etc. for operating table A through the first cache table, and stores statements 21, 22, 23, etc. for operating table B through the second cache table, and so on. At the same time, the statements stored in each cache table can come from different transactions, for example, statement 11 in the first cache table comes from transaction X, statement 12 comes from transaction X, statement 13 comes from transaction Y, etc., and statement 21 in the second cache table comes from transaction X, statement 22 comes from transaction Y, statement 23 comes from transaction Z, etc., and so on.

[0066] In one embodiment, the second synchronization program 201 determines in S103 a first data table for storing the first statement according to the first statement to be stored, and determines a first cache table in the cache according to the first data table and the mapping relationship, thereby storing the first statement in the first cache table. For example, the second synchronization program 201 determines that the first statement is used to operate the data table A, and then determines according to the data table A and the mapping relationship: Figure 4 The first cache table shown in FIG. 1 is used to store the first statement for operating the data table A in the first cache table. The mapping relationship may include multiple data tables and a corresponding relationship between each data table and a cache table. For example, Figure 4 The mapping relationship may include a corresponding relationship between data table A and the first cache table, and a corresponding relationship between data table B and the second cache table, and so on.

[0067] S104 : After the second synchronization program 201 stores the first statement in the cache in S103 , if the characteristic information of the cache meets the preset condition, the second synchronization program 201 executes the statement stored in the cache to store the data operated by the statement in the cache in the destination database 20 .

[0068] In one embodiment, when storing the cache table, the second synchronization program 201 also stores the characteristic information of the cache table in each cache table. For example, Figure 5 A schematic diagram of the cache table of the destination database provided for this application, such as Figure 5As shown, the first cache table of the second synchronization program 201 stores the characteristic information of the first cache table, the second cache table stores the characteristic information of the second cache table, and so on. Taking the characteristic information of the first cache table as an example, the characteristic information of the first cache table includes the number of transactions from which the statements stored in the first cache table come, the number of data tables used for operation, and the time when the first statement is stored. For example, the transactions from which the statements stored in the first cache table come are transaction X and transaction Y, and the number is 2; the data table used for operation stored in the first cache table is A, and the number is 1.

[0069] In one embodiment, the second synchronization program 201 initializes the pendingTableMap container during the instantiation phase, thereby using the pendingTableMap container as the first cache table. At the same time, the pendingTableMap container stores the currently cached table and cache-related information in the form of key-value pairs, where the key is the name of the table and the value is a data structure used to record cache-related information, called TablePendingStatement. The structure contains three attributes, namely, SQL with unbound values, cached PrepareStatement, and the number of transactions that have been merged for the table (reset to 0 after each cache cleanup and sending). PrepareStatement is an interface in the JDBC API, representing a precompiled SQL statement.

[0070] In one embodiment, when the second synchronization program 201 stores the first statement, if it is determined from the mapping relationship that there is no corresponding cache table for the first data table after determining that the first statement is used to operate, then a first cache table corresponding to the first data table is created in the cache, and the corresponding relationship between the first data table and the first cache table is added to the mapping relationship. Subsequently, the second synchronization program 201 can add the first statement to the first cache table. Specifically, the second synchronization program 201 determines the TablePendingStatement of the first data table currently being processed. If the TablePendingStatement corresponding to the table does not exist, it means that the table is not cached, creates the corresponding TablePendingStatement and stores it in the pendingTableMap container, obtains the PrepareStatement from the cache, binds the parameter values, and caches it through JDBC's addBatch.

[0071] After the second synchronization program 201 stores the first statement in the cache, it determines whether the cache characteristic information meets the preset conditions according to the characteristic information of each cache table. If the preset conditions are met, it obtains the PrepareStatement of the corresponding table and executes the executeBatch of JDBC to send it.

[0072] In one embodiment, the preset conditions for whether the characteristic information of the cache provided by the embodiment of the present application meets include: the number of transactions from which the multiple statements stored in the first cache table come is greater than the first preset value. Among them, this embodiment is for the judgment of a single cache table in the cache. After the second synchronization program 201 stores the first statement in the first cache table, if the number of transactions from which the multiple statements stored in the first cache table come is greater than the first preset value, the second synchronization program 201 can execute all the statements already stored in the first cache table to store the data of all the statement operations already stored in the first cache table into the destination database 20. Subsequently, the second synchronization program 201 can delete the statements stored in the first cache table and modify the characteristic information of the first cache table.

[0073] In another embodiment, the preset conditions for whether the characteristic information of the cache provided by the embodiment of the present application meets include: the time difference between the moment when the first cache table stores the first statement and the moment when the first statement is stored is greater than the second preset value. Among them, this also refers to the judgment of a single cache table in the cache. When the second synchronization program 201 stores the first statement in the first cache table, if the time difference between the moment when the first statement is stored in the first cache table and the moment when the first statement is stored is greater than the second preset value, it means that the data has not been stored in the database for a long time. Therefore, the second synchronization program 201 can execute all the statements already stored in the first cache table to store the data of all the statement operations already stored in the first cache table into the destination database 20. Subsequently, the second synchronization program 201 can delete the statements stored in the first cache table and modify the characteristic information of the first cache table.

[0074] In another embodiment, the preset conditions for whether the characteristic information of the cache provided by the embodiment of the present application meets include: the number of cache tables stored in the cache is greater than a third preset value. Among them, this embodiment is for the judgment of multiple cache tables in the cache. After the second synchronization program 201 stores the first statement in the first cache table, if the number of all cache tables stored in the cache is greater than the third preset value, the second synchronization program 201 will execute all the statements stored in each cache table in the cache respectively, so as to store the data of all statement operations stored in all cache tables into the destination database 20. Subsequently, the second synchronization program 201 can delete the statements stored in all cache tables and modify the characteristic information of all cache tables.

[0075] In one embodiment, the above-mentioned first preset value, second preset value and third preset value may be pre-configured by the second synchronization program 201, or may be specified by a user of the second synchronization program 201. The second synchronization program 201 may provide external configuration parameters and receive the first preset value, second preset value and third preset value specified by the user.

[0076] In one embodiment, the second synchronization program 201 executes multiple statements respectively, and the technology of storing the data operated by multiple statements in the database can also be called batch storage. The second synchronization program 201 can specifically perform batch storage through JDBC. Among them, JDBC is a JAVA API for executing SQL statements, which can provide unified access to multiple databases. It consists of a group of classes and interfaces written in Java language. Its main functions are: establishing a connection with the database, sending the language for operating the database, and processing the results. When JDBC is used for batch storage, JDBC can be used to execute a series of SQL statements. Instead of executing them one by one separately, multiple insert statements are collected and submitted to the database for execution at one time. This method can significantly improve the efficiency of data insertion, especially when dealing with large-scale data insertion scenarios.

[0077] In one embodiment, after the second synchronization program 201 executes the statement stored in the cache in S104, the second synchronization program 201 can delete the statement stored in the first cache table, but retain the first cache table, and modify the characteristic information of the first cache table, thereby avoiding multiple establishment of the first cache table and reducing the amount of calculation. Alternatively, in one embodiment, the second synchronization program 201 can delete the first cache table and create the first cache table when needed later, thereby reducing the invalid occupation of cache space by empty cache tables.

[0078] In summary, in the data synchronization method of the database provided in this embodiment, after the second synchronization program 201 of the destination database 20 obtains the log file and parses the first statement, it does not immediately execute the first statement, but stores the first statement in the cache, and the cache of the second synchronization program 201 supports the storage of statements from different things and statements for operating different data tables. In other words, even if the second synchronization program 201 continuously receives statements from different things and statements for operating different data tables, it will not immediately execute the statements already stored in the cache as in the prior art, but will store all these statements. Finally, the second synchronization program 201 will only execute the statements in the cache if the characteristic information of its cache meets the preset conditions, so as to store the data operated by the statements in the cache in the destination database 20 to achieve data synchronization. Therefore, the second synchronization program 201 can store more statements in the cache first in scenarios such as small transactions are dense or a single transaction corresponds to multiple data tables, and after the cache meets the preset conditions, the statements in the cache are merged and executed again, thereby reducing the frequent writing of data to the destination database 20, thereby improving the synchronization performance of the destination database 20 when performing data synchronization.

[0079] Figure 6 A flowchart of another embodiment of the method for synchronizing database data provided by the present application is shown in FIG. Figure 6 The method shown shows that Figure 3 A specific implementation of the method shown is as follows: Figure 6As shown, the second synchronization program 201 initializes the pendingTableMap container in the instantiation phase, and obtains the next piece of data to be synchronized to obtain the first statement (DML) to be processed. Subsequently, the second synchronization program 201 determines the TablePendingStatement of the first data table currently being processed. If the corresponding TablePendingStatement does not exist for the table, the batch warehousing method of JDBC is used, and the PrepareStatement object is cached and stored in the pendingTableMap container. When data needs to be loaded later, it is determined whether the PrepareStatement of the table has been cached. If it exists in the cache, it is directly taken out and used. If the TablePendingStatement corresponding to the table does not exist, it means that the table is not cached. The corresponding TablePendingStatement is created and stored in the pendingTableMap container, the PrepareStatement is obtained from the cache, the parameter values ​​are bound, and it is cached through JDBC's addBatch. Finally, after storing the first statement in the cache, the second synchronization program 201 determines whether the cache meets the preset conditions through the cached cache table. If the preset conditions are met, the statements in the cache are merged and executed again through JDBC's executeBatch, thereby avoiding the phenomenon of repeatedly creating PrepareStatement objects in scenarios where small transactions are dense and a single transaction may involve multiple tables at the same time, greatly improving the warehousing performance in this scenario.

[0080] In the aforementioned embodiments of the present application, the data synchronization method of the database provided by the embodiments of the present application is introduced. In order to realize the various functions in the data synchronization method of the database provided by the embodiments of the present application, the second synchronization program 201 as the execution subject can be implemented by hardware structure and / or software module, for example, in the form of hardware structure, software module, or hardware structure plus software module to realize the above functions. Whether one of the above functions is executed in the form of hardware structure, software module, or hardware structure plus software module depends on the specific application and design constraints of the technical solution.

[0081] For example, Figure 7 A schematic diagram of the structure of a data synchronization device for a database provided in this application, such as Figure 7 The device shown can be used to execute the database data synchronization method provided in any embodiment of the present application. In one embodiment, Figure 7The data synchronization device 1000 of the database shown includes: an acquisition module 1001, a determination module 1002, a cache module 1003 and a synchronization module 1004. The acquisition module 1001 is used to acquire a log file from a source database; the determination module 1002 is used to determine a first statement according to the log file; the cache module 1003 is used to store the first statement in a cache, wherein the cache table supports storing multiple statements from different transactions and multiple statements for operating different data tables; the synchronization module 1004 is used to execute the statement stored in the cache when the first statement is stored in the cache and the characteristic information of the cache meets the preset conditions, so as to store the data operated by the statement in the cache in the destination database.

[0082] The specific implementation method and principle of the data synchronization device for the above database refer to the description of the data synchronization method for the above database, which will not be repeated here.

[0083] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the processing module can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above-mentioned module is determined. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0084] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0085] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0086] For example, Figure 8 A schematic diagram of the structure of an electronic device provided in this application, such as Figure 8 The device shown can be used to execute the database data synchronization method provided in any embodiment of the present application. In one embodiment, Figure 8The electronic device 2000 shown includes one or more processors 2001 and a memory 2002; wherein the memory 2002 is used to store computer executable instructions, and the processor 2001 can execute the computer executable instructions stored in the memory 2002. When the computer executable instructions are executed by the processor 2001, the processor 2001 implements the data synchronization method of any database in the aforementioned embodiments of the present application.

[0087] In one embodiment, if Figure 8 The electronic device 2000 shown further includes a communication interface 2003 , wherein the processor 2001 can communicate with other devices via the communication interface 2003 , for example, the processor 2001 obtains a log file via the communication interface 2003 .

[0088] The present application also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed, they can be used to implement a data synchronization method for any database in the aforementioned embodiments of the present application.

[0089] An embodiment of the present application also provides a chip for executing instructions, wherein the chip is used to execute any of the database data synchronization methods described above in the present application.

[0090] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements the data synchronization method of any database as described above in the present application.

[0091] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for synchronizing data in a database, characterized in that: include: Get the log files from the source database; determining a first statement according to the log file; Storing the first statement in a cache, wherein the cache table supports storing multiple statements from different transactions and multiple statements for operating different data tables; After the first statement is stored in the cache, the characteristic information of the cache meets the preset condition, and the statement stored in the cache is executed to store the data operated by the statement in the cache into the destination database.

2. The method according to claim 1, characterized in that The step of storing the first statement in a cache includes: Determine a first data table for operating the first statement; Determine a first cache table in the cache according to the first data table and the mapping relationship, wherein the mapping relationship includes a plurality of data tables and a corresponding relationship between each data table and a cache table; The first statement is stored in the first cache table.

3. The method according to claim 2, characterized in that The preset conditions include: The number of transactions from which the multiple statements stored in the first cache table respectively come is greater than a first preset value; Alternatively, the time difference between the time when the first cache table stores the first statement and the time when the first statement is stored is greater than a second preset value; Alternatively, the number of cache tables stored in the cache is greater than a third preset value.

4. The method according to claim 2 or 3, characterized in that: After determining the first data table for operation by the first statement, the method further includes: When the first data table does not have a corresponding cache table, a first cache table corresponding to the first data table is created in the cache, and a corresponding relationship between the first data table and the first cache table is added to the mapping relationship.

5. The method according to claim 2 or 3, characterized in that: The first cache table also stores characteristic information of the first cache table, the characteristic information of the first cache table including the number of transactions from which the statements stored in the first cache table come, the number of data tables used for operation, and the time when the first statement is stored; After storing the first statement into the cache, the method further includes: According to the characteristic information of each cache table in the cache, it is determined whether the characteristic information of the cache meets the preset condition.

6. The method according to claim 5, characterized in that After executing the statement stored in the cache, the method further includes: The statements stored in the first cache table are deleted, and the characteristic information of the first cache table is modified.

7. A data synchronization device for a database, characterized in that: include: The acquisition module is used to obtain log files from the source database; A determination module, configured to determine a first statement according to the log file; A cache module, used to store the first statement into a cache, wherein the cache table supports storing multiple statements from different transactions and multiple statements for operating different data tables; The synchronization module is used to execute the statement stored in the cache when the first statement is stored in the cache and the characteristic information of the cache meets the preset conditions, so as to store the data operated by the statement in the cache into the destination database.

8. An electronic device, characterized in that: include: Memory and processor; The memory stores computer executable instructions; The processor executes the computer executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and when the computer executable instructions are executed, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed.

Images