Data synchronization method, system, electronic device and storage medium between databases

Abstract

The present invention relates to the technical field of data synchronization, and discloses a method, system, electronic device and storage medium for synchronizing data between databases. The method obtains synchronization subscription information of an original database, determines a target database of each original database according to a subscription target identifier in the synchronization subscription information, monitors a binary log corresponding to the target database according to the synchronization data granularity in the synchronization subscription information, and then obtains incremental data of the target database, synchronizes data of the original database through the incremental data, determines the original database and its corresponding target database based on the synchronization subscription information corresponding to a system database, and then collects incremental data of the target database according to the binary log, which not only avoids redeveloping database statements for data synchronization, but also saves a transmission link between an interface and each original database, thereby reducing the cost of data synchronization between databases.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a method, system, electronic device and storage medium for synchronizing data between databases. Background Art

[0002] Currently, enterprises store business system reports in databases corresponding to different business systems. In order to reduce the coupling and access pressure between business systems and their databases, common basic detailed data is stored in each database for data dimension screening and business processing.

[0003] In response to each update of basic detailed data, each database needs to synchronize its own basic detailed data accordingly. However, due to the continuous development of the enterprise, the topology of the enterprise network is complex and the database size is huge. Providing data synchronization services to databases one by one in the form of interfaces not only requires additional development costs, but also lengthens the data transmission link, increasing the data synchronization cost between databases. Summary of the invention

[0004] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0005] In view of the above-mentioned shortcomings of the prior art, the present invention discloses a method, system, electronic device and storage medium for synchronizing data between databases, so as to reduce the cost of data synchronization between databases.

[0006] The present invention discloses a method for data synchronization between databases, comprising: obtaining multiple system databases, and obtaining synchronization subscription information corresponding to at least a part of the system databases, wherein the synchronization subscription information includes a subscription target identifier, a synchronization data granularity and a data synchronization mode; determining any synchronization subscription information as target information, determining the system database corresponding to the target information as an original database, matching each of the system databases according to the subscription target identifier of the target information, and obtaining a target database corresponding to the original database; determining the synchronization data granularity of the target information as a target granularity, monitoring a binary log corresponding to the target database according to the target granularity, and if an incremental log corresponding to the target granularity is generated in the binary log, extracting data from the target database according to the incremental log to obtain incremental data; pushing the incremental data to the original database according to the data synchronization mode of the target information, so that the original database performs data synchronization according to the incremental data.

[0007] Optionally, the binary log corresponding to the target database is monitored according to the target granularity, including: the system database includes at least one database table, and the synchronization data granularity includes table data or library data; if the target granularity includes library data, a first monitoring interface corresponding to the target database is established, the binary log corresponding to the target database is monitored using the first monitoring interface, and the incremental log corresponding to the target database is determined from the binary log corresponding to the target database; if the target granularity includes table data, a range identifier corresponding to the target granularity is obtained, a target table corresponding to the target granularity is determined from a database table of the target database according to the range identifier, a second monitoring interface corresponding to the target database is established according to the target table, the binary log corresponding to the target database is monitored using the second monitoring interface, and the incremental log corresponding to the target table is determined from the binary log corresponding to the target database.

[0008] Optionally, the incremental data is pushed to the original database according to the data synchronization method of the target information, including: if the data synchronization method includes interface request synchronization, adding a synchronization push interface to the original database according to preset interface information, wherein the preset interface information includes an interface address and a request method; encoding the incremental data according to the request method to obtain a synchronization request; and sending the synchronization request to the synchronization push interface of the original database through the interface address.

[0009] Optionally, the incremental data is pushed to the original database according to the data synchronization method of the target information, including: if the data synchronization method includes message queue synchronization, obtaining a message server; sending the incremental data to the message server, so that the message server sends the incremental data to the original database in the form of a message queue when a first preset condition or a second preset condition is met, wherein the first preset condition includes receiving the incremental data, and the second preset condition includes receiving a synchronization request sent by the original database.

[0010] Optionally, the incremental data is pushed to the original database according to the data synchronization method of the target information, including: if the data synchronization method includes database synchronization, obtaining the database type of the original database, and matching the synchronization statement template corresponding to the database type from a preset instruction template according to the database type; adding the incremental data to the synchronization statement template to obtain the data synchronization statement corresponding to the incremental data; pushing the data synchronization statement to the original database so that the original database executes the data synchronization statement.

[0011] Optionally, after pushing the incremental data to the original database according to the data synchronization method of the target information, the method further includes: counting the number of failure results in response to the synchronization failure result of the original database; if the number of failure results is greater than a preset failure threshold, pushing the incremental data to the original database again according to the data synchronization method of the target information.

[0012] Optionally, before pushing the incremental data to the original database according to the data synchronization method of the target information, the method also includes: generating a synchronization task according to the data synchronization method, the incremental data, and the original database, and adding the synchronization task to a preset synchronization task queue, wherein the synchronization task includes pushing the incremental data to the original database according to the data synchronization method of the target information; monitoring the synchronization task queue; if the synchronization task queue contains a synchronization task corresponding to the original database, generating a task execution thread corresponding to the original database, wherein the task execution thread is used to execute the synchronization tasks corresponding to the original database in sequence according to the joining time; if the synchronization task queue contains no synchronization task corresponding to the original database, clearing the task execution thread corresponding to the original database.

[0013] The present invention discloses a data synchronization system between databases, comprising: an acquisition module, used for acquiring multiple system databases, and acquiring synchronization subscription information corresponding to at least a part of the system databases, wherein the synchronization subscription information includes a subscription target identifier, a synchronization data granularity and a data synchronization mode; a matching module, used for determining any synchronization subscription information as target information, determining the system database corresponding to the target information as an original database, matching each of the system databases according to the subscription target identifier of the target information, and obtaining a target database corresponding to the original database; an extraction module, used for determining the synchronization data granularity of the target information as a target granularity, monitoring a binary log corresponding to the target database according to the target granularity, and if an incremental log corresponding to the target granularity is generated in the binary log, extracting data from the target database according to the incremental log to obtain incremental data; a synchronization module, used for pushing the incremental data to the original database according to the data synchronization mode of the target information, so that the original database performs data synchronization according to the incremental data.

[0014] The present invention discloses an electronic device, comprising: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above method.

[0015] The present invention discloses a computer-readable storage medium, on which a computer program is stored: when the computer program is executed by a processor, the above method is implemented.

[0016] Beneficial effects of the present invention:

[0017] By obtaining the synchronization subscription information of the original database, the target database of each original database is determined according to the subscription target identifier in the synchronization subscription information, and the binary log corresponding to the target database is monitored according to the synchronization data granularity in the synchronization subscription information, and then the incremental data of the target database is obtained, and the original database is synchronized through the incremental data. In this way, the original database and its corresponding target database are determined based on the synchronization subscription information corresponding to the system database, and the incremental data of the target database is collected according to the binary log, which not only avoids the need to redevelop database statements for data synchronization, but also saves the transmission link between the interface and each original database, thereby reducing the cost of data synchronization between databases.

[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0020] Figure 1 is a flow chart of a method for synchronizing data between databases in an embodiment of the present invention;

[0021] Figure 2-a is a schematic diagram of a method for performing data synchronization through an interface in an embodiment of the present invention;

[0022] Figure 2-b is a schematic diagram of a method for performing data synchronization in a subscription form in an embodiment of the present invention;

[0023] Figure 3 is a structural diagram of a data synchronization system implementing a data synchronization method in an embodiment of the present invention;

[0024] Figure 4 is a schematic diagram of the structure of a data synchronization system between databases in an embodiment of the present invention;

[0025] Figure 5 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and sub-samples in the embodiments can be combined with each other without conflict.

[0027] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0028] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0029] The terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the embodiments of the present disclosure described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions.

[0030] Unless otherwise stated, the term "plurality" means two or more.

[0031] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B indicates: A or B.

[0032] The term "and / or" is a description of the association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B.

[0033] Combination Figure 1 As shown, the embodiment of the present disclosure provides a method for synchronizing data between databases, including:

[0034] Step S101, obtaining multiple system databases, and obtaining synchronization subscription information corresponding to at least a part of the system databases;

[0035] The synchronization subscription information includes the subscription target identifier, synchronization data granularity and data synchronization mode;

[0036] Step S102, determining any synchronization subscription information as target information, determining the system database corresponding to the target information as the original database, matching each system database according to the subscription target identifier of the target information, and obtaining the target database corresponding to the original database;

[0037] Step S103, determining the synchronization data granularity of the target information as the target granularity, monitoring the binary log corresponding to the target database according to the target granularity, and if an incremental log corresponding to the target granularity is generated in the binary log, extracting data from the target database according to the incremental log to obtain incremental data;

[0038] Step S104: Push the incremental data to the original database according to the data synchronization method of the target information, so that the original database performs data synchronization according to the incremental data.

[0039] The method for data synchronization between databases provided by the embodiment of the present disclosure is adopted, by obtaining the synchronization subscription information of the original database, determining the target database of each original database according to the subscription target identifier in the synchronization subscription information, monitoring the binary log corresponding to the target database according to the synchronization data granularity in the synchronization subscription information, and then obtaining the incremental data of the target database, and synchronizing the data of the original database through the incremental data. In this way, the original database and its corresponding target database are determined based on the synchronization subscription information corresponding to the system database, and then the incremental data of the target database is collected according to the binary log, which not only avoids the need to redevelop database statements for data synchronization, but also saves the transmission link between the interface and each original database, thereby reducing the cost of data synchronization between databases.

[0040] Combination Figure 2-a As shown, an embodiment of the present disclosure provides a method for synchronizing data through an interface, including a synchronization interface X and a system database, wherein the system database includes database A, database B, database C, database D, and database E; connection links are established between the synchronization interface X and database A, database B, database C, database D, and database E, respectively, and synchronization data is sent to database A, database B, database C, database D, and database E according to the connection links, so that database A, database B, database C, database D, and database E are synchronized according to the synchronization data.

[0041] Combination Figure 2-bAs shown, an embodiment of the present disclosure provides a method for synchronizing data in the form of subscription, including a system database, wherein the system database includes database A, database B, database C, database D and database E; obtaining synchronization subscription information corresponding to each system database, and determining that database A is the target database of database B, database B is the target database of database C and database D, and database D is the target database of database E according to the synchronization subscription information; monitoring the binary log of database A to obtain incremental data of database A; sending the incremental data of database A to database B for synchronization; monitoring the binary file of database B to obtain incremental data of database B; sending the incremental data of database B to database C and database D respectively for synchronization; monitoring the binary file of database D to obtain incremental data of database D; sending the incremental data of database D to database E for synchronization.

[0042] In this way, compared with data synchronization through an interface, there is no need to establish a connection link between the interface and each original database, thereby saving the transmission link between the interface and each original database and reducing the cost of data synchronization between databases.

[0043] Optionally, the binary log includes a binlog (binnary log) log; a binlog log is a binary format file used to record SQL (Structured Query Language) statement information about a user's update of a database. For example, SQL statements for changing database tables and changing content will be written to the binlog, but queries on library tables and other content will not be recorded; by default, the binlog log is in binary format and cannot be viewed using commands of text viewing tools (such as cat, vi, etc.), but can be viewed using MySQL binlog parsing.

[0044] Optionally, the binary log corresponding to the target database is monitored according to the target granularity, including: the system database includes at least one database table, and the synchronization data granularity includes table data or library data; if the target granularity includes library data, a first monitoring interface corresponding to the target database is established, the binary log corresponding to the target database is monitored by using the first monitoring interface, and the incremental log corresponding to the target database is determined from the binary log corresponding to the target database; if the target granularity includes table data, a range identifier corresponding to the target granularity is obtained, a target table corresponding to the target granularity is determined from a database table of the target database according to the range identifier, a second monitoring interface corresponding to the target database is established according to the target table, the binary log corresponding to the target database is monitored by using the second monitoring interface, and the incremental log corresponding to the target table is determined from the binary log corresponding to the target database.

[0045] In this way, by monitoring the binary log of the target database at different synchronization data granularities, the user's needs for the monitored object can be met, the monitoring flexibility can be improved, and the monitoring computing cost can be reduced.

[0046] In some embodiments, the business system subscribes to the target database required by the original database according to its own business needs. When the table of the target database changes, the original database and data synchronization method of the subscribed target database are queried; the original server and the corresponding target server are configured by subscribing to the target identifier, and the database name, table name, binlog read history file record and offset record file of the target server are configured; for the incremental data to be synchronized, the synchronization data granularity is divided into table data or library data according to the capture granularity level required by the user; if the table level granularity is monitored, only the data changes of the specified table of the specified database under the data connection are monitored, and the changes of other tables under the database will not be monitored; if one or more specific libraries are monitored, changes in the data of each table in the library will be monitored and processed.

[0047] Optionally, the data synchronization method includes one or more of interface request synchronization, message queue synchronization, database synchronization, etc.

[0048] Optionally, the incremental data is pushed to the original database according to the data synchronization method of the target information, including: if the data synchronization method includes interface request synchronization, adding a synchronization push interface to the original database according to preset interface information, wherein the preset interface information includes an interface address and a request method; encoding the incremental data according to the request method to obtain a synchronization request; and sending the synchronization request to the synchronization push interface of the original database through the interface address.

[0049] Optionally, the incremental data is pushed to the original database according to the data synchronization method of the target information, including: if the data synchronization method includes message queue synchronization, obtaining a message server; sending the incremental data to the message server, so that the message server sends the incremental data to the original database in the form of a message queue when the first preset condition or the second preset condition is met, wherein the first preset condition includes receiving the incremental data, and the second preset condition includes receiving a synchronization request sent by the original database.

[0050] Optionally, the incremental data is pushed to the original database according to the data synchronization method of the target information, including: if the data synchronization method includes database synchronization, obtaining the database type of the original database, and matching the synchronization statement template corresponding to the database type from the preset instruction template according to the database type; adding the incremental data to the synchronization statement template to obtain the data synchronization statement corresponding to the incremental data; pushing the data synchronization statement to the original database so that the original database executes the data synchronization statement.

[0051] In this way, multiple data synchronization modes are set, so that incremental data can be pushed to the original database using different data synchronization modes according to user needs, thereby improving the flexibility of data synchronization.

[0052] In some embodiments, when incremental data is obtained, metadata corresponding to the data synchronization method is extracted from a preset subscription configuration table according to the data synchronization method; if the push method is an http (Hyper Text Transfer Protocol) interface request method, the interface address, request method and data are constructed from the metadata to push the data to the specified interface address; if the push method is parsed to be an MQ (Message Queue) method, the topic (i.e., the specified MQ server) is obtained from the metadata, and the data is pushed directly to the MQ server. Downstream businesses can obtain corresponding change data messages by pulling or pushing; if the consumption method is queried to be database warehousing, the corresponding database connection and database information are parsed from the metadata, and the data source is loaded into the context when the service starts. Depending on whether the change data is a create or update operation, the corresponding SQL statement is concatenated, and finally the statement is executed to save the data directly to the target database

[0053] Optionally, after pushing the incremental data to the original database according to the data synchronization method of the target information, the method also includes: counting the number of failure results in response to the synchronization failure result of the original database; if the number of failure results is greater than a preset failure threshold, pushing the incremental data to the original database again according to the data synchronization method of the target information.

[0054] Optionally, if the number of failed results is greater than a preset warning threshold, a synchronization failure reminder is sent to the client to notify relevant personnel to handle it.

[0055] In this way, a retry mechanism is set up for resynchronization and failure reminder to ensure that the original database successfully synchronizes incremental data and improves synchronization reliability.

[0056] Optionally, before pushing the incremental data to the original database according to the data synchronization method of the target information, the method also includes: generating a synchronization task according to the data synchronization method, the incremental data, and the original database, and adding the synchronization task to a preset synchronization task queue, wherein the synchronization task includes pushing the incremental data to the original database according to the data synchronization method of the target information; monitoring the synchronization task queue; if there is a synchronization task corresponding to the original database in the synchronization task queue, generating a task execution thread corresponding to the original database, wherein the task execution thread is used to execute the synchronization tasks corresponding to the original database in sequence according to the joining time; if there is no synchronization task corresponding to the original database in the synchronization task queue, clearing the task execution thread corresponding to the original database.

[0057] In this way, a corresponding task execution thread is configured for each original database, and the synchronization tasks corresponding to the original database are executed in sequence, avoiding the crash of the original database due to insufficient computing power to handle too many synchronization tasks, and clearing the task execution thread after all synchronization tasks are completed, thereby saving computing power costs.

[0058] Combination Figure 3 As shown, the embodiment of the present disclosure provides a data synchronization system for implementing the above-mentioned data synchronization method, including a target database 301, a data synchronization service 302, a background management service 303, a data push service 304, and an original database 305, wherein the target database 301 and the original database 305 are determined in the following manner: by obtaining multiple system databases and obtaining synchronization subscription information corresponding to at least a part of the system databases, wherein the synchronization subscription information includes a subscription target identifier, a synchronization data granularity, and a data synchronization method; determining any synchronization subscription information as the target information, determining the system database corresponding to the target information as the original database, matching each system database according to the subscription target identifier of the target information, and obtaining the target database corresponding to the original database; the data synchronization service 302 is used to The synchronization data granularity of the target information is determined as the target granularity, and the binary log corresponding to the target database is monitored according to the target granularity. If an incremental log corresponding to the target granularity is generated in the binary log, data is extracted from the target database according to the incremental log to obtain incremental data; the background management service 303 is used to respond to the synchronization failure result of the original database and count the number of failure results; if the number of failure results is greater than the preset failure threshold, the incremental data is pushed to the original database again according to the data synchronization method of the target information; the data push service 304 is used to push the incremental data to the original database according to the data synchronization method of the target information, so that the original database performs data synchronization according to the incremental data, wherein the data synchronization method includes one or more of interface request synchronization, message queue synchronization, database synchronization, etc.

[0059] The method for synchronizing data between databases provided by the embodiment of the present disclosure obtains synchronization subscription information of the original database, determines the target database of each original database according to the subscription target identifier in the synchronization subscription information, monitors the binary log corresponding to the target database according to the synchronization data granularity in the synchronization subscription information, and then obtains incremental data of the target database. The incremental data is used to synchronize data with the original database, which has the following advantages:

[0060] First, the original database and its corresponding target database are determined based on the synchronization subscription information corresponding to the system database, and then the incremental data of the target database is collected according to the binary log. This not only avoids the need to redevelop database statements for data synchronization, but also saves the transmission link between the interface and each original database, thereby reducing the cost of data synchronization between databases.

[0061] Second, compared with data synchronization through interfaces, there is no need to establish a connection link between the interface and each original database, thus saving the transmission link between the interface and each original database and reducing the cost of data synchronization between databases;

[0062] Third, by monitoring the binary logs of the target database at different synchronization data granularities, users’ needs for monitoring objects are met, monitoring flexibility is improved, and monitoring computing costs are reduced;

[0063] Fourth, set up multiple data synchronization methods, so that incremental data can be pushed to the original database using different data synchronization methods according to user needs, improving the flexibility of data synchronization;

[0064] Fifth, set up a retry mechanism for resynchronization and failure reminders to ensure that the original database successfully synchronizes incremental data and improves synchronization reliability;

[0065] Sixth, configure a corresponding task execution thread for each original database, and execute the synchronization tasks corresponding to the original database in sequence to avoid the original database crashing due to insufficient computing power to handle too many synchronization tasks, and clear the task execution thread after all synchronization tasks are completed, thereby saving computing power costs.

[0066] Combination Figure 4As shown, the embodiment of the present disclosure provides a data synchronization system between databases, including an acquisition module 401, a matching module 402, an extraction module 403 and a synchronization module 404. The acquisition module 401 is used to acquire multiple system databases, and acquire synchronization subscription information corresponding to at least a part of the system databases, wherein the synchronization subscription information includes a subscription target identifier, a synchronization data granularity and a data synchronization mode; the matching module 402 is used to determine any synchronization subscription information as the target information, determine the system database corresponding to the target information as the original database, match each system database according to the subscription target identifier of the target information, and obtain the target database corresponding to the original database; the extraction module 403 is used to determine the synchronization data granularity of the target information as the target granularity, monitor the binary log corresponding to the target database according to the target granularity, and if an incremental log corresponding to the target granularity is generated in the binary log, extract data from the target database according to the incremental log to obtain incremental data; the synchronization module 404 is used to push the incremental data to the original database according to the data synchronization mode of the target information, so that the original database performs data synchronization according to the incremental data.

[0067] The data synchronization system between databases provided by the embodiment of the present disclosure is adopted, by obtaining the synchronization subscription information of the original database, determining the target database of each original database according to the subscription target identifier in the synchronization subscription information, monitoring the binary log corresponding to the target database according to the synchronization data granularity in the synchronization subscription information, and then obtaining the incremental data of the target database, and synchronizing the original database through the incremental data. In this way, the original database and its corresponding target database are determined based on the synchronization subscription information corresponding to the system database, and then the incremental data of the target database is collected according to the binary log, which not only avoids the need to redevelop database statements for data synchronization, but also saves the transmission link between the interface and each original database, thereby reducing the cost of data synchronization between databases.

[0068] Figure 5 The structure diagram of the computer system suitable for implementing the electronic device of the embodiment of the present application is shown. It should be noted that: Figure 5 The computer system 500 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0069] like Figure 5As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503, such as executing the method in the above embodiment. In the RAM 503, various programs and data required for system operation are also stored. The CPU 501, the ROM 502 and the RAM 503 are connected to each other through the bus 504. The input / output (I / O) interface 505 is also connected to the bus 504.

[0070] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed so that a computer program read therefrom is installed into the storage section 508 as needed.

[0071] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 509, and / or installed from a removable medium 511. When the computer program is executed by a central processing unit (CPU) 501, various functions defined in the system of the present application are executed.

[0072] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal can take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. A computer program contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0073] The embodiments of the present disclosure further provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods in the embodiments is implemented.

[0074] The computer-readable storage medium in the embodiments of the present disclosure can be understood by ordinary technicians in the field: all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the execution includes the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0075] The electronic device disclosed in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run the computer program so that the electronic device executes each step of the above method.

[0076] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0077] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0078] The above description and accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. Parts and subsamples of some embodiments may be included in or replace parts and subsamples of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of a stated sub-sample, whole, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other sub-samples, wholes, steps, operations, elements, components and / or groups of these. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0079] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0080] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some sub-samples can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0081] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for synchronizing data between databases, characterized in that: include: Acquire multiple system databases, and acquire synchronization subscription information corresponding to at least a portion of the system databases, wherein the synchronization subscription information includes a subscription target identifier, synchronization data granularity, and data synchronization mode; Determine any synchronization subscription information as target information, determine the system database corresponding to the target information as the original database, match each of the system databases according to the subscription target identifier of the target information, and obtain the target database corresponding to the original database; Determine the synchronization data granularity of the target information as the target granularity, monitor the binary log corresponding to the target database according to the target granularity, and if an incremental log corresponding to the target granularity is generated in the binary log, extract data from the target database according to the incremental log to obtain incremental data; Pushing the incremental data to the original database according to the data synchronization method of the target information, so that the original database performs data synchronization according to the incremental data; Monitoring the binary log corresponding to the target database according to the target granularity, including: the system database includes at least one database table, and the synchronization data granularity includes table data or library data; if the target granularity includes library data, establishing a first monitoring interface corresponding to the target database, using the first monitoring interface to monitor the binary log corresponding to the target database, and determining the incremental log corresponding to the target database from the binary log corresponding to the target database; if the target granularity includes table data, obtaining a range identifier corresponding to the target granularity, determining the target table corresponding to the target granularity from the database table of the target database according to the range identifier, establishing a second monitoring interface corresponding to the target database according to the target table, using the second monitoring interface to monitor the binary log corresponding to the target database, and determining the incremental log corresponding to the target table from the binary log corresponding to the target database; The incremental data is pushed to the original database according to the data synchronization method of the target information, including, if the data synchronization method includes message queue synchronization, obtaining a message server; sending the incremental data to the message server, so that the message server sends the incremental data to the original database in the form of a message queue when a first preset condition or a second preset condition is met, wherein the first preset condition includes receiving the incremental data, and the second preset condition includes receiving a synchronization request sent by the original database.

2. The method according to claim 1, characterized in that Pushing the incremental data to the original database according to the data synchronization method of the target information includes: If the data synchronization method includes interface request synchronization, adding a synchronization push interface to the original database according to preset interface information, wherein the preset interface information includes an interface address and a request method; Encoding the incremental data according to the request mode to obtain a synchronization request; The synchronization request is sent to the synchronization push interface of the original database through the interface address.

3. The method according to claim 1, characterized in that Pushing the incremental data to the original database according to the data synchronization method of the target information includes: If the data synchronization method includes database synchronization, obtaining the database type of the original database, and matching a synchronization statement template corresponding to the database type from a preset instruction template according to the database type; Adding the incremental data to the synchronization statement template to obtain a data synchronization statement corresponding to the incremental data; The data synchronization statement is pushed to the original database, so that the original database executes the data synchronization statement.

4. The method according to claim 1, characterized in that: After pushing the incremental data to the original database according to the data synchronization mode of the target information, the method further includes: In response to the synchronization failure result of the original database, counting the number of failure results; If the number of failure results is greater than a preset failure threshold, the incremental data is pushed to the original database again according to the data synchronization method of the target information.

5. The method according to any one of claims 1 to 4, characterized in that: Before pushing the incremental data to the original database according to the data synchronization mode of the target information, the method further includes: Generate a synchronization task according to the data synchronization method, the incremental data, and the original database, and add the synchronization task to a preset synchronization task queue, wherein the synchronization task includes pushing the incremental data to the original database according to the data synchronization method of the target information; Monitoring the synchronization task queue; If the synchronization task queue has a synchronization task corresponding to the original database, a task execution thread corresponding to the original database is generated, wherein the task execution thread is used to execute the synchronization tasks corresponding to the original database in sequence according to the joining time; If the synchronization task queue does not contain the synchronization task corresponding to the original database, the task execution thread corresponding to the original database is cleared.

6. A data synchronization system between databases, characterized in that: include: An acquisition module, used to acquire multiple system databases, and acquire synchronization subscription information corresponding to at least a part of the system databases, wherein the synchronization subscription information includes a subscription target identifier, synchronization data granularity, and data synchronization mode; A matching module, used to determine any synchronization subscription information as target information, determine the system database corresponding to the target information as the original database, match each of the system databases according to the subscription target identifier of the target information, and obtain the target database corresponding to the original database; An extraction module is used to determine the synchronization data granularity of the target information as the target granularity, monitor the binary log corresponding to the target database according to the target granularity, and if an incremental log corresponding to the target granularity is generated in the binary log, extract data from the target database according to the incremental log to obtain incremental data; A synchronization module, configured to push the incremental data to the original database according to the data synchronization mode of the target information, so that the original database performs data synchronization according to the incremental data; The extraction module monitors the binary log corresponding to the target database according to the target granularity in the following manner, the system database includes at least one database table, and the synchronization data granularity includes table data or library data; if the target granularity includes library data, a first monitoring interface corresponding to the target database is established, the binary log corresponding to the target database is monitored by using the first monitoring interface, and the incremental log corresponding to the target database is determined from the binary log corresponding to the target database; if the target granularity includes table data, a range identifier corresponding to the target granularity is obtained, the target table corresponding to the target granularity is determined from the database table of the target database according to the range identifier, a second monitoring interface corresponding to the target database is established according to the target table, the binary log corresponding to the target database is monitored by using the second monitoring interface, and the incremental log corresponding to the target table is determined from the binary log corresponding to the target database; The synchronization module pushes the incremental data to the original database according to the data synchronization method of the target information in the following manner: if the data synchronization method includes message queue synchronization, a message server is obtained; the incremental data is sent to the message server, so that the message server sends the incremental data to the original database in the form of a message queue when a first preset condition or a second preset condition is met, wherein the first preset condition includes receiving the incremental data, and the second preset condition includes receiving a synchronization request sent by the original database.

7. An electronic device, characterized in that: include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images