Database synchronization method, system, device, electronic device and medium

Abstract

The present disclosure provides a database synchronization method, which belongs to the field of database technology. The method includes: in response to receiving a user request for data operation from any one of M databases, obtaining the user request; based on the user request, obtaining a data synchronization message, wherein the data synchronization message includes identification information for indicating the data operated by the user request, and direction information for indicating the synchronization direction of the data between databases; pushing the data synchronization message to a distributed message queue; subscribing to the data synchronization message from the distributed message queue; and after subscribing to the data synchronization message, obtaining the complete data indicated by the identification information from the source database indicated by the direction information, and writing the complete data to the target database indicated by the direction information. The present disclosure also provides a database synchronization system and device, an electronic device, and a computer-readable storage medium.

Description

Technical Field

[0001] The present disclosure belongs to the field of database technology, and more specifically, relates to a database synchronization method, system, device, electronic device and medium. Background Art

[0002] In many scenarios such as when the system needs to replace the database for technological transformation, two or more clusters are usually run in parallel. Among them, quasi-real-time data synchronization is required between the databases of different clusters to ensure that the data between users on the two or more clusters can interact with each other during the parallel period, and when users switch from one cluster to another, business continuity can be guaranteed.

[0003] In related technologies, various databases basically have their own data synchronization mechanisms, but they generally rely on the core technology of the database itself and are limited to synchronization between homogeneous databases or a small number of databases. When the database of the cluster that needs data synchronization is a heterogeneous database, the source database often needs to export the data to be synchronized into a file, and then use certain external tools to transfer the file to the server where the target database is located for processing and import. The whole process has problems such as low processing efficiency and difficulty in handling the relationship between multiple tables. Summary of the invention

[0004] In view of this, embodiments of the present disclosure provide a database synchronization method, system and device, an electronic device, and a computer-readable storage medium.

[0005] One aspect of an embodiment of the present disclosure provides a database synchronization method. The method includes: in response to receiving a user request for data operation from any one of M databases, obtaining the user request, where M is an integer greater than or equal to 1; based on the user request, obtaining a data synchronization message, wherein the data synchronization message includes identification information for indicating the data operated by the user request, and direction information for indicating the synchronization direction of the data between databases; pushing the data synchronization message to a distributed message queue; subscribing to the data synchronization message from the distributed message queue; and after subscribing to the data synchronization message, obtaining the complete data indicated by the identification information from the source database indicated by the direction information, and writing the complete data to the target database indicated by the direction information.

[0006] According to an embodiment of the present disclosure, in response to any one of the M databases receiving a user request for data operation, obtaining the user request includes: in response to receiving the user request, identifying a transaction name in the user request, the transaction name being used to indicate a business function performed by the user request; and obtaining the user request when the transaction name belongs to a preset transaction name set, wherein the transaction name set is a preset set of transaction names that require data synchronization.

[0007] According to an embodiment of the present disclosure, the data synchronization message also includes the transaction name. Wherein, obtaining the data synchronization message based on the user request includes: extracting the transaction name and the identification information from the user request; obtaining the direction information according to the data synchronization configuration of the database operated by the user request; and encapsulating the transaction name, the identification information, and the direction information to obtain the data synchronization message.

[0008] According to an embodiment of the present disclosure, obtaining the complete data indicated by the identification information from the source database indicated by the direction information includes: obtaining information of N data tables involved in the transaction indicated by the transaction name; querying data from the N data tables in the source database to obtain the complete data, specifically including: using a query statement that obtains data manipulation results from each of the data tables in the source database, obtaining the data indicated by the identification information from each of the data tables in the source database, and obtaining local data corresponding to each of the data tables, wherein N local data corresponding to the N data tables are obtained; and merging the N local data to obtain the complete data.

[0009] According to an embodiment of the present disclosure, the writing of the complete data into the target database indicated by the direction information includes: querying data from the N data tables in the target database to obtain verification data; comparing the complete data and the verification data to obtain a comparison result; and when the comparison result indicates that the complete data does not violate the database constraints of the target database, writing the complete data into the target database. Wherein, querying data from the N data tables in the target database to obtain verification data includes: using a query statement that obtains data manipulation results from each data table in the target database, obtaining the data indicated by the identification information from each data table in the target database, and obtaining local verification data corresponding to each data table, wherein N local verification data are obtained corresponding to N data tables; and merging the N local verification data to obtain the verification data.

[0010] According to an embodiment of the present disclosure, writing the complete data into the target database indicated by the direction information includes: assembling the replay execution statement for data synchronization in the target database according to all field data in the complete data to form a data manipulation statement; and submitting the data manipulation statement to the target database for execution.

[0011] According to an embodiment of the present disclosure, when N is greater than 1, data is queried from S data tables concurrently, where S is an integer and 2≤S≤N.

[0012] According to an embodiment of the present disclosure, the method also includes: pre-setting information of N data tables involved in the transaction indicated by the transaction name; pre-setting query statements for obtaining data manipulation results from each of the data tables in the source database and the target database respectively; and pre-setting a replay execution statement of the target database, wherein the replay execution statement is used to manipulate data in the target database.

[0013] According to an embodiment of the present disclosure, M is greater than 1, and the source database and the target database both belong to the M databases.

[0014] Another aspect of the embodiments of the present disclosure provides a database synchronization system. The system includes an interceptor, a distributed message queue, and a data synchronization cluster. The interceptor is used to: in response to receiving a user request for data operation from any one of M databases, obtain the user request, where M is an integer greater than or equal to 1; based on the user request, obtain a data synchronization message, wherein the data synchronization message includes identification information indicating the data operated by the user request, and direction information indicating the synchronization direction of the data between databases; and push the data synchronization message to the distributed message queue. The distributed message queue is used to receive the data synchronization message and notify the data synchronization cluster to consume the data synchronization message. The data synchronization cluster is used to: subscribe to the data synchronization message from the distributed message queue; and after subscribing to the data synchronization message, obtain the complete data indicated by the identification information from the source database indicated by the direction information, and write the complete data to the target database indicated by the direction information.

[0015] The embodiment of the present disclosure provides a database synchronization device. The device includes a request acquisition module, a synchronization message acquisition module, a push module, a subscription module, and a data synchronization module. The request acquisition module is used to respond to a user request for data operation received by any one of M databases, and obtain the user request, where M is an integer greater than or equal to 1. The synchronization message acquisition module is used to obtain a data synchronization message based on the user request, wherein the data synchronization message includes identification information indicating the data operated by the user request, and direction information indicating the synchronization direction of the data between databases. The push module is used to push the data synchronization message to a distributed message queue. The subscription module is used to subscribe to the data synchronization message from the distributed message queue. The data synchronization module is used to obtain the complete data indicated by the identification information from the source database indicated by the direction information after subscribing to the data synchronization message, and write the complete data into the target database indicated by the direction information.

[0016] According to an embodiment of the present disclosure, the device further includes a setting module. The setting module is used to: pre-set information of the N data tables involved in the transaction indicated by the transaction name; pre-set a query statement for obtaining data manipulation results from each of the data tables in the source database and the target database; and pre-set a replay execution statement of the target database, the replay execution statement being used to manipulate data in the target database.

[0017] Another aspect of the embodiments of the present disclosure provides an electronic device. The electronic device includes one or more memories and one or more processors. The memories store executable instructions. The processors execute the executable instructions to implement the above method.

[0018] Another aspect of an embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method described above when executed.

[0019] Another aspect of an embodiment of the present disclosure provides a computer program, wherein the computer program includes computer executable instructions, and the instructions are used to implement the method described above when executed.

[0020] One or more of the above embodiments have the following advantages or beneficial effects: the operation of the database itself and data synchronization are decoupled by using a distributed message queue, so that the availability of external services of the source database and the target database for data synchronization are not dependent on each other, thereby improving the efficiency of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0022] Figure 1 The application scenario of the data synchronization method and system according to the embodiment of the present disclosure is schematically shown;

[0023] Figure 2 A flowchart of a data synchronization method according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 3 A flowchart of obtaining a data synchronization message according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 4 A flowchart of writing to a target database after subscribing to a data synchronization message according to an embodiment of the present disclosure is schematically shown;

[0026] Figure 5 A flowchart of writing to a target database after subscribing to a data synchronization message according to another embodiment of the present disclosure is schematically shown;

[0027] Figure 6 A block diagram schematically shows a data synchronization device according to an embodiment of the present disclosure; and

[0028] Figure 7 A block diagram of an electronic device suitable for implementing a data synchronization method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0029] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0030] The embodiments of the present disclosure provide a database synchronization method, system, device, electronic device, and computer-readable storage medium. Specifically, the method includes first responding to a user request for data operation received by any one of M databases, obtaining the user request, where M is an integer greater than or equal to 1; then based on the user request, obtaining a data synchronization message, wherein the data synchronization message includes identification information for indicating the data operated by the user request, and direction information for indicating the synchronization direction of the data between databases; then pushing the data synchronization message to a distributed message queue; then subscribing to the data synchronization message from the distributed message queue; and after subscribing to the data synchronization message, obtaining the complete data indicated by the identification information from the source database indicated by the direction information, and writing the complete data to the target database indicated by the direction information.

[0031] In this way, when performing database synchronization, it is not dependent on the database's own synchronization mechanism or log format, so it can be applied to data synchronization between a wide range of relational databases. Moreover, the embodiment of the present disclosure uses a distributed message queue to decouple the operation and data synchronization of the database itself, so it is not only applicable to data synchronization between homogeneous databases, but also to data synchronization between heterogeneous databases, and also makes the availability of external services of the source database and the target database for data synchronization independent of each other, so as not to affect business continuity.

[0032] It should be noted that the data synchronization method, system and device determined in each embodiment of the present disclosure can be used in the financial field, and can also be used in any field other than the financial field. The present disclosure does not limit the application field.

[0033] Figure 1 An application scenario 100 of the data synchronization method and system according to an embodiment of the present disclosure is schematically shown.

[0034] like Figure 1 As shown, the application scenario 100 may include an old system 10, a new system 20, and a data synchronization system according to an embodiment of the present disclosure. The data synchronization system may implement bidirectional data synchronization (also referred to as dual write synchronization) between the old system 10 and the new system 20.

[0035] Specifically, the cluster composed of the old application server cluster 11 and the old application database 12 represents the old system 10. The cluster composed of the new application server cluster 21 and the new application database 22 represents the new system 20 after reconstruction or transformation.

[0036] The data synchronization system may include interceptors 101A and 101B, a distributed message queue 102 , and a data synchronization cluster 103 .

[0037] Among them, interceptors 101A and 101B can be set in the old system 10 and the new system 20 respectively. Each interceptor can be used to intercept user requests of the system in which it is located. Specifically, interceptor 101A is used for the old system 10 to receive user requests for data operations, obtain the user request, and encapsulate and obtain a data synchronization message (for example, one of data synchronization messages 1, 2, 3...) based on the user request, and then push the data synchronization message to the distributed message queue 102. Correspondingly, interceptor 101B is used for the new system 20 to receive user requests for data operations, and obtain a data synchronization message (for example, one of data synchronization messages 1, 2, 3...) according to the user request, and then push the data synchronization message to the distributed message queue 102.

[0038] The distributed message queue 102 is used to receive and store the data synchronization messages pushed by the interceptors 101A and 101B, and to notify the data synchronization cluster 103 to consume the data synchronization messages.

[0039] The data synchronization cluster 103 is used to subscribe to data synchronization messages from the distributed message queue 102 , and after subscribing to each data synchronization message, implement dual-write synchronization of data in the old system 10 and the new system 20 according to the information in each data synchronization message.

[0040] In the application scenario 100 , the old system 10 and the new system 20 can provide services to the outside world at the same time. For example, non-grayscale users access the old system 10 , and grayscale users access the new system 20 .

[0041] Then, in the normal online access mode, the request of non-grayscale users can enter through the old application server cluster 11. After executing the relevant operations, the data in the old application database 12 can be directly operated according to the application logic operation. At the same time, the interceptor 101A set in the old application server cluster 11 intercepts the user request and assembles the identification information of the data to be operated (for example, the primary key of the data) and other information and pushes it to the distributed message queue 102.

[0042] Correspondingly, the grayscale user's request can enter through the new application server cluster 21. After executing relevant operations, the data in the new application database 22 can be directly operated according to the application logic operation. At the same time, the interceptor 101B set in the new application server cluster 21 intercepts the user request and assembles the identification information of the data to be operated (for example, the primary key of the data) and other information and pushes it to the distributed message queue 102.

[0043] The distributed message queue 102 receives push messages from clusters on both sides and notifies the data synchronization cluster 103 to consume.

[0044] The data synchronization cluster 103 can also be a server cluster with an application deployed. After subscribing to the data synchronization messages pushed by the new and old systems from the distributed message queue 102, it obtains the complete data from the source database according to the source of each message after obtaining it, and writes it directly into the target database, thereby realizing double-write synchronization of data between the new and old systems.

[0045] According to the embodiment of the present disclosure, the distributed message queue 102 is used to decouple the business operation and data synchronization of the old system 10 and the new system 20, so that the availability of their external services is not dependent on each other, which can ensure to a large extent that the data in the databases of the new and old systems can achieve final consistency in a short time. Moreover, when the old and new systems (including the application layer and the database) are switched in grayscale release mechanism as a whole, the grayscale user range can be flexibly switched according to the embodiment of the present disclosure, and the timeliness of data interaction between users of the old and new systems can be effectively guaranteed.

[0046] According to the application scenario 100, when the system needs to undergo a technical transformation and the entire system needs to be reconstructed or the database selection needs to be changed, the method and system of the disclosed embodiment can effectively ensure to a certain extent that the data of the cluster databases on both sides of the distributed message queue 102 can achieve final consistency in a relatively short period of time, and can enable the new and old systems to run in parallel, so as to fully verify the reliability of the new technology in actual production, gradually switch user traffic to the new system, and do not generate a scenario that has a significant impact on stable operation. When encountering problems that affect user use, user traffic can also be flexibly switched back without affecting business continuity.

[0047] It should be noted that Figure 1 What is shown are merely examples of application scenarios to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0048] For example, the system and method of the embodiments of the present disclosure may also be applied to a database backup scenario, that is, the source database unidirectionally synchronizes data to a backup database (ie, target database) through the data synchronization system of the embodiments of the present disclosure.

[0049] For another example, the system and method of the embodiment of the present disclosure can also be applied to more systems to realize data synchronization between two systems, or in one or more data backup scenarios. For example, each of three or more systems can be distributed to push data synchronization messages to the distributed message queue 102, wherein the data synchronization message can include direction information of data synchronization, so that after the data synchronization cluster 103 subscribes to each data synchronization message from the distributed message queue 102, it determines the source database and the target database according to the direction information therein to perform data synchronization. It can be seen that in the data synchronization system and method of the embodiment of the present disclosure, by utilizing the distributed message queue 102, data synchronization is decoupled from the business of the database itself. In this way, after building a set of data synchronization system, any number of systems that need to perform data synchronization can be connected according to actual needs.

[0050] The following combination Figure 1 Application scenarios 100, reference Figure 2 to Figure 5 The method of the embodiment of the present disclosure is exemplified.

[0051] Figure 2 The flowchart of the data synchronization method according to the embodiment of the present disclosure is schematically shown.

[0052] like Figure 2 As shown, the data synchronization method may include operations S210 to S250.

[0053] In operation S210, in response to any one of the M databases receiving a user request for data operation, the user request is obtained, where M is an integer greater than or equal to 1. For example, when the old system 10 and the new system 20 receive user requests to operate data in their respective databases, the interceptors 101A and 101B intercept the corresponding user requests.

[0054] In some embodiments, the interceptors 101A and 101B can capture any user request for data operation, that is, once the old system 10 and / or the new system 20 receives the user request, they can perform corresponding interception.

[0055] In other embodiments, when capturing user requests, the interceptors 101A and 101B may capture only part of the requests specified by the user, rather than all of the requests. For example, in operation S210, in response to receiving a user request, the transaction name in the user request may be first identified, where the transaction name is information in the user request that indicates the business function executed by the user request, and then the user request may be acquired when the transaction name belongs to a preset transaction name set, where the transaction name set is a preset set of transaction names that require data synchronization.

[0056] For example, in application scenario 100 involving the migration of old and new systems, transactions involving user information are usually transactions that require data synchronization (for example, transfer payment transactions, batch salary transactions, user information modification transactions, etc. in the financial field), while some transactions such as parameter setting, log viewing, and log recording may have little impact on business operations and can be regarded as non-important transactions and do not need to be intercepted and synchronized.

[0057] In operation S220, based on the user request, a data synchronization message is obtained, wherein the data synchronization message includes identification information for indicating the data operated by the user request, and direction information for indicating the synchronization direction of the data between databases. The identification information may be, for example, a data primary key, or other information that can locate the data record operated by the user. The direction information is used to clarify the information from which database the data synchronization is synchronized to which database, for example, by characterizing the direction of the data synchronization through a predetermined character. The capacity of the data synchronization message is generally not too large, so only some key information can be encapsulated therein.

[0058] After the identification information and the direction information are encapsulated to obtain a data synchronization message, when the data synchronization cluster 103 consumes a data synchronization message, the source database and the target database can be determined by the direction information, and the complete data in the source database can be located by the identification information.

[0059] According to some embodiments of the present disclosure, when the interceptors 101A and 101B capture a user request for a transaction belonging to a preset transaction name set, the data synchronization message also includes the transaction name. The process of encapsulating the data synchronization message in operation S230 may be, on the one hand, extracting the transaction name and identification information from the user request, on the other hand, obtaining direction information according to the data synchronization configuration of the database operated on the user request, and then encapsulating the transaction name, identification information, and direction information to obtain the data synchronization message.

[0060] In operation S230 , the data synchronization message is pushed to the distributed message queue 102 .

[0061] In operation S240, a data synchronization message is subscribed from the distributed message queue 102. For example, the data synchronization cluster 103 subscribes.

[0062] In operation S250 , after subscribing to the data synchronization message, the data synchronization cluster 103 obtains the complete data indicated by the identification information from the source database indicated by the direction information, and writes the complete data to the target database indicated by the direction information.

[0063] According to the embodiments of the present disclosure, the distributed message queue 102 is used to realize the decoupling and asynchronous operation of data synchronization and system services, so that the source database and the target database are independent of each other. In addition, the peak shaving characteristics of the distributed message queue 102 can be used to improve the efficiency and timeliness of data synchronization and reduce the impact on the performance of the source database and the target database.

[0064] Figure 3 The flowchart of obtaining a data synchronization message according to an embodiment of the present disclosure is schematically shown.

[0065] like Figure 3 As shown, according to this embodiment, the process of obtaining a data synchronization message may include operations S301 to S306.

[0066] In operation S301, a user request is captured. Prior to this, a mechanism for triggering data synchronization may be added to the application server (i.e., the old application server cluster 11 and the new application server cluster 21), including setting interceptors 101A and 101B, as well as the timing and conditions for intercepting user requests (e.g., only intercepting user-specified requests), etc.

[0067] Then in operation S302, the user's request operation is continued to be executed to complete the reading and writing of the database of the cluster. After the filter is triggered, the corresponding program is firstly called according to the original functional logic to perform corresponding operations on the database in the cluster.

[0068] Next, in operation S203, it is determined whether the transaction requested by the user requires dual write synchronization. It is determined whether the transaction needs to trigger dual write synchronization to another cluster based on pre-configured parameters.

[0069] In operation S304, if double writing is not required, the process returns normally.

[0070] In operation S305, if dual writing is required, an asynchronous processing request is called to obtain the transaction name, data primary key and other parameters in the intercepted request to determine the direction of data synchronization (i.e. direction information).

[0071] Then, in operation S306 , the transaction name, data primary key, and direction information in the intercepted request are encapsulated as a data synchronization message and pushed to the distributed message queue 102 .

[0072] Figure 4 The flowchart of operation S250 of writing into a target database after subscribing to a data synchronization message according to an embodiment of the present disclosure is schematically shown.

[0073] like Figure 4As shown, according to the embodiment of the present disclosure, operation S250 may include operations S401 to S407. In this embodiment, the data synchronization message includes a transaction name, identification information, and direction information.

[0074] In operation S401, information of N data tables involved in the transaction indicated by the transaction name is obtained. The transactions that can be executed by each system can be determined when the system is put into operation. Assuming that the old system 10 is a system that provides credit card services to users, the transactions involved may include, for example, credit card processing, credit card repayment, credit card withdrawal, etc. It is assumed that the credit card processing transaction requires the user to provide basic information (name, phone number, ID number, etc.), income information, asset information, etc., and accordingly, the N data tables involved in the credit card processing may be, for example, a basic information data table, a salary income data table, a personal asset information table, etc.

[0075] In operation S402, data is queried from N data tables in a source database to obtain complete data.

[0076] For example, a query statement (select statement, denoted as SQL-A) for obtaining data manipulation results from each data table in the source database can be used to obtain data indicated by identification information from each data table in the source database to obtain local data corresponding to each data table, wherein N local data are obtained corresponding to N data tables. The N local data are then merged to obtain complete data.

[0077] For example, if it is determined based on the transaction name and identification information in the data synchronization message that the transaction data of Zhang San's application for a credit card needs to be synchronized, then query statements can be used to query the basic information data table, salary income data table, and personal asset information table from the source database respectively, and Zhang San's information can be queried from these tables respectively. Then, the queried information can be merged to obtain the information registered by Zhang San when applying for the credit card transaction (i.e., complete data).

[0078] In operation S403, data is queried from N data tables in the target database to obtain verification data.

[0079] Correspondingly, for example, a query statement (also a select statement, denoted as SQL-A) for obtaining data manipulation results from each data table in the target database can be used to obtain data indicated by identification information from each data table in the target database, and obtain local verification data corresponding to each data table, wherein N local verification data are obtained corresponding to N data tables. Then, the N local verification data are merged to obtain verification data.

[0080] Similarly, for example, when synchronizing the transaction data of Zhang San's application for a credit card, you can use query statements to query the basic information data table, salary income data table, and personal asset information table from the target database respectively, and query Zhang San's information (if any) from these tables respectively, and then merge the queried information to obtain the information recorded in the target database when Zhang San applied for the credit card transaction (i.e., verification data).

[0081] In operation S404, the complete data and the verification data are compared to obtain a comparison result.

[0082] In operation S405, based on the comparison result, determine whether the complete data violates the database constraints of the target database (for example, primary key conflict). If not, perform operation S406; if yes, perform operation S407. For example, in one case, assuming that Zhang San's information is not found from the target database, it is obvious that the database constraints are not violated. For another example, in one case, Zhang San's information is found from the target database, but the timestamp of this information is earlier than the information from the source database, which also shows that the complete data does not violate the target database constraints. For another example, in one case, the timestamp of Zhang San's information found from the target database is later than the complete information in the source database. In this case, the complete data violates the database constraints. It should be noted that here are only several common situations based on database constraint judgment. The specific judgment logic of database constraints depends on the settings of each database.

[0083] In operation S406 , when the complete data does not violate the database constraints of the target database, the complete data is written into the target database.

[0084] Let's continue to use the example of synchronizing Zhang San's credit card application data. For example, the replay execution statement (remembered as SQL-B) for data synchronization in the target database can be assembled according to all the field data in the complete data to form a data manipulation statement (DML, Data Manipulation Language), and then the data manipulation statement is submitted to the target database for execution.

[0085] In operation S407, when the complete data violates the database constraint of the target database, it means that the complete data has already existed in the target database, so the complete data is abandoned to avoid adverse effects on the target database.

[0086] According to the embodiments of the present disclosure, by verifying the synchronization data through database constraints, data security and accuracy can be improved, and adverse effects on the database itself can be avoided.

[0087] According to the disclosed embodiments, the application layer can use an asynchronous trigger mechanism through an interceptor without using database triggers. At the same time, a simple select statement can be used to query and obtain original data from the source database, and a simple DML statement can be used to add, delete, and modify the target database, which has a relatively low impact on the performance of the source database and the target database.

[0088] Figure 5 A flowchart of writing to a target database after subscribing to a data synchronization message according to another embodiment of the present disclosure is schematically shown.

[0089] like Figure 5 As shown, in this embodiment, the data synchronization cluster 103 can subscribe to the data synchronization message from the distributed message queue 102 through operations S501 to S511 and write the message into the target database.

[0090] First, in operation S501, a message is subscribed from the distributed message queue 102. Before the subscription, the information of the N data tables involved in the transaction indicated by the transaction name can be pre-set in the data synchronization cluster 103, the select query statement SQL-A for obtaining the data manipulation result from each data table in the source database and the target database can be pre-set, and the replay execution statement (SQL-B) of the target database can be pre-set, and the replay execution statement SQL-B is used to manipulate data in the target database (for example, insert, update, delete, etc.).

[0091] Then in operation S502, M messages can be obtained each time, where M can be configured by parameters. The transaction name, parameters, and the direction of double writing are obtained from each message.

[0092] In operation S503, it is determined whether there are any unconsumed messages in the distributed message queue 102. If yes, after the current message consumption is completed, the distributed message queue 102 is returned to continue to obtain messages. If no, the message notification of the distributed message queue 102 is waited.

[0093] Next, in operation S504, the transaction name, data primary key, and dual write direction are obtained from the data synchronization message.

[0094] Next, in operation S505, it is determined whether the transaction requires double writing according to the information in operation S504. If it is required, operation S506 is continued to be executed, and if it is not required, the process returns to operation S503.

[0095] In operation S506, for transactions that require double writing, the tables involved are obtained according to the transaction name, and each table can be processed in parallel, but the concurrency limit needs to be controlled to S, which can be configured through parameters. S is an integer, and if the transaction involves N data tables, 2≤S≤N.

[0096] In operation S507, it is determined whether there are any tables to be processed. If not, the process returns to operation S506. If yes, the process continues to process each table.

[0097] In operation S508 , SQL-A is used to query data from the source database and the target database respectively.

[0098] In operation S509, after the data is acquired, a comparison is performed to ensure that the DML operation executed by the subsequent SQL-B replay does not violate the database-related constraints.

[0099] In operation S510, it is determined whether the target database related constraints are violated. If so, the obtained data is abandoned and the process returns to operation S507. If not, the process continues to operation S511.

[0100] In operation S511, all field data in SQL-B are assembled to form a simple DML statement, which is executed and submitted in the target database. After submission, the processed message is submitted to the distributed message queue 102, and the next round of messages is obtained.

[0101] In this way, according to the embodiments of the present disclosure, an asynchronous trigger mechanism is used at the application layer, without using database triggers, using simple select statements to query the original database to obtain raw data, and using simple DML statements to perform add, delete, and modify operations on the target database, which has a low impact on the performance of the source database and the target database. In addition, the decoupling, asynchronous, and peak-cutting characteristics of the message queue are used to ensure that the availability of external services of the two clusters performing data synchronization will not be dependent on each other.

[0102] Figure 6 The block diagram of a data synchronization device 600 according to an embodiment of the present disclosure is schematically shown.

[0103] like Figure 6 As shown, the database synchronization device 600 may include a request acquisition module 610, a synchronization message acquisition module 620, a push module 630, a subscription module 640, and a data synchronization module 650. According to other embodiments of the present disclosure, the data synchronization device 600 may further include a setting module 660. The data synchronization device 600 may be used to implement reference Figure 2 to Figure 5 The method described.

[0104] The request acquisition module 610 is used to acquire a user request in response to any one of the M databases receiving a user request for data operation, where M is an integer greater than or equal to 1.

[0105] The synchronization message obtaining module 620 is used to obtain a data synchronization message based on a user request, wherein the data synchronization message includes identification information indicating the data operated by the user request and direction information indicating the synchronization direction of the data between databases.

[0106] The push module 630 is used to push the data synchronization message to the distributed message queue.

[0107] The subscription module 640 is used to subscribe to data synchronization messages from the distributed message queue.

[0108] The data synchronization module 650 is used to obtain the complete data indicated by the identification information from the source database indicated by the direction information after subscribing to the data synchronization message, and write the complete data into the target database indicated by the direction information.

[0109] The setting module 660 is used to: pre-set the information of N data tables involved in the transaction indicated by the transaction name; pre-set the query statement for obtaining data manipulation results from each data table in the source database and the target database; and pre-set the replay execution statement of the target database, wherein the replay execution statement is used to manipulate data in the target database.

[0110] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0111] For example, any multiple of the request acquisition module 610, the synchronization message acquisition module 620, the push module 630, the subscription module 640, the data synchronization module 650, and the setting module 660 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the request acquisition module 610, the synchronization message acquisition module 620, the push module 630, the subscription module 640, the data synchronization module 650, and the setting module 660 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the request acquisition module 610, the synchronization message acquisition module 620, the push module 630, the subscription module 640, the data synchronization module 650, and the setting module 660 can be at least partially implemented as a computer program module, which can perform corresponding functions when executed.

[0112] Figure 7 A block diagram of an electronic device 700 suitable for implementing a data synchronization method according to an embodiment of the present disclosure is schematically shown. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0113] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0114] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0115] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0116] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0117] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0118] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0119] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains a program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the image recognition method provided by the embodiment of the present disclosure.

[0120] When the computer program is executed by the processor 701, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0121] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0122] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0123] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A database synchronization method, comprising: In response to any one of the M databases receiving a user request for data operation, obtaining the user request by using an interceptor, where M is an integer greater than or equal to 1, and a corresponding interceptor is set for each of the M databases; Based on the user request, a data synchronization message is obtained, wherein the data synchronization message includes a transaction name, identification information for indicating the data operated by the user request, and direction information for indicating a synchronization direction of the data between databases; Pushing the data synchronization message to a distributed message queue; Subscribing to the data synchronization message from the distributed message queue; and After subscribing to the data synchronization message, obtaining the complete data indicated by the identification information from the source database indicated by the direction information, and writing the complete data into the target database indicated by the direction information; Wherein, obtaining a data synchronization message based on the user request includes: Extracting a transaction name and identification information from the user request; Obtaining the direction information according to the data synchronization configuration of the database operated by the user request; and Encapsulating the transaction name, the identification information and the direction information to obtain the data synchronization message; The acquiring the complete data indicated by the identification information from the source database indicated by the direction information comprises: Obtain information of N data tables involved in the transaction indicated by the transaction name; Querying data from the N data tables in the source database to obtain the complete data, including: using a query statement that obtains data manipulation results from each of the data tables in the source database, obtaining the data indicated by the identification information from each of the data tables in the source database, and obtaining local data corresponding to each of the data tables, wherein N local data correspond to N data tables; and merging the N local data to obtain the complete data.

2. The method according to claim 1, wherein: The step of obtaining the user request for data operation in response to any one of the M databases receiving the user request includes: In response to receiving the user request, identifying a transaction name in the user request, the transaction name being used to indicate a business function executed by the user request; and When the transaction name belongs to a preset transaction name set, the user request is obtained, wherein the transaction name set is a preset set of transaction names that require data synchronization.

3. The method according to claim 1, wherein: The step of writing the complete data into the target database indicated by the direction information comprises: Querying data from the N data tables in the target database to obtain verification data includes: Using a query statement for obtaining data manipulation results from each of the data tables in the target database, the data indicated by the identification information is obtained from each of the data tables in the target database to obtain local verification data corresponding to each of the data tables, wherein N local verification data are obtained corresponding to N of the data tables; and Merging N pieces of local verification data to obtain the verification data; comparing the complete data and the verification data to obtain a comparison result; and When the comparison result indicates that the complete data does not violate the database constraint of the target database, the complete data is written into the target database.

4. The method according to any one of claims 1 to 3, wherein: The step of writing the complete data into the target database indicated by the direction information comprises: Assembling the replay execution statement for data synchronization in the target database according to all field data in the complete data to form a data manipulation statement; and Submit the data manipulation statement to the target database for execution.

5. The method according to claim 4, wherein: When N is greater than 1, data is queried from S data tables concurrently, where S is an integer and 2≤S≤N.

6. The method according to claim 4, wherein: The method further comprises: Presetting information of N data tables involved in the transaction indicated by the transaction name; Presetting a query statement for obtaining data manipulation results from each of the data tables in the source database and the target database; and A replay execution statement of the target database is preset, and the replay execution statement is used to manipulate data in the target database.

7. The method according to claim 1, wherein: M is greater than 1, and the source database and the target database both belong to the M databases.

8. A database synchronization system, comprising: Interceptor, wherein a corresponding interceptor is set for each of the M databases, where M is an integer greater than or equal to 1; Distributed message queues, and Data synchronization cluster; in, The interceptor is used to: In response to any one of the M databases receiving a user request for data operation, obtaining the user request; Based on the user request, a data synchronization message is obtained, wherein the data synchronization message includes identification information for indicating the data operated by the user request and direction information for indicating the synchronization direction of the data between databases; and Pushing the data synchronization message to the distributed message queue; The distributed message queue is used for: receiving the data synchronization message; and Notify the data synchronization cluster to consume the data synchronization message; The data synchronization cluster is used for: Subscribing to the data synchronization message from the distributed message queue; and After subscribing to the data synchronization message, obtaining the complete data indicated by the identification information from the source database indicated by the direction information, and writing the complete data into the target database indicated by the direction information; The interceptor obtains the data synchronization message based on the user request, specifically including: Extracting a transaction name and identification information from the user request; Obtaining the direction information according to the data synchronization configuration of the database operated by the user request; and Encapsulating the transaction name, the identification information and the direction information to obtain the data synchronization message; The distributed message queue obtains the complete data indicated by the identification information from the source database indicated by the direction information, specifically including: Obtain information of N data tables involved in the transaction indicated by the transaction name; Querying data from the N data tables in the source database to obtain the complete data, including: using a query statement that obtains data manipulation results from each of the data tables in the source database, obtaining the data indicated by the identification information from each of the data tables in the source database, and obtaining local data corresponding to each of the data tables, wherein N local data correspond to N data tables; and merging the N local data to obtain the complete data.

9. A database synchronization device, comprising: A request acquisition module, configured to, in response to receiving a user request for data operation from any one of the M databases, acquire the user request using an interceptor, wherein M is an integer greater than or equal to 1, and a corresponding interceptor is provided for each of the M databases; A synchronization message obtaining module, used to obtain a data synchronization message based on the user request, wherein the data synchronization message includes a transaction name, identification information indicating the data operated by the user request, and direction information indicating the synchronization direction of the data between databases; A push module, used to push the data synchronization message to a distributed message queue; A subscription module, configured to subscribe to the data synchronization message from the distributed message queue; and A data synchronization module, configured to, after subscribing to the data synchronization message, obtain the complete data indicated by the identification information from the source database indicated by the direction information, and write the complete data into the target database indicated by the direction information; Wherein, the synchronization message obtaining module is specifically used for: Extracting a transaction name and identification information from the user request; Obtaining the direction information according to the data synchronization configuration of the database operated by the user request; and Encapsulating the transaction name, the identification information and the direction information to obtain the data synchronization message; Wherein, the data synchronization module is specifically used for: Obtain information of N data tables involved in the transaction indicated by the transaction name; Querying data from the N data tables in the source database to obtain the complete data, including: using a query statement that obtains data manipulation results from each of the data tables in the source database, obtaining the data indicated by the identification information from each of the data tables in the source database, and obtaining local data corresponding to each of the data tables, wherein N local data correspond to N data tables; and merging the N local data to obtain the complete data.

10. An electronic device, comprising: One or more memories storing executable instructions; as well as One or more processors execute the executable instructions to implement the method according to any one of claims 1 to 7.

11. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 7.

Images