Data synchronization method, system and device based on incremental data synchronization component

Abstract

The present disclosure provides a data synchronization method, device, electronic device, and storage medium based on an incremental data synchronization component, which can be applied to the field of big data technology. The method is applied to a target-end device, and the incremental data synchronization component includes a log parsing module and a consumption module. The method includes: using the consumption module to consume messages in a message queue to obtain multiple query statements, wherein the message is sent by a source-end device to the message queue, and the source-end device is configured to record incremental data in a log of a source-end database to obtain an incremental log, and using the log parsing module to parse the incremental log to obtain multiple query statements, and then encapsulating the multiple query statements to obtain a message; based on the grammatical category of each query statement in the multiple query statements, the multiple query statements are classified to obtain a classification result; and based on the classification result, the multiple query statements are used to process the target-end database to synchronize the incremental data to the target-end database.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and more specifically, to a data synchronization method, system, device, electronic device, and storage medium based on an incremental data synchronization component. Background Art

[0002] With the development of big data and cloud computing technologies, more and more enterprises have built data processing centers and big data cloud service platforms to store increasingly large amounts of business data. Enterprise business data is generally stored in different domains, and incremental data synchronization components such as Canal are used to achieve cross-domain data synchronization.

[0003] In the process of implementing the present disclosure, the inventors discovered that there is a problem of data loss when applying the native incremental data synchronization component to perform cross-domain data synchronization. Summary of the Invention

[0004] In view of this, the present disclosure provides a data synchronization method, apparatus, electronic device, readable storage medium, and computer program product based on an incremental data synchronization component.

[0005] One aspect of the present disclosure provides a data synchronization method based on an incremental data synchronization component, which is applied to a target-end device. The incremental data synchronization component includes a log parsing module and a consumption module. The method includes: using the consumption module to consume messages in a message queue to obtain multiple query statements, wherein the messages are sent by a source-end device to the message queue, and the source-end device is configured to record incremental data in a log of a source-end database to obtain an incremental log, and using the log parsing module to parse the incremental log to obtain the multiple query statements, and then encapsulating the multiple query statements to obtain the messages; based on the grammatical category of each of the multiple query statements, classifying the multiple query statements to obtain classification results; and based on the classification results, using the multiple query statements to process the target-end database to synchronize the incremental data to the target-end database.

[0006] According to an embodiment of the present disclosure, the above-mentioned grammatical categories include a data definition language category and a data manipulation language category; wherein, based on the grammatical category of each of the above-mentioned query statements, the multiple query statements are classified to obtain classification results, including: classifying the multiple query statements into a first query statement belonging to the above-mentioned data definition language category and a second query statement belonging to the above-mentioned data manipulation language category; and determining the above-mentioned classification results based on the number of the above-mentioned first query statements and the number of the above-mentioned second query statements.

[0007] According to an embodiment of the present disclosure, based on the above-mentioned classification results, multiple query statements are used to process the target-side database to synchronize the above-mentioned incremental data to the above-mentioned target-side database, including: when the number of the above-mentioned first query statements is zero or the number of the above-mentioned second query statements is 0, multiple query statements are executed in sequence in the above-mentioned target-side database to synchronize the above-mentioned incremental data to the above-mentioned target-side database; and when the number of the above-mentioned first query statements and the number of the above-mentioned second query statements are both greater than 0, multiple query statements are executed in the above-mentioned target-side database based on a preset execution strategy to synchronize the above-mentioned incremental data to the above-mentioned target-side database.

[0008] According to an embodiment of the present disclosure, the above-mentioned execution of multiple query statements in the above-mentioned target-side database based on a preset execution strategy includes: executing at least one of the above-mentioned second query statements in sequence in the above-mentioned target-side database; and when at least one of the above-mentioned second query statements is executed, executing at least one of the above-mentioned first query statements in sequence in the above-mentioned target-side database.

[0009] Another aspect of the present disclosure provides a data synchronization method based on an incremental data synchronization component, which is applied to a source-end device. The incremental data synchronization component includes a log parsing module and a consumption module. The method includes: recording incremental data in the log of the source-end database to obtain incremental logs; in response to triggering a data synchronization event, parsing the incremental logs using the log parsing module to obtain multiple query statements, and the data synchronization event represents data synchronization of a target-end database of a target-end device; encapsulating the multiple query statements into messages; and sending the messages to a message queue so that the target-end device consumes the messages in the message queue using the consumption module to obtain the multiple query statements, and classifying the multiple query statements to obtain classification results, and then based on the classification results, using the multiple query statements to process the target-end database to synchronize the incremental data to the target-end database.

[0010] According to an embodiment of the present disclosure, the above-mentioned data synchronization event carries target domain information, and the above-mentioned target domain information corresponds to the above-mentioned target end device; the above-mentioned method also includes: in response to triggering the above-mentioned data synchronization event, based on the above-mentioned target domain information, obtaining the target data synchronization instance from the distributed cluster; and based on the above-mentioned target data synchronization instance, determining the target log parsing module and the target data table; wherein, the above-mentioned use of the above-mentioned log parsing module to parse the above-mentioned incremental log to obtain multiple query statements includes: using the above-mentioned target log parsing module to parse the target incremental log associated with the above-mentioned target data table in the above-mentioned incremental log to obtain multiple of the above-mentioned query statements.

[0011] According to an embodiment of the present disclosure, the above method also includes: determining a target message queue based on the above target data synchronization instance; wherein the above sending the above message to the message queue includes: sending the above message to the above target message queue in the message queue cluster.

[0012] Another aspect of the present disclosure provides a data synchronization system based on an incremental data synchronization component, wherein the incremental data synchronization component includes a log parsing module and a consumption module, and the system includes: a source-side device, configured to record incremental data in the log of a source-side database to obtain incremental logs, and in response to triggering a data synchronization event, to parse the incremental logs using the log parsing module to obtain multiple query statements, encapsulate the multiple query statements into messages, and send the messages to a message queue; a target-side device, configured to consume messages in the message queue using the consumption module to obtain the multiple query statements, classify the multiple query statements to obtain classification results, and based on the classification results, use the multiple query statements to process the target-side database to synchronize the incremental data to the target-side database; and a message middleware, including one or more message queues.

[0013] Another aspect of the present disclosure provides a data synchronization device based on an incremental data synchronization component, which is arranged on a target-end device. The incremental data synchronization component includes a log parsing module and a consumption module. The device includes: a consumption module for consuming messages in a message queue using the consumption module to obtain multiple query statements, wherein the messages are sent by a source-end device to the message queue, and the source-end device is configured to record incremental data in a log of a source-end database to obtain an incremental log, and parse the incremental log using the log parsing module to obtain the multiple query statements, and then encapsulate the multiple query statements to obtain the messages; a classification module for classifying the multiple query statements based on the grammatical category of each of the multiple query statements to obtain classification results; and a processing module for processing the target-end database using the multiple query statements based on the classification results to synchronize the incremental data to the target-end database.

[0014] Another aspect of the present disclosure provides a data synchronization device based on an incremental data synchronization component, which is arranged on a source-end device. The incremental data synchronization component includes a log parsing module and a consumption module. The device includes: a recording module, which is used to record incremental data in the log of the source-end database to obtain incremental logs; a parsing module, which is used to parse the incremental logs using the log parsing module in response to triggering a data synchronization event to obtain multiple query statements, and the data synchronization event represents data synchronization of a target-end database of a target-end device; an encapsulation module, which is used to encapsulate the multiple query statements into messages; and a sending module, which is used to send the messages to a message queue, so that the target-end device uses the consumption module to consume the messages in the message queue to obtain the multiple query statements, and classify the multiple query statements to obtain classification results, and then based on the classification results, use the multiple query statements to process the target-end database to synchronize the incremental data to the target-end database.

[0015] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors implement the method described above.

[0016] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0017] Another aspect of the present disclosure provides a computer program product, which includes computer-executable instructions. When the instructions are executed, they are used to implement the method described above.

[0018] According to the embodiments of the present disclosure, by rewriting the synchronization method in the consumption module, that is, after using the consumption module to consume the messages in the message queue and obtaining multiple query statements, classifying them based on the grammatical category of each query statement, and then adopting corresponding strategies to use the classified multiple query statements to process the target database, the technical means at least partially overcomes the technical problem that the native consumption module cannot consume query statements of data definition language and data operation language at the same time, thereby achieving the technical effect of ensuring data integrity during cross-domain data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0020] Figure 1Schematically illustrates an exemplary system architecture to which the data synchronization method and apparatus based on the incremental data synchronization component according to an embodiment of the present disclosure can be applied;

[0021] Figure 2 A flowchart of a data synchronization method based on an incremental data synchronization component applied to a target end device according to an embodiment of the present disclosure is schematically shown;

[0022] Figure 3 A flowchart of a data synchronization method based on an incremental data synchronization component applied to a source device according to an embodiment of the present disclosure is schematically shown;

[0023] Figure 4 Schematically shows a schematic diagram of a data synchronization system based on an incremental data synchronization component according to an embodiment of the present disclosure;

[0024] Figure 5 A block diagram schematically illustrates a data synchronization apparatus based on an incremental data synchronization component applied to a target end device according to an embodiment of the present disclosure;

[0025] Figure 6 A block diagram schematically illustrates a data synchronization apparatus based on an incremental data synchronization component applied to a source device according to an embodiment of the present disclosure;

[0026] Figure 7 A block diagram of an electronic device is schematically shown according to an embodiment of the present disclosure, which shows a data synchronization method based on an incremental data synchronization component. DETAILED DESCRIPTION

[0027] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0028] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0030] When expressions such as “at least one of A, B, and C, etc.” are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., “a system having at least one of A, B, and C” should include but is not limited to systems having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, and C, etc.). When expressions such as “at least one of A, B, or C, etc.” are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., “a system having at least one of A, B, or C” should include but is not limited to systems having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, and C, etc.).

[0031] When a data update operation submitted on the source side includes multiple types of query statements, the native incremental data synchronization component cannot fully execute these multiple types of query statements on the target side. Specifically, during cross-domain data synchronization, when the source side submits batches of statements in the data definition language and the data manipulation language, the consumption module of the native incremental data synchronization component can only consume one type of query statement in the received message. For example, if the received message contains statements belonging to both the data definition language and the data manipulation language, only one type of statement can be consumed in the order in which it is received. Therefore, when using the native incremental data synchronization component for cross-domain data synchronization, there is a problem of data loss.

[0032] In view of this, the embodiments of the present disclosure provide a data synchronization method, system, device, electronic device, readable storage medium and computer program product based on an incremental data synchronization component. The method can be applied to a target-end device, and the method includes: using a consumption module to consume messages in a message queue to obtain multiple query statements, wherein the message is sent by a source-end device to the message queue, and the source-end device is configured to record the incremental data in a log of a source-end database to obtain an incremental log, and using a log parsing module to parse the incremental log to obtain multiple query statements, and then encapsulating the multiple query statements to obtain a message; based on the grammatical category of each query statement in the multiple query statements, the multiple query statements are classified to obtain a classification result; and based on the classification result, the multiple query statements are used to process the target-end database to synchronize the incremental data to the target-end database.

[0033] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0034] In the technical solution disclosed herein, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0035] Figure 1 The following schematically illustrates an exemplary system architecture to which the data synchronization method and apparatus based on the incremental data synchronization component according to an embodiment of the present disclosure can be applied. Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure may not be used in other devices, systems, environments or scenarios.

[0036] like Figure 1 As shown, the system architecture 100 according to this embodiment may include servers 101, 102, 103 and a network 104. The network 104 is used to provide a medium for communication links between the servers 101, 102, 103. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0037] The servers 101, 102, and 103 may be various types of cloud servers for providing cloud computing services to users. The servers 101, 102, and 103 may be located in different domains.

[0038] It should be noted that the data synchronization method based on the incremental data synchronization component provided in the embodiment of the present disclosure can generally be executed by the servers 101, 102, and 103. Accordingly, the data synchronization device based on the incremental data synchronization component provided in the embodiment of the present disclosure can generally be set in the servers 101, 102, and 103.

[0039] For example, server 101 can serve as a source device, and server 102 can serve as a target device. Server 101 can accept data change operations submitted by terminal devices, thereby performing data changes on the database configured in server 101. The data change operations can be recorded as incremental logs. Using the data synchronization method based on the incremental data synchronization component provided in the embodiment of the present disclosure, the incremental log can be parsed into a query statement, which can be executed in the database of server 102 to achieve data synchronization between server 101 and server 102.

[0040] It should be understood that Figure 1 The number of networks and servers in the embodiment is only for illustration. Any number of networks and servers may be provided as required.

[0041] Figure 2 A flowchart of a data synchronization method based on an incremental data synchronization component applied to a target end device according to an embodiment of the present disclosure is schematically shown.

[0042] like Figure 2 As shown, the method is applied to a target-end device, the incremental data synchronization component includes a log parsing module and a consumption module, and the method includes operations S201 to S203.

[0043] In operation S201 , a consumption module is used to consume messages in a message queue to obtain multiple query statements.

[0044] In operation S202 , the multiple query statements are classified based on the grammatical category of each query statement in the multiple query statements to obtain a classification result.

[0045] In operation S203 , based on the classification result, a plurality of query statements are used to process the target database to synchronize the incremental data to the target database.

[0046] According to an embodiment of the present disclosure, the incremental data synchronization component may refer to the Canal component, and accordingly, the log parsing module may refer to the Canal-Server module of the Canal component, and the consumption module may refer to the Canal-Adapter module of the Canal component.

[0047] According to embodiments of the present disclosure, a message may refer to a data representation in a message queue. The message may be sent by a source device to a message queue. The message queue may refer to a queue in any distributed message queue cluster, for example, a Topic in a RocketMQ cluster.

[0048] According to an embodiment of the present disclosure, the source-end device can be configured to record incremental data in the log of the source-end database to obtain an incremental log, and the incremental log can be parsed using a log parsing module to obtain multiple query statements, which can then be encapsulated to obtain the above-mentioned message. Taking the source-end database as MySQL and the log parsing module as the Canal-Server module as an example, the log of the source-end database can refer to the binlog log of MySQL, which can be a binary log, mainly used to record SQL (Structured Query Language) statements that update or potentially update MySQL data and save them on disk; the Canal-Server module is used to process the binlog log, and the incremental log of the binlog log can be parsed to obtain the corresponding query statement.

[0049] According to an embodiment of the present disclosure, the grammatical category may be a category determined according to a grammatical feature of a query statement, and the grammatical feature may refer to a function name used in the query statement.

[0050] According to the embodiments of the present disclosure, by rewriting the synchronization method in the consumption module, that is, after using the consumption module to consume the messages in the message queue and obtaining multiple query statements, classifying them based on the grammatical category of each query statement, and then adopting corresponding strategies to use the classified multiple query statements to process the target database, the technical means at least partially overcomes the technical problem that the native consumption module cannot consume query statements of data definition language and data operation language at the same time, thereby achieving the technical effect of ensuring data integrity during cross-domain data synchronization.

[0051] According to an embodiment of the present disclosure, the syntax category may include a Data Definition Language (DDL) category and a Data Manipulation Language (DML) category.

[0052] According to an embodiment of the present disclosure, operation S202 may specifically include the following operations:

[0053] Classifying the plurality of query statements into first query statements belonging to a data definition language category and second query statements belonging to a data manipulation language category; and determining a classification result based on the number of the first query statements and the number of the second query statements.

[0054] According to an embodiment of the present disclosure, query statements can be classified based on keywords contained in the function names of the query statements. For example, a first query statement belonging to the data definition language category may contain keywords such as CREAT, ALTER, DROP, and TRUNCATE; a second query statement belonging to the data manipulation language category may contain keywords such as SELECT, INSERT, UPDATE, and DELETE. Classification of the first and second query statements can be achieved by performing keyword round-robin matching on each query statement.

[0055] According to an embodiment of the present disclosure, operation S203 may specifically include the following operations:

[0056] When the number of first query statements is zero or the number of second query statements is 0, multiple query statements are executed in sequence in the target database to synchronize the incremental data to the target database; and when the number of first query statements and the number of second query statements are both greater than 0, multiple query statements are executed in the target database based on a preset execution strategy to synchronize the incremental data to the target database.

[0057] For example, if the number of DDL statements is 0, multiple DML statements are executed sequentially, or if the number of DML statements is 0, multiple DDL statements are executed sequentially to synchronize incremental data to the target database. If the number of both DDL and DML statements is greater than 0, multiple query statements are executed in the target database based on the preset execution strategy to synchronize incremental data to the target database.

[0058] According to an embodiment of the present disclosure, by classifying multiple query statements and then consuming the query statements according to the classification results, it is possible to avoid the situation where, when data definition language and data operation language exist in multiple query statements at the same time, the consumption module only consumes query statements of data definition language or query statements of data operation language.

[0059] According to an embodiment of the present disclosure, multiple query statements are executed in the target database based on a preset execution strategy, and at least one second query statement may be executed in sequence in the target database; and, when the execution of at least one second query statement is completed, at least one first query statement may be executed in sequence in the target database.

[0060] According to an embodiment of the present disclosure, by executing multiple query statements in the target database based on a preset execution strategy, the consumption module can completely consume the query statements when data definition language and data operation language exist in multiple query statements at the same time, thereby avoiding data loss during transmission and ensuring the integrity of the transmitted data.

[0061] Figure 3 A flowchart of a data synchronization method based on an incremental data synchronization component applied to a source device according to an embodiment of the present disclosure is schematically shown.

[0062] like Figure 3 As shown, the method is applied to a source device, the incremental data synchronization component includes a log parsing module and a consumption module, and the method includes operations S301 to S304.

[0063] In operation S301 , incremental data is recorded in a log of a source database to obtain an incremental log.

[0064] In operation S302 , in response to triggering a data synchronization event, a log parsing module is used to parse the incremental log to obtain a plurality of query statements.

[0065] In operation S303, multiple query statements are encapsulated into a message.

[0066] In operation S304, a message is sent to the message queue so that the target device uses the consumption module to consume the message in the message queue, obtain multiple query statements, and classify the multiple query statements to obtain classification results. Based on the classification results, multiple query statements are used to process the target database to synchronize the incremental data to the target database.

[0067] According to an embodiment of the present disclosure, by recording incremental data in the log of the source database to obtain an incremental log, when the source device responds to triggering a data synchronization event, the log parsing module can be used to parse the incremental log to obtain multiple query statements and encapsulate the multiple query statements into messages and send the messages to the message queue, so that the target device can use the consumption module to obtain the above query statements by consuming the messages in the message queue, and classify the above query statements, and use the classified query statements to process the target database according to the corresponding execution strategy. The technical means of synchronizing the incremental data to the target database at least partially overcomes the technical problem that the native consumption module cannot consume query statements of data definition language and data operation language at the same time, thereby achieving the technical effect of assisting in ensuring data integrity during cross-domain data synchronization.

[0068] According to an embodiment of the present disclosure, the consumption module, message queue, database or data table that needs to be synchronized, and other information used in cross-domain data synchronization can be determined based on the input information of the operator, or can also be determined based on the data synchronization instance formed by the operator's historical input information.

[0069] According to an embodiment of the present disclosure, the data synchronization event may carry target domain information, and the target domain information may correspond to the target end device. The data synchronization method may also include the following operations:

[0070] In response to the triggering of the data synchronization event, the source end obtains a target data synchronization instance from the distributed cluster based on the target domain information; and determines a target log parsing module and a target data table based on the target data synchronization instance.

[0071] According to an embodiment of the present disclosure, after determining the target log parsing module and the target data table, the target log parsing module can be used to parse the target incremental log associated with the target data table in the incremental log to obtain multiple query statements.

[0072] According to an embodiment of the present disclosure, the message middleware required for cross-domain data synchronization can also be determined based on the target data synchronization instance, that is, the target message queue can be determined based on the target data synchronization instance. Sending information to the message queue can also be sending a message to the target message queue in the message queue cluster.

[0073] Figure 4A schematic diagram of a data synchronization system based on an incremental data synchronization component according to an embodiment of the present disclosure is schematically shown.

[0074] like Figure 4 As shown, the incremental data synchronization component can include a log parsing module and a consumption module. The system includes a source device 401, a target device 402, and a message middleware 403. The source device 401 is configured with a source database and a log parsing module. The target device 402 is configured with a consumption module and a target database.

[0075] The source-end device 401 is configured to record incremental data in the log of the source-end database to obtain an incremental log. In response to triggering a data synchronization event, the incremental log is parsed using a log parsing module to obtain multiple query statements, encapsulate the multiple query statements into messages, and send the messages to a message queue.

[0076] The target device 402 is configured to use the consumption module to consume messages in the message queue, obtain multiple query statements, classify the multiple query statements, obtain classification results, and use multiple query statements to process the target database based on the classification results to synchronize incremental data to the target database.

[0077] The message middleware 403 includes one or more message queues. Specifically, the message middleware 403 can be a distributed message queue cluster, which can include a RocketMQ cluster.

[0078] The system may also include a distributed cluster, which may be a ZooKeeper cluster, which may be used to receive the configuration information of the target data synchronization instance and may communicate with the log parsing module, thereby realizing the management of the master-slave mode of the log parsing module through the distributed cluster, and recording the location information of the incremental log read by the log parsing module. For example, there may be two log parsing modules. When the log parsing module reads incremental information, the location of the read information will be sent to the distributed cluster for location synchronization. Then, when the log parsing module that reads the incremental log crashes, the incremental location read by the crashed log parsing module can be synchronized to another intact log parsing module through the distributed cluster, so that it can continue to read the incremental information, thereby ensuring data transmission efficiency.

[0079] Figure 5 A block diagram of a data synchronization apparatus based on an incremental data synchronization component applied to a target end device according to an embodiment of the present disclosure is schematically shown.

[0080] like Figure 5As shown, a data synchronization device based on an incremental data synchronization component is set on a target end device. The incremental data synchronization component includes a log parsing module and a consumption module. The device includes: a consumption module 501, a classification module 502 and a processing module 503.

[0081] The consumption module 501 is used to consume the messages in the message queue using the consumption module to obtain multiple query statements, wherein the message is sent by the source device to the message queue, and the source device is configured to record the incremental data in the log of the source database to obtain the incremental log, and use the log parsing module to parse the incremental log to obtain multiple query statements, and then encapsulate the multiple query statements to obtain the message.

[0082] The classification module 502 is configured to classify the multiple query statements based on the grammatical category of each query statement in the multiple query statements to obtain a classification result.

[0083] The processing module 503 is configured to process the target database using multiple query statements based on the classification result, so as to synchronize the incremental data to the target database.

[0084] According to the embodiments of the present disclosure, by rewriting the synchronization method in the consumption module, that is, after using the consumption module to consume the messages in the message queue and obtaining multiple query statements, the query statements are classified through the classification module based on the grammatical category of each query statement, and then corresponding strategies are adopted to utilize the classified multiple query statements to process the target database through the processing module. This technical means at least partially overcomes the technical problem that the native consumption module cannot consume query statements of data definition language and data operation language at the same time, thereby achieving the technical effect of ensuring data integrity during cross-domain data synchronization.

[0085] According to an embodiment of the present disclosure, the classification module 502 includes a definition unit and a determination unit. The definition unit is configured to classify a plurality of query statements into a first query statement belonging to a data definition language category and a second query statement belonging to a data manipulation language category; and the determination unit is configured to determine a classification result based on the number of the first query statements and the number of the second query statements.

[0086] According to an embodiment of the present disclosure, the processing module 503 includes a first execution unit and a second execution unit. The first execution unit is configured to sequentially execute multiple query statements in the target database to synchronize incremental data to the target database when the number of first query statements is zero or the number of second query statements is zero; and the second execution unit is configured to execute multiple query statements in the target database based on a preset execution strategy to synchronize incremental data to the target database when the number of first query statements and the number of second query statements are both greater than zero.

[0087] According to an embodiment of the present disclosure, the second execution unit includes a first execution sub-unit and a second execution sub-unit. The first execution sub-unit is configured to sequentially execute at least one second query statement in the target database; and the second execution sub-unit is configured to sequentially execute at least one first query statement in the target database when the at least one second query statement is executed.

[0088] Figure 6 A block diagram of a data synchronization apparatus based on an incremental data synchronization component applied to a source device according to an embodiment of the present disclosure is schematically shown.

[0089] like Figure 6 As shown, a data synchronization device based on an incremental data synchronization component is set on a source device. The incremental data synchronization component includes a log parsing module and a consumption module. The device includes: a recording module 601, a log parsing module 602, an encapsulation module 603 and a sending module 604.

[0090] The recording module 601 is used to record the incremental data in the log of the source database to obtain the incremental log.

[0091] The log parsing module 602 is configured to parse the incremental log using the log parsing module in response to triggering a data synchronization event to obtain multiple query statements. The data synchronization event indicates that data synchronization is being performed on the target database of the target device.

[0092] The encapsulation module 603 is configured to encapsulate multiple query statements into a message.

[0093] The sending module 604 is used to send messages to the message queue so that the target device uses the consumption module to consume the messages in the message queue, obtain multiple query statements, and classify the multiple query statements to obtain classification results. Based on the classification results, multiple query statements are used to process the target database to synchronize the incremental data to the target database.

[0094] According to an embodiment of the present disclosure, incremental data is recorded in the log of the source database through a recording module to obtain an incremental log, so that when the source device responds to triggering a data synchronization event, the log parsing module can be used to parse the incremental log to obtain multiple query statements, and the multiple query statements can be encapsulated into messages using the encapsulation module and the messages can be sent to the message queue using the sending module, so that the target device can use the consumption module to obtain the above-mentioned query statements by consuming the messages in the message queue, and classify the above-mentioned query statements, and use the classified query statements to process the target database according to the corresponding execution strategy. The technical means of synchronizing the incremental data to the target database at least partially overcomes the technical problem that the native consumption module cannot consume query statements of data definition language and data operation language at the same time, thereby achieving the technical effect of assisting in ensuring data integrity during cross-domain data synchronization.

[0095] According to an embodiment of the present disclosure, the log parsing module 602 further includes an acquisition unit, a determination unit, and a parsing unit. The acquisition unit is configured to, in response to a data synchronization event being triggered, acquire a target data synchronization instance from the distributed cluster based on target domain information; the determination unit is configured to determine a target log parsing module and a target data table based on the target data synchronization instance; and the parsing unit is configured to utilize the target log parsing module to parse the target incremental log associated with the target data table in the incremental log to obtain multiple query statements.

[0096] According to an embodiment of the present disclosure, the sending module 604 includes a second determining unit and a sending unit, wherein the second determining unit is configured to determine a target message queue based on a target data synchronization instance; and the sending unit is configured to send a message to the target message queue in the message queue cluster.

[0097] According to the modules, submodules, units, and subunits of the embodiments of the present invention, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules, submodules, units, and subunits of the embodiments of the present invention, any one or more thereof can be split into multiple modules for implementation. According to the modules, submodules, units, and subunits of the embodiments of the present invention, any one or more thereof can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware, and firmware or in an appropriate combination of any of them. Alternatively, according to the modules, submodules, units, and subunits of the embodiments of the present invention, one or more thereof can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is run.

[0098] For example, any of the consumption module 501, classification module 502, processing module 503, or the recording module 601, parsing module 602, encapsulation module 603, and sending module 604 can be combined into one module / unit / sub-unit for implementation, or any of these modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functionality of one or more of these modules / units / sub-units can be combined with at least part of the functionality of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the consumption module 501, the classification module 502, the processing module 503 or the recording module 601, the parsing module 602, the encapsulation module 603 and the sending module 604 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the consumption module 501, the classification module 502, the processing module 503 or the recording module 601, the parsing module 602, the encapsulation module 603 and the sending module 604 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed.

[0099] Figure 7 A block diagram of an electronic device is schematically shown according to an embodiment of the present disclosure, which shows a data synchronization method based on an incremental data synchronization component. Figure 7 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0100] like Figure 7 As shown, the computer electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (such as an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for cache purposes. The processor 801 may include a single processing unit or multiple processing units for executing different actions of the method flow according to an embodiment of the present disclosure.

[0101] Various programs and data required for the operation of the electronic device 700 are stored in the RAM 703. The processor 701, ROM 702, and RAM 703 are connected to each other via a bus 704. The processor 701 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than the ROM 702 and RAM 703. The processor 701 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0102] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage portion 708 including a hard disk; and a communication portion 709 including a network interface card such as a LAN card or a modem. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 710 as needed, so that a computer program read therefrom can be installed into the storage portion 708 as needed.

[0103] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0104] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0105] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0106] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 702 and / or the RAM 703 described above and / or one or more memories other than the ROM 702 and the RAM 703 .

[0107] An embodiment of the present disclosure also includes a computer program product, which includes a computer program containing program code for executing the method provided by the embodiment of the present disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the data synchronization method based on the incremental data synchronization component provided by the embodiment of the present disclosure.

[0108] When the computer program is executed by the processor 701, the above functions defined in the system / device of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0109] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 709, and / or installed from a removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0110] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or coupled in various ways, and all such combinations and / or couplings fall within the scope of the present disclosure.

[0112] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data synchronization method based on an incremental data synchronization component, applied to a target device, wherein the incremental data synchronization component includes a log parsing module and a consumption module, and the method includes: consuming messages in a message queue using the consumption module to obtain a plurality of query statements, wherein the messages are sent to the message queue by a source-end device, the source-end device is configured to record incremental data in a log of a source-end database to obtain an incremental log, parsing the incremental log using the log parsing module to obtain the plurality of query statements, and then encapsulating the plurality of query statements to obtain the message; Classifying the multiple query statements based on the grammatical category of each of the multiple query statements to obtain a classification result; and Based on the classification result, using a plurality of the query statements to process a target-side database to synchronize the incremental data to the target-side database; Wherein, the grammar category includes a data definition language category and a data manipulation language category; The classifying of the plurality of query statements based on the grammatical category of each of the plurality of query statements to obtain a classification result includes: classifying the plurality of query statements into a first query statement belonging to the data definition language category and a second query statement belonging to the data manipulation language category; and The classification result is determined based on the number of the first query statements and the number of the second query statements.

2. The method according to claim 1, wherein The step of processing a target-side database based on the classification result using a plurality of query statements to synchronize the incremental data to the target-side database includes: When the number of the first query statements is zero or the number of the second query statements is 0, executing the plurality of query statements in sequence in the target database to synchronize the incremental data to the target database; and When the number of the first query statements and the number of the second query statements are both greater than 0, multiple query statements are executed in the target database based on a preset execution strategy to synchronize the incremental data to the target database.

3. The method according to claim 2, wherein: The executing the plurality of query statements in the target database based on a preset execution strategy includes: Execute at least one of the second query statements in sequence in the target database; and When at least one of the second query statements is executed, at least one of the first query statements is executed sequentially in the target database.

4. A data synchronization method based on an incremental data synchronization component, applied to a source device, wherein the incremental data synchronization component includes a log parsing module and a consumption module, and the method includes: Record incremental data in the log of the source database to obtain incremental logs; In response to triggering a data synchronization event, parsing the incremental log using the log parsing module to obtain multiple query statements, wherein the data synchronization event indicates that data synchronization is performed on a target-side database of a target-side device; Encapsulating the plurality of query statements into a message; as well as Sending the message to a message queue so that the target-end device consumes the message in the message queue using the consumption module, obtains the multiple query statements, classifies the multiple query statements to obtain classification results, and then processes the target-end database using the multiple query statements based on the classification results to synchronize the incremental data to the target-end database; The syntax category of the query statement includes a data definition language category and a data manipulation language category, and the target end device is specifically configured to classify the plurality of query statements into a first query statement belonging to the data definition language category and a second query statement belonging to the data manipulation language category; and determining the classification result based on the number of the first query statements and the number of the second query statements.

5. The method according to claim 4, wherein The data synchronization event carries target domain information, and the target domain information corresponds to the target end device; The method further comprises: In response to triggering the data synchronization event, obtaining a target data synchronization instance from a distributed cluster based on the target domain information; and Based on the target data synchronization instance, determine the target log parsing module and the target data table; The log parsing module is used to parse the incremental log to obtain multiple query statements, including: The target log parsing module is utilized to parse the target incremental log associated with the target data table in the incremental log to obtain a plurality of the query statements.

6. The method according to claim 5, further comprising: Determining a target message queue based on the target data synchronization instance; The step of sending the message to the message queue includes: The message is sent to the target message queue in the message queue cluster.

7. A data synchronization system based on an incremental data synchronization component, wherein the incremental data synchronization component includes a log parsing module and a consumption module, and the system includes: The source device is configured to record incremental data in a log of a source database to obtain an incremental log, and in response to triggering a data synchronization event, parse the incremental log using the log parsing module to obtain multiple query statements, encapsulate the multiple query statements into a message, and send the message to a message queue; The target-side device is configured to consume the messages in the message queue using the consumption module to obtain the multiple query statements, classify the multiple query statements to obtain classification results, and process the target-side database using the multiple query statements based on the classification results to synchronize the incremental data to the target-side database; as well as Message middleware, including one or more message queues; The syntax category of the query statement includes a data definition language category and a data manipulation language category, and the target end device is specifically configured to classify the plurality of query statements into a first query statement belonging to the data definition language category and a second query statement belonging to the data manipulation language category; and determining the classification result based on the number of the first query statements and the number of the second query statements.

8. A data synchronization device based on an incremental data synchronization component, provided on a target-end device, wherein the incremental data synchronization component includes a log parsing module and a consumption module, and the device includes: a consumption module, configured to consume messages in a message queue using the consumption module to obtain a plurality of query statements, wherein the messages are sent to the message queue by a source-end device, the source-end device being configured to record incremental data in a log of a source-end database to obtain an incremental log, and to parse the incremental log using the log parsing module to obtain the plurality of query statements, and then to encapsulate the plurality of query statements to obtain the message; a classification module, configured to classify the plurality of query statements based on the grammatical category of each of the plurality of query statements to obtain a classification result; and a processing module, configured to process a target-side database using a plurality of query statements based on the classification result, so as to synchronize the incremental data to the target-side database; Wherein, the grammar category includes a data definition language category and a data manipulation language category; Wherein, the classification module is used for: classifying the plurality of query statements into a first query statement belonging to the data definition language category and a second query statement belonging to the data manipulation language category; and The classification result is determined based on the number of the first query statements and the number of the second query statements.

9. A data synchronization device based on an incremental data synchronization component, provided on a source device, wherein the incremental data synchronization component includes a log parsing module and a consumption module, and the device includes: The recording module is used to record incremental data in the log of the source database to obtain incremental logs; a parsing module configured to parse the incremental log using the log parsing module in response to triggering a data synchronization event, to obtain a plurality of query statements, wherein the data synchronization event indicates data synchronization of a target-side database of a target-side device; An encapsulation module, configured to encapsulate the plurality of query statements into a message; as well as a sending module, configured to send the message to a message queue, so that the target-end device consumes the message in the message queue using the consumption module, obtains the multiple query statements, classifies the multiple query statements to obtain classification results, and then processes the target-end database using the multiple query statements based on the classification results to synchronize the incremental data to the target-end database; The syntax category of the query statement includes a data definition language category and a data manipulation language category, and the target end device is specifically configured to classify the plurality of query statements into a first query statement belonging to the data definition language category and a second query statement belonging to the data manipulation language category; and determining the classification result based on the number of the first query statements and the number of the second query statements.

10. An electronic device comprising: one or more processors; a memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors are enabled to implement the method of any one of claims 1 to 3 or the method of any one of claims 4 to 6.

11. A computer-readable storage medium having executable instructions stored thereon, wherein when the executable instructions are executed by a processor, the processor is enabled to implement the method according to any one of claims 1 to 3 or the method according to any one of claims 4 to 6.

12. A computer program product comprising computer executable instructions, wherein the computer executable instructions are used to implement the method of any one of claims 1 to 3 or the method of any one of claims 4 to 6 when executed.

Images