Method, system and computer readable storage medium for data synchronization

Abstract

The application provides a data synchronization method, system and computer readable storage medium, comprising: parsing a table structure change event recorded in a binary log of an upstream database to obtain a table change statement; in response to a judgment result that a row change event of a native table structure performed by a downstream database ends, storing a target table structure to a cache module; and based on the target table structure in the cache module, executing the table change statement in the downstream database to update the native table structure in the downstream database to the target table structure. By caching the native table structure in the cache module, the problem that the native table structure cannot be obtained when the table structure of the upstream database changes, so that the data synchronization is frequently interrupted, is avoided; in addition, by modifying a generation protocol of the binary log of the upstream database, the binary log can record more structure information involved in the table change event.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data synchronization method, system and computer readable storage medium. BACKGROUND

[0002] In order to solve the problems of data migration, data synchronization or data subscription of enterprises, major cloud vendors provide data transmission services for heterogeneous and homogeneous databases. The data transmission service is divided into three stages of structure migration, full data migration and incremental data migration. The incremental data migration mainly uses the feature that the addition, deletion and modification operations of the upstream database are recorded in the binary log, uses the upstream table structure to parse the addition, deletion and modification operation records in the binary log, and restores them into structured query language to execute in the downstream database.

[0003] However, when a relational database (such as MySQL) is used as an upstream database for data transmission service, the upstream database cannot well support table structure changes. Mainly reflected in: after the upstream database adds columns or deletes columns to the table structure, using the changed table structure to parse the old binary log will cause the synchronization task to fail, which needs to be manually checked and repaired; in order to reduce the impact of table structure changes on the performance of the upstream database, open source tools are often used to change the table structure, which will cause the data inconsistency between the upstream and downstream databases. Therefore, the frequent interruption of incremental data migration and the low efficiency of manual troubleshooting are the main defects of the current data transmission service. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art. To this end, one object of the present application is to provide a data synchronization method, system and computer readable storage medium capable of supporting table structure changes of the upstream database.

[0005] One aspect of the present application provides a data synchronization method, which can include: parsing a table structure change event recorded in a binary log of an upstream database to obtain a table change statement, wherein the table structure change event represents a change operation from an original table structure to a target table structure of the upstream database, and the table change statement is an operation language for changing the table structure of the database; in response to a judgment result that a row change event of the original table structure of the downstream database is executed, storing the target table structure to a cache module; and based on the target table structure in the cache module, executing the table change statement in the downstream database to update the original table structure of the downstream database to the target table structure.

[0006] In some embodiments, parsing the table structure change event recorded in the binary log of the upstream database to obtain the table change statement can include: determining whether an open source tool is used when changing the native table structure to the target table structure, wherein the open source tool is used to modify the native table structure; in response to a determination result that the open source tool is used when changing the native table structure to the target table structure, performing open source tool matching on a temporary table generated in the table structure change event according to a suffix of the temporary table; and in combination with the open source tool, analyzing the target table structure to obtain a table change statement of the target table structure, and caching the table change statement to the meta database.

[0007] In some embodiments, after analyzing the target table structure in combination with the open source tool to obtain the table change statement of the target table structure and caching the table change statement to the meta database, it can include: determining whether to execute the table change statement in the meta database; and in response to a determination result that the table change statement in the meta database is executed, opening a reading channel of the table change statement in the meta database.

[0008] In some embodiments, parsing the table structure change event recorded in the binary log of the upstream database to obtain the table change statement can include: determining whether an open source tool is used when changing the native table structure to the target table structure, wherein the open source tool is used to modify the native table structure; and in response to a determination result that the open source tool is not used when changing the native table structure to the target table structure, reading the table change statement from the binary log record.

[0009] In some embodiments, before parsing the table structure change event recorded in the binary log of the upstream database to obtain the table change statement, it can include: modifying a generation protocol of the binary log of the upstream database to make the binary log record a plurality of table structure information of the table structure change event, wherein the table structure information includes: column name information, column attribute information, table character set and column character set.

[0010] In some embodiments, modifying the generation protocol of the binary log of the upstream database to make the binary log record a plurality of table structure information of the table structure change event can include: analyzing the target table structure in the upstream database to determine the table structure information missing in the table structure change event of the binary log, wherein the table structure information includes: column name information, column attribute information, table character set and column character set; inserting meta information representing the table structure information at the end of the initial generation protocol of the binary log to obtain a target generation protocol of the binary log; and based on the target generation protocol, recording the table structure change event containing a plurality of table structure information in the binary log.

[0011] In some embodiments, after parsing the table structure change event recorded in the binary log of the upstream database to obtain the table change statement, the method can further include: determining an execution state of the row change event of the native table structure by the downstream database, wherein the execution state includes an execution end and an execution in progress; and when the execution state is the execution in progress, waiting for the downstream database to execute the row change event of the native table structure until the execution state changes to the execution end.

[0012] In some embodiments, before storing the target table structure into the cache module in response to the determination result that the execution of the row change event of the native table structure by the downstream database is ended, the method can further include: clearing the native table structure in the cache module.

[0013] The present application also proposes a data synchronization system, which can include a statement parsing module, a cache updating module, and a synchronization module. The statement parsing module is configured to parse a table structure change event recorded in a binary log of an upstream database to obtain a table change statement, wherein the table structure change event represents a change operation from a native table structure of the upstream database to a target table structure, and the table change statement is an operation language for changing the table structure of the database. The cache updating module is configured to store the target table structure into a cache module in response to a determination result that the execution of a row change event of the native table structure by a downstream database is ended. The synchronization module is configured to execute the table change statement in the downstream database based on the target table structure in the cache module, so as to update the native table structure in the downstream database to the target table structure.

[0014] In some embodiments, the execution steps of the statement parsing module can include: determining whether an open source tool is used when changing the native table structure to the target table structure, wherein the open source tool is configured to modify the native table structure; in response to a determination result that the open source tool is used when changing the native table structure to the target table structure, matching the open source tool to a temporary table generated in the table structure change event according to a suffix of the temporary table; and in combination with the open source tool, analyzing the target table structure to obtain a table change statement of the target table structure, and caching the table change statement into a meta database.

[0015] In some embodiments, after analyzing the target table structure in combination with the open source tool to obtain the table change statement of the target table structure and caching the table change statement into the meta database, the method can further include: determining whether to execute the table change statement in the meta database; and in response to a determination result that the table change statement in the meta database is executed, opening a reading channel of the table change statement in the meta database.

[0016] In some embodiments, the execution step of the statement parsing module can include: determining whether an open source tool is used when changing the native table structure to the target table structure, wherein the open source tool is used to modify the native table structure; and reading the table change statement from the binary log record in response to a determination that the open source tool is not used when changing the native table structure to the target table structure.

[0017] In some embodiments, the method further includes a protocol modifying module configured to modify a generation protocol of the binary log of the upstream database to cause the binary log to record a plurality of table structure information of a table change event, wherein the table structure information includes column name information, column attribute information, a table character set, and a column character set.

[0018] In some embodiments, the execution step of the protocol modifying module can include: analyzing the target table structure in the upstream database to determine table structure information that is missing in the table change event of the binary log, wherein the table structure information includes column name information, column attribute information, a table character set, and a column character set; inserting meta information representing the table structure information at the end of an initial generation protocol of the binary log to obtain a target generation protocol of the binary log; and recording the table change event including the plurality of table structure information in the binary log based on the target generation protocol.

[0019] In some embodiments, the method further includes an execution status determining module configured to determine an execution status of the row change event of the native table structure in the downstream database, wherein the execution status includes an execution end and an execution in progress; and when the execution status is the execution in progress, waiting for the downstream database to execute the row change event of the native table structure until the execution status is changed to the execution end.

[0020] In some embodiments, the method further includes a cache clearing module configured to clear the native table structure in the cache module.

[0021] The application also provides a computer readable storage medium, which stores a computer program adapted to be loaded into a processor to execute the steps of the data synchronization method.

[0022] According to the technical solutions of the above embodiments, at least one of the following beneficial effects can be achieved.

[0023] According to the data synchronization method, system, and computer readable storage medium, the native table structure is cached in the cache module, so that when the table structure of the upstream database is changed, the native table structure can be obtained, and the problem of frequent interruption of data synchronization is avoided. In addition, the generation protocol of the binary log of the upstream database is modified, so that the binary log can record more structure information involved in the table change event. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a flowchart of a delta data synchronization procedure of a related art;

[0025] Figure 2 is a flowchart of a method of data synchronization according to an aspect of the present application;

[0026] Figure 3 is a flowchart of a delta synchronization procedure;

[0027] Figure 4 is a flowchart of a method of modifying table structure by an open source tool;

[0028] Figure 5 is a flowchart of a table structure cache;

[0029] Figure 6 is a schematic diagram of a system structure of data synchronization according to another aspect of the present application;

[0030] Figure 7 is a schematic diagram of an architecture of a data transfer service according to yet another aspect of the present application;

[0031] Figure 8 is a schematic diagram of an electronic device structure according to yet another aspect of the present application; and

[0032] Figure 9 is a schematic diagram of a computer readable storage medium structure according to yet another aspect of the present application. DETAILED DESCRIPTION

[0033] For a better understanding of the present application, various aspects of the present application will be described in more detail below with reference to the accompanying drawings. It is to be understood that the detailed description is merely descriptive of exemplary embodiments of the present application and is not intended to limit the scope of the present application in any way. Throughout the specification, like reference numerals refer to like elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0034] It is to be noted that, in the present specification, the expressions first, second, third, etc. are used merely to distinguish one feature from another feature area, and do not represent any limitation on the features, especially do not represent any order of precedence. Therefore, the first file type discussed in the present application can also be referred to as the second file type, the first file level can also be referred to as the second file level, and vice versa, without departing from the teachings of the present application.

[0035] In the drawings, the thicknesses of components, sizes, and shapes can be exaggerated for clarity. The drawings are merely schematic and are not drawn to scale. As used in this document, the term “substantially” means that the recited characteristic, parameter, or value need not be achieved exactly, but that deviations, within measurement uncertainty, can occur. In addition, as used in this document the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise.

[0036] It should also be understood that any reference in this specification to something “according to” a disclosure refers to a disclosure of something that is at least partially included in that disclosure. It should also be understood that the use of the term “about” in this specification refers to something that is approximately the same as the recited value, but within a reasonable range of variation, such as a range of variation that would be recognized by one of ordinary skill in the art as being equivalent to the recited value. It should also be understood that the use of the term “comprises” or “comprising” in this specification refers to a recitation that is inclusive of the recited features, elements, or components, but not exclusive of one or more other features, elements, or components. In addition, when the term “at least one of” appears in a list of two or more items, it means that any combination of one or more of the listed items is included, but that the combination of the one or more items is not the only combination of the one or more items that is included. Furthermore, when describing embodiments of the application, the use of “can” means “one or more embodiments of the application.” Also, the term “exemplary” is intended to mean an example or an illustration.

[0037] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an overly literal or overly formal sense unless expressly so defined herein.

[0038] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other, as long as there is no conflict. In addition, unless specifically limited or clearly contradicted by context, the specific steps of the methods described in the present application can not be limited to the order described, but can be performed in any order or in parallel. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0039] In the DTS (Data Transmission Service), data in an upstream database is synchronized to a downstream database mainly in three stages of table structure migration, full data migration and incremental data synchronization. The table structure migration mainly executes a structured query language (SQL) statement of showing a table structure in the upstream database, such as a "show create table;" SQL statement, to obtain a current table structure and then execute in the downstream. The full data migration mainly executes a SQL statement of searching segmented data in the upstream database, such as a "select*from table limit xxx;" SQL statement, to scan out data in the upstream database in segments and splice into a SQL statement based on the current table structure and then execute in the downstream database. The incremental data synchronization mainly utilizes a feature that operations of adding, deleting and modifying in the MySQL are recorded in a binary log, parses the operations of adding, deleting and modifying recorded in the binary log using the current table structure of the upstream database, and restores the operations into a SQL statement to execute in the downstream database.

[0040] The technical means of the present application is proposed to solve the problem that the table structure change of the upstream database cannot be supported in the incremental data synchronization process, therefore, the incremental data synchronization process of the related art is briefly described below.

[0041] Figure 1 is a flowchart of the incremental data synchronization process of the related art. As shown in Figure 1 Step S110, a user Actor writes data in the upstream MySQL: insert into student(id, name, sex) values(1, "Xiaoming", "male"), that is, writes data "1, 'Xiaoming','male'" in the student table. Since the MySQL database has row changes, based on the feature that operations of adding, deleting and modifying in the MySQL are recorded in a binary log, when the data is changed, the binary log writes the row change event in step S120, that is, "id=1, name='Xiaoming', sex='male';". In step S130, the DTS reads the row change event in the binary log. In step S140, the DTS reads the current native table structure in the upstream database MySQL. In step S150, the DTS integrates the row change event in the binary log and the obtained native table structure and restores into a SQL statement: insert into student(id, name, sex) values(1, "Xiaoming", "male"), and finally writes the SQL statement into the downstream database, and the data synchronization is completed.

[0042] Further, step S131 adds a class column to the native table structure of the MySQL database, so that the native table structure is changed to the target table structure. When step S131 occurs between step S130 and step S140, the target table structure obtained has one more class column than the native table structure, so that the target table structure with the added class column cannot parse the row change operation in the binary log for the native table structure, and ultimately causes the synchronization task to fail.

[0043] Therefore, in order to solve the above synchronization interruption problem caused by the table structure change event, the present application proposes two improvements to the incremental data synchronization process of the current DTS: one is to modify the generation protocol of the binary log of the upstream database, so that the table structure change event is more completely recorded; the other is to add a cache module between the upstream database and the downstream database, to cache the current table structure of the upstream database during the data synchronization process, so that after the table structure of the upstream database changes, the current table structure can still be obtained and used to parse the table structure change event recorded in the binary log, avoiding synchronization interruption.

[0044] Figure 2 is a method flowchart of data synchronization according to an aspect of the present application. As shown in Figure 2 An aspect of the present application provides a data synchronization method, which can include: step S210, parsing a table structure change event recorded in the binary log of the upstream database to obtain a table change statement, wherein the table structure change event represents a change operation from an original table structure to a target table structure of the upstream database, and the table change statement is an operation language for changing the table structure of the database; step S220, storing the target table structure to the cache module in response to a judgment result of the end of the row change event of the native table structure of the downstream database; and step S230, executing the table change statement in the downstream database based on the target table structure in the cache module, so that the native table structure in the downstream database is updated to the target table structure.

[0045] In some embodiments, before step S210, the following step is further included: modifying the generation protocol of the binary log of the upstream database, so that the binary log records multiple table structure information of the table structure change event. Specifically, because the generation protocol of the current binary log of the upstream database lacks field mapping of column name information, column attribute information, table character set and column character set during the table structure change process, so that the binary log lacks the record of the column name information, the column attribute information, the table character set and the column character set when recording the table structure change event, therefore, in order to make the binary log more completely record the table structure information of the table structure change event of the upstream database, the generation protocol of the binary log of the upstream database needs to be modified.

[0046] More specifically, the target table structure in the upstream database is analyzed, and the table structure information missing in the table structure change event in the binary log is determined, such as column name information, column attribute information (for example, whether it is a primary key), table character set, and column character set. Further, meta information representing the table structure information is inserted at the end of the initial generation protocol of the binary log, and the target generation protocol of the binary log is obtained. Due to the update of the generation protocol of the binary log, the upstream database is converted from the MySQL 5.6 open source version to the Databus version. The Databus version database records more column information persistently on the basis of the open source version, such as column name information, column attribute information (for example, whether it is a primary key), table character set, and column character set. In order to be compatible with the open source version MySQL, the newly added meta information is inserted at the end of the initial generation protocol, and only the generation protocol of the binary log is modified, so the open source version MySQL can also be used as its slave and master. In the above manner, the binary log of the upstream database can record the table structure change event containing multiple table structure information.

[0047] Table 1

[0048]

[0049]

[0050] Table 1 is a binary log generation protocol table of the Databus version database, and the binary log generation protocol of the Databus version database is superimposed by the generation protocol table of the MySQL 5.6 open source version and the meta information representing the table structure information newly added by the Databus version. It can be seen that the meta information representing the table structure information is inserted at the end of the generation protocol table of the MySQL 5.6 open source version. In Table 1, the structure name, the number of characters occupied, and the description of the meta information of each table structure information correspond one by one, for example, the meta information of the table structure information with the structure name “event header” occupies 19 bytes, and the description thereof is: event header information, type TABLE_MAP_EVENT, and value 19. The meta information of the table structure information with the structure name “table id” occupies 6 bytes, and the description thereof is: table ID, and so on. The contents in Table 1 are not listed one by one here.

[0051] Figure 3 is an incremental synchronization flowchart. As Figure 3As shown, after the upstream database is updated from the MySQL 5.6 open source version to the Databus version, the incremental data synchronization process is started. Specifically, after the incremental transmission task is started, the DTS sends a connection creation request to the upstream database; after the connection is successfully created, the DTS sends a request to the upstream database to register the slave, for registering the meta information necessary for pulling data; further, after the slave is successfully registered, the DTS sends a request to the upstream database to pull the binary log, to obtain the data packets about the addition, deletion and modification of the upstream database in the binary log; still further, the data packets in the binary log are parsed and converted, and are restored into SQL statements recognizable by the DTS, and are then executed in the downstream database.

[0052] Specifically, the data packets in the binary log include table change events, row change events and other events. The table change events are used to represent the table structure information when the data changes; the row change events are used to represent the specific operations of the row data change, such as the insertion, deletion and update of the row data, etc.; the other events can include table structure change events, used to record the table change statements of the table structure, such as the SQL statements of adding a column, deleting a column, etc. The DTS updates the table structure events to the cache module, and writes the row change events and the other events into the built-in reading module; further, the reading module reads the table change events in the cache module. Then, the table change events, the row change events and the other events are integrated by the scheduling module, to obtain the SQL statements recognizable by the DTS, which are executed in the downstream database by the writing module.

[0053] In some embodiments, in the actual DTS, the more common scenarios mainly include: synchronizing the data of a relational database to a relational database, synchronizing the data of a relational database to a message queue, synchronizing the data of a relational database to a non-relational database, and the like. Specifically, synchronizing the data of a relational database to a relational database can for example synchronize the data of a relational database management system MySQL to a relational database management system MySQL, which is mainly used for cluster splitting, i.e., due to the increase of data volume, the upstream MySQL disk storage space is not enough, or the business volume is too large, limited by the write bottleneck of a single instance, so it is necessary to split one MySQL into multiple MySQLs. Synchronizing the data of a relational database to a message queue can for example synchronize the data of a MySQL to an open source stream processing platform Kafka, which is mainly used for subscribing to database data changes, and the business only needs to consume the data in Kafka, which can be used for real-time analysis, business monitoring and the like. Synchronizing the data of a relational database to a non-relational database can for example synchronize the data of a MySQL to a distributed real-time search and analysis engine (Elasticsearch, ES), which is used for real-time search, real-time analysis and the like. Therefore, in the present application, the downstream database can be a relational database management system MySQL, an open source stream processing platform Kafka and a distributed real-time search and analysis engine ES.

[0054] Obviously, in the incremental data synchronization process described above, the cache module is provided to cache the table structure event, and the table structure caching process will be described in more detail below.

[0055] In some embodiments, when modifying the native table structure of the upstream database (e.g., MySQL), the table change operation can be directly performed on the native table structure, but in order to avoid the problem of performance decline of the upstream database caused by changing the native table structure of the upstream database, an open source tool is usually used to perform the table change operation, which is mainly used to modify the native table structure in the present application.

[0056] Figure 4 is a method flowchart of the open source tool modifying the table structure. As Figure 4As shown, when the table structure change is performed using the open source table structure change tool, the process mainly includes a data replication stage and a modification completion stage. In the data replication stage, a new table structure of the upstream database MySQL is created, for example, when the table structure change event is to add a new column class to the original table structure, a new table student_new including the column class is created. Specifically, assuming that the original table structure includes an id int column, a name varchar(20) column, and a sex varchar(20) column, the new table structure will have an id int column, a name varchar(20) column, a sex varchar(20) column, and a class varchar(20) column, and of course there is no data content in the new table at this time. Further, the data content in the original table is copied to the new table by a certain means. Further, when the data content in the original table and the data content in the new table are basically consistent, the original table can be prohibited from data writing and other operations by locking the original table, and waiting for the data content of the new table and the original table to be completely consistent. Finally, the new table and the original table are renamed, the original table name is covered by the new table name, and the original table is renamed, and finally the new table with the original table name is obtained, that is, the target table structure. For example, the new table is renamed as student, and the original table structure is renamed as student_old. It should be noted that as long as the name of the original table is covered by the name of the new table, and the name of the original table is changed, the specific name content is not limited. Of course, the above operations are recorded in the binary log of the upstream database.

[0057] Obviously, when the open source tool is used to modify the original table structure of the upstream database, multiple temporary tables such as student_old and student_new will be generated. If the temporary tables are synchronized to the downstream database, it will cause trouble to the user, and the data consistency of the synchronization process is difficult to guarantee, so as to cause the failure of data synchronization. Therefore, according to the table structure change event recorded in the binary log, the table change statement using the open source tool to change the table structure is parsed, that is, the SQL statement about the table change; and then the table change statement is directly synchronized to the downstream database, so as to solve the above problems.

[0058] Figure 5 is a flowchart of the table structure cache. As shown in Figure 5 , at the beginning of the incremental synchronization stage, the original table structure to be synchronized is obtained in the upstream database MySQL (Databus version) by the "show column from table;" statement, and is cached in the cache module for table structure cache.

[0059] Further, the binary log of the upstream database is read, and the data packets in the binary log include table structure events, row change events, and table structure change events. Specifically, an operation of one data in the binary log is referred to as an event, a table change event is used to represent table structure information when data changes; a row change event is used to represent specific operations of row data changes, such as row data insertion, deletion, and update operations; and other events can include a table structure change event, which is used to record table change statements of the table structure, such as adding a column, deleting a column, and the like.

[0060] Further, because the target table structure obtained by using the open source tool is the result of the table structure change operation on the constructed new table structure, rather than the result of the table structure change directly based on the original table structure. Therefore, when the table structure change event exists in the binary log, it is necessary to determine whether the open source tool is used when the original table structure change is performed. If the open source tool is used when the original table structure change is performed, the table change statement for changing the original table structure to the target table structure needs to be parsed, and if the table structure change operation is not performed by the open source tool, the table change statement can be directly read from the binary log.

[0061] Specifically, if the original table structure change operation of the upstream database is completed through an open-source tool, the temporary tables (such as student_old and student_new) generated in the table structure change event are matched with the open-source tool based on their suffixes. For example, "_ghc" and "_gho" are the table name suffixes of the temporary tables in the open-source tool gh-ost, which is an online tool for changing the MySQL table structure open-sourced by GitHub. Furthermore, the target table structure is analyzed in conjunction with the specific open-source tool, and the table change statement that changes the original table structure to the target table structure is parsed out, such as "alter table add column add class varchar(20) comment 'Class Name';", and the table change statement is cached in the meta-database, where the meta-database is used to cache the table change statement parsed from the table structure change event based on the binary log. Further, it determines whether the table change statements cached in the metadata database need to be executed, i.e., whether all table structure change events recorded in the binary log have been parsed. If not, the above steps are repeated without further operation; if all table structure change events have been parsed, the reading channel for table change statements in the metadata database is opened. Further, it determines the execution status of row change events for the native table structure in the downstream database, including execution completed and execution in progress. When the result shows the execution status as "in progress," meaning the current row change event has not been completed in the downstream database, DTS will wait for the downstream database to execute the row change events for the native table structure until the execution status changes to "execution completed." During this period, DTS will not perform any operation. If the row change events for the native table structure in the cache module have been completed in the downstream database (i.e., the execution status has changed to "execution completed"), the native table structure stored in the cache module is cleared, and the target table structure is updated in the cache module. Furthermore, the table modification statement is read from the metadata database and executed downstream, for example, "alter table add column add class varchar(20) comment 'Class Name';".

[0062] In some embodiments, if the table structure change operation is not performed by the open source tool, the execution state of the row change event of the native table structure in the downstream database is determined, wherein the execution state includes execution end and execution in progress. When the result of the determination shows that the execution state is execution in progress, that is, the current row change event is not executed in the downstream database, at this time, the DTS will wait for the execution of the row change event of the native table structure in the downstream database to change the execution state to execution end, during which the DTS will not perform any operation; if the row change event of the native table structure in the cache module has been executed in the downstream database, that is, the execution state is changed to execution end, the native table structure stored in the cache module is cleaned up, and the target table structure is updated to the cache module; further, the table change statement is read in the table structure change event of the binary log, and the statement is executed in the downstream database, for example, "alter table add column add class varchar (20) comment 'class name';".

[0063] In some embodiments, the downstream database of the incremental data synchronization process can be a relational database MySQL, an open source stream processing platform Kafka, and a non-relational database ES, etc.

[0064] According to the data synchronization method of the present application, by caching the native table structure in the cache module, the problem that the native table structure cannot be obtained when the table structure of the upstream database is changed, so that the data synchronization is frequently interrupted, is avoided; in addition, by modifying the generation protocol of the binary log of the upstream database, the binary log can record more structure information involved in the table change event.

[0065] Figure 6 is a schematic diagram of the system structure of data synchronization according to another aspect of the present application. As shown in Figure 6 The present application also provides a data synchronization system, which can include: a statement analysis module 100, a cache update module 200, and a synchronization module 300. The statement analysis module 100 is used to analyze the table structure change event recorded in the binary log of the upstream database to obtain a table change statement, wherein the table structure change event represents the change operation of the original table structure of the upstream database to the target table structure, and the table change statement is an operation language for changing the table structure of the database. The cache update module 200 is used to store the target table structure to the cache module in response to the determination result of the execution end of the row change event of the native table structure in the downstream database. The synchronization module 300 is used to execute the table change statement in the downstream database based on the target table structure in the cache module, so that the native table structure in the downstream database is updated to the target table structure.

[0066] In some embodiments, the execution step of the statement parsing module 100 can include: determining whether an open source tool is used when changing the native table structure to the target table structure, wherein the open source tool is used to modify the native table structure; in response to the determination result that the open source tool is used when changing the native table structure to the target table structure, performing matching of the open source tool on the temporary table according to the suffix of the temporary table generated in the table structure change event; and in combination with the open source tool, analyzing the target table structure, obtaining the table change statement of the target table structure, and caching the table change statement to the meta database.

[0067] In some embodiments, after the step of analyzing the target table structure in combination with the open source tool, obtaining the table change statement of the target table structure, and caching the table change statement to the meta database, the execution step of the statement parsing module 100 can include: determining whether to execute the table change statement in the meta database; and in response to the determination result that the table change statement in the meta database is executed, opening a read channel of the table change statement in the meta database.

[0068] In some embodiments, the execution step of the statement parsing module 100 can include: determining whether an open source tool is used when changing the native table structure to the target table structure, wherein the open source tool is used to modify the native table structure; and in response to the determination result that the open source tool is not used when changing the native table structure to the target table structure, reading the table change statement from the binary log record.

[0069] In some embodiments, a protocol modification module (not shown) is further included for modifying a generation protocol of the binary log of the upstream database, so that the binary log records the plurality of table structure information of the table structure change event, wherein the table structure information includes: column name information, column attribute information, table character set, and column character set.

[0070] In some embodiments, the execution step of the protocol modification module can include: analyzing the target table structure in the upstream database to determine the table structure information that is missing in the table structure change event of the binary log, wherein the table structure information includes: column name information, column attribute information, table character set, and column character set; inserting meta information representing the table structure information at the end of the initial generation protocol of the binary log to obtain a target generation protocol of the binary log; and based on the target generation protocol, recording the table structure change event containing the plurality of table structure information in the binary log.

[0071] In some embodiments, an execution status determination module (not shown) is further included for determining an execution status of the downstream database on the row change event of the native table structure, wherein the execution status includes execution end and execution in progress; and when the execution status is execution in progress, waiting for the downstream database to execute the row change event of the native table structure until the execution status is changed to execution end.

[0072] In some embodiments, a cache clearing module (not shown) is further included for clearing the native table structure in the cache module.

[0073] According to the data synchronization system of the present application, by caching the native table structure in the cache module, the problem that the native table structure cannot be obtained when the table structure of the upstream database is changed, so that the data synchronization is frequently interrupted, is avoided. In addition, by modifying the generation protocol of the binary log of the upstream database, the binary log can record more structural information involved in the table change event.

[0074] Figure 7 is a schematic diagram of an architecture of a data transmission service according to still another aspect of the present application. As shown in Figure 7 The overall architecture of the DTS mainly includes a front-end module, a server module and a task synchronization module. The front-end module is used for information interaction with users and provides an interactive visual configuration information filling platform for users. The server module mainly includes a high availability module, a communication module, a data verification module, a pre-check module, a task allocation module, a task management module and a meta database. The server module mainly performs task allocation, configuration scheduling and the like. The task synchronization module includes a cache module, a data matching module, a reading module, a filtering module, a scheduling module, a routing module and a writing module and the like. The task synchronization module is used for specific data synchronization, including database and table data synchronization operations. The upstream database has a master database and a slave database. After modification of the generation protocol of the binary log, the Databus database can be obtained. The binary log of the database can more completely record the table structure change event including the table structure information. The data of the Databus database is synchronized to various downstream databases, including the relational database MySQL, the open source stream processing platform Kafka and the non-relational database ES and the like.

[0075] Obviously, in the architecture of the data transmission service, the generation protocol of the binary log of the upstream database is adjusted to obtain the Databus database. In addition, the cache module is added to the task synchronization module to cache the table structure being synchronized by the upstream. For the above two improvements, the method of data synchronization described in an embodiment of the present application can be referred to, which will not be repeated here.

[0076] Figure 8 is a schematic diagram of an electronic device structure according to still another aspect of the present application. As shown in Figure 8 According to still another aspect of the present application, an electronic device is further provided. The electronic device can include one or more processors and one or more memories. The memory has computer readable code stored therein, which, when executed by the one or more processors, can perform the method of determining the target three-dimensional position as described above.

[0077] The method or apparatus according to the embodiments of the present application can also be implemented by means of Figure 8 The architecture of the electronic device is shown. As Figure 8 shown, the electronic device can include a bus 401, one or more CPUs 402, a read-only memory (ROM) 403, a random access memory (RAM) 404, a communication port 405 connected to a network, an input / output component 406, a hard disk 407, etc. The storage device in the electronic device, such as the ROM 403 or the hard disk 407, can store the method of data synchronization provided by the present application. The method of data synchronization may, for example, include parsing a table structure change event recorded in a binary log of an upstream database, obtaining a table change statement, wherein the table structure change event represents a change operation from an original table structure to a target table structure of the upstream database, and the table change statement is an operation language for changing the table structure of the database; in response to a result of a judgment that a row change event of the native table structure of the downstream database is executed, storing the target table structure to a cache module; and based on the target table structure in the cache module, executing the table change statement in the downstream database, so that the native table structure of the downstream database is updated to the target table structure. Further, the electronic device can also include a user interface 408. Of course, Figure 8 The architecture shown is only exemplary, and when implementing different devices, one or more components in the electronic device shown can be omitted Figure 8 according to actual needs.

[0078] Figure 9 is a computer readable storage medium structure diagram according to still another aspect of the present application. As Figure 9 shown, is a computer readable storage medium 500 according to an embodiment of the present application. The computer readable storage medium 500 stores computer readable instructions. When the computer readable instructions are run by a processor, the method of data synchronization according to the embodiments of the present application described with reference to the above figures can be executed. The storage medium 500 includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and cache memory, etc. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, etc.

[0079] In addition, according to the embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium storing machine-readable instructions executable by a processor to perform instructions corresponding to the method steps provided by the present application, for example: parsing a table structure change event recorded in a binary log of an upstream database, obtaining a table change statement, wherein the table structure change event represents a change operation from an original table structure to a target table structure of the upstream database, and the table change statement is an operation language for changing a database table structure; in response to a judgment result that a row change event of the native table structure is executed by the downstream database, storing the target table structure to a cache module; and based on the target table structure in the cache module, executing the table change statement in the downstream database, so that the native table structure in the downstream database is updated to the target table structure. When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present application are executed.

[0080] The method and apparatus, device of the present application can be implemented in many ways. For example, the method and apparatus, device of the present application can be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present application are not limited to the above specifically described order, unless otherwise specifically described. In addition, in some embodiments, the present application can also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the method according to the present application. Thus, the present application also covers a recording medium storing a program for executing the method according to the present application.

[0081] In addition, the part of the above technical solutions provided in the embodiments of the present application that is consistent with the implementation principle of the corresponding technical solutions in the prior art is not described in detail, so as not to be too verbose.

[0082] The above description is only for the embodiments of the present application and the explanation of the technical principles used. Those skilled in the art should understand that the scope of protection involved in the present application is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the technical concept. For example, the technical solutions formed by mutually replacing the above features with the technical features disclosed in the present application (but not limited to) having similar functions.

Claims

1. A method of data synchronization, characterized by, The method comprises: parsing a table structure change event recorded in a binary log of an upstream database to obtain a table change statement, wherein the table structure change event represents a change operation from a native table structure of the upstream database to a target table structure, and the table change statement is an operation language for changing a database table structure; in response to a result of judging that a row change event of the native table structure is executed in a downstream database, storing the target table structure to a cache module; and based on the target table structure in the cache module, executing the table change statement in the downstream database to update the native table structure in the downstream database to the target table structure. The parsing of the table structure change event recorded in the binary log of the upstream database to obtain the table change statement comprises: judging whether an open source tool is used when the native table structure is changed to the target table structure, wherein the open source tool is used to modify the native table structure; in response to a result of judging that the open source tool is used when the native table structure is changed to the target table structure, matching the open source tool to a temporary table generated in the table structure change event according to a suffix of the temporary table; and combining the open source tool to analyze the target table structure to obtain a table change statement of the target table structure, and caching the table change statement to a meta database.

2. The data synchronization method of claim 1, wherein, After the combining of the open source tool to analyze the target table structure to obtain a table change statement of the target table structure, and caching the table change statement to a meta database, the method comprises: judging whether to execute the table change statement in the meta database; and in response to a result of judging to execute the table change statement in the meta database, opening a reading channel of the table change statement in the meta database.

3. The data synchronization method of claim 1, wherein, The parsing of the table structure change event recorded in the binary log of the upstream database to obtain the table change statement comprises: judging whether an open source tool is used when the native table structure is changed to the target table structure, wherein the open source tool is used to modify the native table structure; and in response to a result of judging that the open source tool is not used when the native table structure is changed to the target table structure, reading a table change statement from the binary log record.

4. The data synchronization method according to any one of claims 1 to 3, characterized in that, Before the parsing of the table structure change event recorded in the binary log of the upstream database to obtain the table change statement, the method comprises: modifying a generation protocol of a binary log of an upstream database to make the binary log record a plurality of table structure information of a table structure change event, wherein the table structure information comprises column name information, column attribute information, table character set and column character set.

5. The data synchronization method of claim 4, wherein, The modifying of the generation protocol of the binary log of the upstream database to make the binary log record a plurality of table structure information of a table structure change event comprises: analyzing the target table structure in the upstream database to determine table structure information that is missing in the table structure change event of the binary log, wherein the table structure information comprises column name information, column attribute information, table character set and column character set; inserting meta information representing the table structure information at the end of the initial generation protocol of the binary log, obtaining a target generation protocol of the binary log; and based on the target generation protocol, recording the table structure change event containing a plurality of the table structure information into the binary log.

6. The data synchronization method of claim 1, wherein, After parsing the table structure change event recorded in the binary log of the upstream database to obtain a table change statement, the method comprises: judging an execution state of the row change event of the native table structure of the downstream database, wherein the execution state comprises execution end and execution in progress; and when the execution state is execution in progress, waiting for the downstream database to execute the row change event of the native table structure until the execution state changes to execution end.

7. The data synchronization method of claim 1, wherein, Before storing the target table structure into the cache module in response to the judgment result of the execution end of the row change event of the native table structure of the downstream database, the method comprises: clearing the native table structure in the cache module.

8. A system for data synchronization, characterized by The method comprises: a statement parsing module, configured to parse the table structure change event recorded in the binary log of the upstream database to obtain a table change statement, wherein the table structure change event represents a change operation from a native table structure of the upstream database to a target table structure, and the table change statement is an operation language for changing the table structure of the database; and judge whether an open source tool is used when changing the native table structure to the target table structure, wherein the open source tool is used to modify the native table structure; in response to the judgment result of using the open source tool when changing the native table structure to the target table structure, matching the open source tool to a temporary table generated in the table structure change event according to a suffix of the temporary table; and combining the open source tool, analyzing the target table structure to obtain a table change statement of the target table structure, and caching the table change statement to a meta database; a cache updating module, configured to store the target table structure into a cache module in response to the judgment result of the execution end of the row change event of the native table structure of the downstream database; and a synchronization module, configured to execute the table change statement in the downstream database based on the target table structure in the cache module, so as to update the native table structure in the downstream database to the target table structure.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is adapted to be loaded by a processor to execute the steps in the data synchronization method according to any one of claims 1-7.

Images