A data synchronization method, system, device and readable storage medium

Abstract

The application provides a data synchronization method, system and device and a readable storage medium, wherein the data synchronization method comprises the following steps: reading a synchronization condition preset by a user; performing syntax analysis on the synchronization condition to generate an abstract syntax tree; extracting a condition part in the abstract syntax tree as a judgment template to judge to-be-judgment data to obtain a judgment result, wherein the to-be-judgment data is incremental data obtained by analyzing a log file of a source database; and if the judgment result indicates that the to-be-judgment data meets the synchronization condition, synchronizing the to-be-judgment data to a target database. The method is used for realizing the efficiency of condition-based synchronization of incremental data.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data synchronization method, system, apparatus, and readable storage medium. Background Technology

[0002] With societal development and technological advancements, users increasingly rely on network systems to handle various transactions. The normal operation of these network systems depends on the stable functioning of data center systems. If data is lost for any reason, certain functions will be paralyzed, causing incalculable losses to users. To ensure data security in the source database, backups are typically used to guarantee data continuity. This involves remotely synchronizing data from the source database on a server to the target database, so that data from the target database can be used in the event of a server failure.

[0003] How to achieve data synchronization has become an urgent technical problem to be solved. Summary of the Invention

[0004] This invention provides a data synchronization method, system, apparatus, and readable storage medium for achieving efficient incremental data synchronization based on conditions.

[0005] In a first aspect, embodiments of the present invention provide a data synchronization method, comprising:

[0006] Read the user-preset synchronization conditions;

[0007] The synchronization conditions are parsed to generate an abstract syntax tree;

[0008] The condition part in the abstract syntax tree is extracted as a judgment template, and the data to be judged is judged to obtain the judgment result. The data to be judged is incremental data obtained by parsing the log data of the source database.

[0009] If the determination result indicates that the data to be determined meets the synchronization condition, then the data to be determined is synchronized to the target database.

[0010] In one possible implementation, extracting the conditional portion from the abstract syntax tree as a decision template, and using it to determine the data to be determined to obtain a decision result includes:

[0011] Identify the field names and literal constants included in the determination template;

[0012] Determine whether the field names of the data to be judged are the same as the field names included in the judgment template;

[0013] If they are the same, then the corresponding field name in the judgment template is replaced with the literal constant corresponding to the field name of the data to be judged;

[0014] If all field names in the judgment template are replaced, a condition tree is generated and the condition tree is converted into an expression that only includes literal constants;

[0015] Obtain the evaluation result of the expression, and use the evaluation result as the judgment result.

[0016] In one possible implementation, after replacing the corresponding field name in the determination template with a literal constant corresponding to the field name of the data to be determined, the method further includes:

[0017] If at least one field name in the determination template is not replaced, the determination result indicates that the data to be determined does not meet the synchronization condition.

[0018] In one possible implementation, determining whether the field name of the data to be determined is the same as the field name included in the determination template includes:

[0019] The data to be judged is represented by an array comprising multiple structures, wherein each structure includes a corresponding field name and a literal constant;

[0020] Iterate through the array and determine whether the field names included in the array are the same as the field names included in the determination template.

[0021] Secondly, embodiments of the present invention also provide a data synchronization system, comprising:

[0022] A source database, a target database, and synchronization worker nodes that interact with the source database and the target database respectively; wherein: the synchronization worker nodes are configured as follows:

[0023] Read the user-preset synchronization conditions;

[0024] The synchronization conditions are parsed to generate an abstract syntax tree;

[0025] The condition part in the abstract syntax tree is extracted as a judgment template, and the data to be judged is judged to obtain the judgment result. The data to be judged is incremental data obtained by parsing the log file of the source database.

[0026] If the determination result indicates that the data to be determined meets the synchronization condition, then the data to be determined is synchronized to the target database.

[0027] Thirdly, embodiments of the present invention also provide a data synchronization device, comprising:

[0028] The reading unit is used to read the user-preset synchronization conditions;

[0029] A generation unit is used to perform syntax parsing on the synchronization conditions and generate an abstract syntax tree;

[0030] The judgment unit is used to extract the condition part in the abstract syntax tree as a judgment template, judge the data to be judged, and obtain the judgment result. The data to be judged is incremental data obtained by parsing the log file of the source database.

[0031] A synchronization unit is used to synchronize the data to be judged to the target database if the judgment result indicates that the data to be judged meets the synchronization condition.

[0032] In one possible implementation, the determination unit is used to:

[0033] Identify the field names and literal constants included in the determination template;

[0034] Determine whether the field names of the data to be judged are the same as the field names included in the judgment template;

[0035] If they are the same, then the corresponding field name in the judgment template is replaced with the literal constant corresponding to the field name of the data to be judged;

[0036] If all field names in the judgment template are replaced, a condition tree is generated and the condition tree is converted into an expression that only includes literal constants;

[0037] Obtain the evaluation result of the expression, and use the evaluation result as the judgment result.

[0038] In one possible implementation, after the determination unit replaces the corresponding field name in the determination template with the literal constant corresponding to the field name of the data to be determined, it is further used for:

[0039] If at least one field name in the determination template is not replaced, the determination result indicates that the data to be determined does not meet the synchronization condition.

[0040] In one possible implementation, the determination unit is used to:

[0041] The data to be judged is represented by an array comprising multiple structures, wherein each structure includes a corresponding field name and a literal constant;

[0042] Iterate through the array and determine whether the field names included in the array are the same as the field names included in the determination template.

[0043] Fourthly, embodiments of the present invention also provide a data synchronization device, the data synchronization device including a processor, the processor being configured to execute a computer program stored in a memory to implement the steps of the data synchronization method as described in any of the preceding claims.

[0044] Fifthly, embodiments of the present invention also provide a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data synchronization method as described in any of the preceding claims.

[0045] The beneficial effects of this invention are as follows:

[0046] This invention provides a data synchronization method, system, apparatus, and readable storage medium. First, user-preset synchronization conditions are read. Then, the synchronization conditions are parsed to generate an abstract syntax tree (AST). Next, the conditional portion of the AST is extracted as a decision template to determine the data to be determined, obtaining a decision result. The data to be determined is incremental data obtained by parsing log files from the source database. This allows for the determination of whether the incremental data obtained by directly parsing log files needs synchronization based on the decision template extracted from the AST. The entire determination process does not require reverse database lookups or database engine calls. When the determination result indicates that the synchronization conditions are met, the incremental data to be determined is synchronized to the target database. This improves the efficiency of conditional synchronization of incremental data based on log parsing and avoids the problem of historical data no longer existing in the current database, making reverse lookups impossible. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of one possible structure of a data synchronization system to which a data synchronization method provided in an embodiment of the present invention is applicable;

[0048] Figure 2 A flowchart of a data synchronization method provided in an embodiment of the present invention;

[0049] Figure 3 for Figure 1 Flowchart of one method for step S103;

[0050] Figure 4 for Figure 3 Flowchart of one method for step S202;

[0051] Figure 5 This is a schematic diagram of one structure of the abstract syntax tree in a data synchronization method provided by an embodiment of the present invention;

[0052] Figure 6This is a schematic diagram of one structure of the determination template in a data synchronization method provided by an embodiment of the present invention;

[0053] Figure 7 This is a schematic diagram of one possible structure of a condition tree in a data synchronization method provided by an embodiment of the present invention;

[0054] Figure 8 This is a schematic diagram of one structure of a data synchronization system provided in an embodiment of the present invention;

[0055] Figure 9 This is a schematic diagram of one structure of a data synchronization device provided in an embodiment of the present invention. Detailed Implementation

[0056] In the specification, claims, and accompanying drawings of this invention, the terms "first," "second," etc., are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0057] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0058] To better understand the above technical solutions, the technical solutions of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solutions of the present invention, rather than limitations on the technical solutions of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0059] Currently, mainstream database incremental synchronization technologies are either based on parsing source logs or implemented using the database's own Application Programming Interface (API). However, implementing conditional synchronization is cumbersome and lacks flexibility in combining conditions.

[0060] In view of this, embodiments of the present invention provide a data synchronization method, system, apparatus, and readable storage medium for achieving efficient incremental data synchronization based on conditions.

[0061] Before introducing the data synchronization method provided in the embodiments of the present invention, a brief description of the data synchronization system to which this data synchronization method is applicable will be given. For example... Figure 1 The diagram shown illustrates one possible structure of a data synchronization system. (Combined with...) Figure 1 As shown, the data synchronization system includes a source database, a target database, a management node cluster, a synchronization node cluster, and a coordinator cluster. The source database supports distributed relational databases, distributed file systems, and unstructured databases. The target database also supports distributed relational databases, distributed file systems, and unstructured databases. The synchronization node cluster includes at least one synchronization node, and each synchronization node includes at least one task node. The management node cluster is used for data verification configuration, pushing the data verification configuration to the verification nodes; it also receives data synchronization status and progress information from the verification nodes. The synchronization node cluster is the module that executes the specific data verification process. The coordinator cluster is the module that coordinates the data verification. The "synchronization worker node" mentioned in this embodiment can be one of the modules in the synchronization node cluster. This data synchronization system can read the synchronization conditions set by the user, perform syntax parsing on these conditions to generate an abstract syntax tree (AST), and extract the conditional parts from the AST as a decision template. Based on this template, it can then evaluate the incremental data in the source database to be evaluated. The system determines whether the incremental data meets the synchronization conditions, and if so, synchronizes it to the target database. This achieves the technical effect of synchronizing incremental data according to conditions. Figure 1 The specific implementation process of data synchronization in the data synchronization system can be found in the relevant sections below, and will not be detailed here.

[0062] like Figure 2 As shown, this embodiment of the invention provides a data synchronization method, including:

[0063] S101: Read the user-preset synchronization conditions;

[0064] In practice, the synchronization conditions set by the user on the data synchronization system can be read. These synchronization conditions need to meet the basic Structured Query Language (SQL) syntax and not support functions specific to certain database types, such as the CAST function in PostgreSQL and the TO_DATE function in Oracle.

[0065] S102: Perform syntax parsing on the synchronization conditions to generate an abstract syntax tree;

[0066] In practical implementation, open-source offline SQL parsing tools can be invoked to parse the synchronization conditions and generate an abstract syntax tree. For example, open-source SQL parsing engines such as Antlr and Druid can be used to parse the synchronization conditions and generate an abstract syntax tree, which can also identify metadata such as database name, table name, field name, field alias, WHERE clause, and conditions. In this way, the data synchronization method provided in this embodiment of the invention is compatible with common SQL syntax.

[0067] S103: Extract the condition part from the abstract syntax tree as a judgment template, judge the data to be judged, and obtain the judgment result. The data to be judged is incremental data obtained by parsing the log file of the source database.

[0068] In practical implementation, the syntax tree corresponding to the condition part in the abstract syntax tree can be extracted as a decision template. Then, the data to be decided is judged according to this decision template to obtain the decision result. The data to be decided can be incremental data obtained by parsing the log file of the source database. The incremental data can be data written after the current time, where the current time can be the time when the source database transmits the data to be decided in real time. Since the decision template extracted from the abstract syntax tree can determine whether the incremental data to be decided obtained by directly parsing the log file needs to be synchronized, the entire decision process does not require reverse database lookup or database engine calls. This improves the efficiency of conditional synchronization of incremental data based on log parsing and avoids the problem that some historical data no longer exists in the current database and cannot be retrieved.

[0069] S104: If the determination result indicates that the data to be determined meets the synchronization condition, then the data to be determined is synchronized to the target database.

[0070] In practical implementation, after obtaining the judgment result, it can be used to determine whether the data to be judged meets the synchronization conditions. If the judgment result shows that the data to be judged meets the synchronization conditions, then the data to be judged is synchronized to the target database, thereby achieving the technical effect of synchronizing incremental data according to conditions. Moreover, the entire synchronization process has no additional network interaction overhead, improving synchronization efficiency. Furthermore, it can also filter historical data according to conditions without needing to look up the database, greatly improving the efficiency of incremental conditional synchronization based on log parsing.

[0071] In embodiments of the present invention, such as Figure 3As shown, step S103: Extract the condition part from the abstract syntax tree as a decision template, make a decision on the data to be decided, and obtain the decision result, including:

[0072] S201: Identify the field names and literal constants included in the determination template;

[0073] S202: Determine whether the field names of the data to be judged are the same as the field names included in the judgment template;

[0074] S203: If they are the same, then replace the corresponding field name in the determination template with the literal constant corresponding to the field name of the data to be determined;

[0075] S204: If all field names in the determination template are replaced, a condition tree is generated and the condition tree is converted into an expression that only includes literal constants;

[0076] S205: Obtain the evaluation result of the expression and use the evaluation result as the judgment result.

[0077] In the specific implementation process, steps S201 to S205 are implemented as follows:

[0078] First, identify the field names and literal constants included in the decision template. Then, determine if the field names of the data to be decided match those in the decision template. This can be done by parsing each specific data entry generated from the log file and matching the corresponding data's field names against those in the decision template. If they match, replace the corresponding field name in the decision template with the literal constant corresponding to the field name of the data to be decided. If all fields in the decision template are replaced, a condition tree can be generated, which can then be transformed into an expression containing only literal constants. This allows for runtime field replacement, enabling complex conditions with field names to be matched and replaced at the SQL syntax tree level. The newly generated condition tree can be converted into an expression containing only literal constants, avoiding the efficiency issues caused by directly querying the database and resolving the problem of inability to make decisions due to the absence of historical data in the current database.

[0079] Then, the expression evaluation module can be invoked to evaluate the expression, thereby obtaining the evaluation result, which is then used as the judgment result. In one exemplary embodiment, open-source expression evaluation tools such as Aviator and Druid can be used to evaluate expressions that only include literal constants, outputting True or False. True indicates that the current data to be judged meets the synchronization conditions set by the user; False indicates that the current data to be judged does not meet the synchronization conditions set by the user. It should be noted that the condition tree in this embodiment of the invention only includes literal constants, and has the characteristics of simple structure, clear predicates, and clear data. Moreover, it is easy to connect to open-source expression judgment tools, has fast parsing speed, and strong versatility.

[0080] In this embodiment of the invention, after replacing the corresponding field name in the determination template with the literal constant corresponding to the field name of the data to be determined in step S203, the method further includes:

[0081] If at least one field name in the determination template is not replaced, the determination result indicates that the data to be determined does not meet the synchronization condition.

[0082] In practice, if after all field names in the judgment template that are identical to those in the data to be judged have been replaced, at least one field name in the judgment template remains unreplaced, False can be output directly, indicating that the data to be judged does not meet the synchronization conditions. "At least one" can be one or more fields, depending on the synchronization conditions set by the user and the data to be judged; no specific restrictions are imposed here.

[0083] In embodiments of the present invention, such as Figure 4 As shown, step S202: Determining whether the field names of the data to be judged are the same as the field names included in the judgment template includes:

[0084] S301: The data to be determined is represented by an array including multiple structures, wherein each structure includes a corresponding field name and a literal constant;

[0085] S302: Traverse the array and determine whether the field names included in the array are the same as the field names included in the determination template.

[0086] In the specific implementation process, steps S301 to S302 are implemented as follows:

[0087] First, each piece of data to be judged generated from the log file can be parsed. The data to be judged can be represented by an array containing multiple structures. Each structure includes a field name and its corresponding literal constant. When running the data synchronization system, the array can be traversed to determine whether the field names included in the array are the same as the field names included in the judgment template. If they are the same, the corresponding field names in the judgment template can be replaced with the literal constants corresponding to the field names included in the array. The specific replacement process can be referred to the description in the relevant section above, and will not be elaborated here.

[0088] The following example uses a MySQL database as the source database and a PostgreSQL database as the target database to explain the specific implementation process of the data synchronization method provided in this embodiment of the invention.

[0089] First, you can create the test.table table in the demo library using the following statement:

[0090] CREATE TABLE `test_table`(

[0091] `id`int(11)NOT NULL,

[0092] `name`varchar(255)COLLATE utf8mb4_unicode_ci DEFAULT NULL,

[0093] `type`varchar(255)COLLATE utf8mb4_unicode_ci DEFAULT NULL,

[0094] `city`varchar(255)COLLATE utf8mb4_unicode_ci DEFAULT NULL,

[0095] PRIMARY KEY(`id`)

[0096] )ENGINE=InnoDB DEFAULT CHARSET=utf8mb4

[0097] COLLATE=utf8mb4_unicode_ci;

[0098] Then, during incremental synchronization, data filtering is performed. Data in the test.table table with id > 10 or whose name field is 'kitty' and city field starts with 'Bei' can be allowed to be synchronized, while other data is not synchronized.

[0099] Then, table data synchronization can be performed on the data synchronization system provided in this embodiment of the invention. By filling in the corresponding source database information and target server information, the synchronization mapping relationship can be automatically generated with one click.

[0100] Then, you can configure the synchronization conditions on the visual interface by entering the corresponding SQL syntax: `select * from demo.test_table where id>10 or name='kitty'and city like'Bei%'`

[0101] Then, after reading the synchronization conditions, the syntax tree is parsed to generate a result such as... Figure 5 The abstract syntax tree shown is used to extract the conditional part to generate a decision template:

[0102] [id>10or name='kitty'and city like'Bei%']

[0103] The judgment template can be as follows: Figure 6 As shown.

[0104] Taking the data to be judged: [id=12,name='tom',type='dummy',city='HK'] as an example, the transmission structure of this data within the data synchronization system is as follows:

[0105] [EventColumn[index=1,columnType=12,columnName=id,

[0106] columnValue=12, isNull=false, isKey=false, isUpdate=true]

[0107] EventColumn[index=2,columnType=-5,columnName=name,

[0108] columnValue=tom,isNull=false,isKey=false,isUpdate=true]

[0109] EventColumn[index=3,columnType=-5,columnName=type,

[0110] columnValue=dummy,isNull=false,isKey=false,isUpdate=true]

[0111] EventColumn[index=4,columnType=-5,columnName=city,

[0112] columnValue=HK,isNull=false,isKey=false,isUpdate=true]]

[0113] Then, the runtime field replacement step is performed, and all fields in the decision template of the data are successfully replaced, resulting in a condition tree that only includes literal constants. This condition tree can be as follows: Figure 7 As shown. It can also be converted into a constant expression:

[0114] [12>10or'tom'='kitty'and'HK'like'Bei%']

[0115] Then, the constant expression is evaluated:

[0116] eval(12>10or'tom'='kitty'and'HK'like'Bei%')=true, therefore it can be determined that the data meets the synchronization conditions and needs to be released.

[0117] Of course, in the specific implementation process, in addition to the aforementioned synchronization conditions and data to be judged, other synchronization conditions can be set according to the actual application needs, and other judgment data generated during the operation of the data synchronization system can be synchronized accordingly. No limitation is made here.

[0118] This invention provides a data synchronization method with a general conditional judgment framework, high execution efficiency, and loose coupling between modules. This method can be applied to incremental conditional synchronization between heterogeneous databases, enabling rapid conditional judgment in scenarios without direct database queries, achieving on-demand data synchronization. It also allows for offline data storage, categorizing and backing up database data to other media. Furthermore, it facilitates data analysis and mining; by conditionally filtering incrementally synchronized data, it extracts data with specific characteristics for integration into data analysis platforms or data mining systems. Accordingly, besides applying this data synchronization method to data synchronization systems, it can also be applied to big data analysis systems and disaster recovery backup systems, depending on specific application needs. The specific implementation process will not be elaborated further.

[0119] Based on the same inventive concept, such as Figure 8 As shown, this embodiment of the invention also provides a data synchronization system, including:

[0120] Source database 10, target database 20, and synchronization worker nodes 30 that interact with the source database 10 and the target database 20 respectively; wherein: the synchronization worker node 30 is configured as follows:

[0121] Read the user-preset synchronization conditions;

[0122] The synchronization conditions are parsed to generate an abstract syntax tree;

[0123] The condition part in the abstract syntax tree is extracted as a judgment template, and the data to be judged is judged to obtain the judgment result. The data to be judged is incremental data obtained by parsing the log file of the source database.

[0124] If the determination result indicates that the data to be determined meets the synchronization condition, then the data to be determined is synchronized to the target database.

[0125] In this embodiment of the invention, the synchronization working node 30 is configured as follows:

[0126] Identify the field names and literal constants included in the determination template;

[0127] Determine whether the field names of the data to be judged are the same as the field names included in the judgment template;

[0128] If they are the same, then the corresponding field name in the judgment template is replaced with the literal constant corresponding to the field name of the data to be judged;

[0129] If all field names in the judgment template are replaced, a condition tree is generated and the condition tree is converted into an expression that only includes literal constants;

[0130] Obtain the evaluation result of the expression, and use the evaluation result as the judgment result.

[0131] In this embodiment of the invention, after replacing the corresponding field name in the judgment template with the literal constant corresponding to the field name of the data to be judged, the synchronization worker node is further configured as follows:

[0132] If at least one field name in the determination template is not replaced, the determination result indicates that the data to be determined does not meet the synchronization condition.

[0133] In this embodiment of the invention, the synchronization working node is configured as follows:

[0134] The data to be judged is represented by an array comprising multiple structures, wherein each structure includes a corresponding field name and a literal constant;

[0135] Iterate through the array and determine whether the field names included in the array are the same as the field names included in the determination template.

[0136] In the specific implementation process, since the technical problem to be solved by the data synchronization system provided in the embodiments of the present invention is the same as that of the aforementioned data synchronization method, the specific implementation process of the data synchronization system can be referred to the description of the relevant parts mentioned above, and will not be repeated here.

[0137] Based on the same inventive concept, such as Figure 9 As shown, this embodiment of the invention also provides a data synchronization device, including:

[0138] The reading unit 100 is used to read the user-preset synchronization conditions;

[0139] The generation unit 200 is used to perform syntax parsing on the synchronization conditions and generate an abstract syntax tree;

[0140] The judgment unit 300 is used to extract the condition part in the abstract syntax tree as a judgment template, judge the data to be judged, and obtain the judgment result. The data to be judged is incremental data obtained by parsing the log file of the source database.

[0141] The synchronization unit 400 is used to synchronize the data to be judged to the target database if the judgment result indicates that the data to be judged meets the synchronization condition.

[0142] In this embodiment of the invention, the determination unit 300 is used for:

[0143] Identify the field names and literal constants included in the determination template;

[0144] Determine whether the field names of the data to be judged are the same as the field names included in the judgment template;

[0145] If they are the same, then the corresponding field name in the judgment template is replaced with the literal constant corresponding to the field name of the data to be judged;

[0146] If all field names in the judgment template are replaced, a condition tree is generated and the condition tree is converted into an expression that only includes literal constants;

[0147] Obtain the evaluation result of the expression, and use the evaluation result as the judgment result.

[0148] In this embodiment of the invention, after replacing the corresponding field name in the determination template with the literal constant corresponding to the field name of the data to be determined, the determination unit 300 is further configured to:

[0149] If at least one field name in the determination template is not replaced, the determination result indicates that the data to be determined does not meet the synchronization condition.

[0150] In this embodiment of the invention, the determination unit 300 is used for:

[0151] The data to be judged is represented by an array comprising multiple structures, wherein each structure includes a corresponding field name and a literal constant;

[0152] Iterate through the array and determine whether the field names included in the array are the same as the field names included in the determination template.

[0153] Based on the same inventive concept, embodiments of the present invention also provide a data synchronization device, the data synchronization device including a processor, the processor being used to execute a computer program stored in a memory to implement the steps of the data synchronization method as described in any of the above claims.

[0154] Based on the same inventive concept, embodiments of the present invention also provide a readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the data synchronization method as described in any of the above claims.

[0155] This invention provides a data synchronization method, system, apparatus, and readable storage medium. First, user-preset synchronization conditions are read. Then, the synchronization conditions are parsed to generate an abstract syntax tree (AST). Next, the conditional portion of the AST is extracted as a decision template to determine the data to be determined, obtaining a decision result. The data to be determined is incremental data obtained by parsing log files from the source database. This allows for the determination of whether the incremental data obtained by directly parsing log files needs synchronization based on the decision template extracted from the AST. The entire determination process does not require reverse database lookups or database engine calls. When the determination result indicates that the synchronization conditions are met, the incremental data to be determined is synchronized to the target database. This improves the efficiency of conditional synchronization of incremental data based on log parsing and avoids the problem of historical data no longer existing in the current database, making reverse lookups impossible.

[0156] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0158] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0159] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0160] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data synchronization method, characterized in that, include: Read the user-preset synchronization conditions; The synchronization conditions are parsed to generate an abstract syntax tree; The condition part in the abstract syntax tree is extracted as a judgment template, and the data to be judged is judged to obtain the judgment result. The data to be judged is incremental data obtained by parsing the log file of the source database. If the determination result indicates that the data to be determined meets the synchronization condition, then the data to be determined is synchronized to the target database. The step of extracting the conditional portion from the abstract syntax tree as a decision template, judging the data to be judged, and obtaining the judgment result includes: Identify the field names and literal constants included in the determination template; Determine whether the field names of the data to be judged are the same as the field names included in the judgment template; If they are the same, then the corresponding field name in the judgment template is replaced with the literal constant corresponding to the field name of the data to be judged; If all field names in the judgment template are replaced, a condition tree is generated and the condition tree is converted into an expression that only includes literal constants; Obtain the evaluation result of the expression, and use the evaluation result as the judgment result.

2. The method as described in claim 1, characterized in that, After replacing the corresponding field name in the determination template with the literal constant corresponding to the field name of the data to be determined, the method further includes: If at least one field name in the determination template is not replaced, the determination result indicates that the data to be determined does not meet the synchronization condition.

3. The method as described in claim 1, characterized in that, The step of determining whether the field names of the data to be determined are the same as the field names included in the determination template includes: The data to be judged is represented by an array comprising multiple structures, wherein each structure includes a corresponding field name and a literal constant; Iterate through the array and determine whether the field names included in the array are the same as the field names included in the determination template.

4. A data synchronization system, characterized in that, include: A source database, a target database, and synchronization worker nodes that interact with the source database and the target database respectively; wherein: the synchronization worker nodes are configured as follows: Read the user-preset synchronization conditions; The synchronization conditions are parsed to generate an abstract syntax tree; The condition part in the abstract syntax tree is extracted as a judgment template, and the data to be judged is judged to obtain the judgment result. The data to be judged is incremental data obtained by parsing the log file of the source database. If the determination result indicates that the data to be determined meets the synchronization condition, then the data to be determined is synchronized to the target database. The step of extracting the conditional portion from the abstract syntax tree as a decision template, judging the data to be judged, and obtaining the judgment result includes: Identify the field names and literal constants included in the determination template; Determine whether the field names of the data to be judged are the same as the field names included in the judgment template; If they are the same, then the corresponding field name in the judgment template is replaced with the literal constant corresponding to the field name of the data to be judged; If all field names in the judgment template are replaced, a condition tree is generated and the condition tree is converted into an expression that only includes literal constants; Obtain the evaluation result of the expression, and use the evaluation result as the judgment result.

5. A data synchronization device, characterized in that, include: The reading unit is used to read the user-preset synchronization conditions; A generation unit is used to perform syntax parsing on the synchronization conditions and generate an abstract syntax tree; The judgment unit is used to extract the condition part in the abstract syntax tree as a judgment template, judge the data to be judged, and obtain the judgment result. The data to be judged is incremental data obtained by parsing the log file of the source database. A synchronization unit is used to synchronize the data to be judged to the target database if the judgment result indicates that the data to be judged meets the synchronization condition. The determination unit is used for: Identify the field names and literal constants included in the determination template; Determine whether the field names of the data to be judged are the same as the field names included in the judgment template; If they are the same, then the corresponding field name in the judgment template is replaced with the literal constant corresponding to the field name of the data to be judged; If all field names in the judgment template are replaced, a condition tree is generated and the condition tree is converted into an expression that only includes literal constants; Obtain the evaluation result of the expression, and use the evaluation result as the judgment result.

6. The apparatus as claimed in claim 5, characterized in that, The determination unit is used for: The data to be judged is represented by an array comprising multiple structures, wherein each structure includes a corresponding field name and a literal constant; Iterate through the array and determine whether the field names included in the array are the same as the field names included in the determination template.

7. A data synchronization device, characterized in that, The data synchronization device includes a processor, which executes a computer program stored in a memory to implement the steps of the data synchronization method as described in any one of claims 1-3.

8. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the data synchronization method as described in any one of claims 1-3.

Images