A log analysis method, a data synchronization system, an electronic device, and a storage medium
By retrieving and verifying metadata information from a relational database and using reverse instructions to handle metadata inconsistencies, the problem of historical log parsing failures was solved, and the stability and consistency of data migration and synchronization were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ALIBABA CLOUD COMPUTING CO LTD
- Filing Date
- 2022-11-10
- Publication Date
- 2026-07-14
Smart Images
Figure CN115730008B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data synchronization, specifically to a log parsing method, a data synchronization system, an electronic device, and a storage medium. Background Technology
[0002] In some scenarios where relational databases perform real-time data migration and synchronization from source to target databases, it is necessary to parse historical logs. However, inconsistencies often arise between the historical metadata and the current metadata actually obtained by the parsing module, leading to the failure of historical log parsing. No effective solution has yet been proposed for this technical problem. Summary of the Invention
[0003] This application provides a log parsing method, a data synchronization system, an electronic device, and a storage medium to at least solve the technical problem in the prior art where the historical metadata and the current metadata actually obtained by the parsing module are inconsistent, leading to the failure of historical log parsing.
[0004] According to one aspect of this application, a log parsing method is provided, comprising: obtaining first metadata information at a first time from a source database, and verifying the first metadata information with second metadata information at a second time stored locally, wherein the second metadata information is information after the first metadata information is executed with a specified instruction; if the verification fails, parsing the incremental log from the first time to the second time to obtain the specified instruction; obtaining the reverse instruction corresponding to the specified instruction to obtain the corresponding target instruction; obtaining third metadata information based on the second metadata information and the target instruction, and replacing the second metadata information with the third metadata information to pass a second verification; and parsing the log at the first time based on the third metadata information.
[0005] Optionally, obtaining the reverse instruction corresponding to the specified instruction and obtaining the corresponding target instruction includes: determining whether the specified instruction is a reversible instruction, wherein the reversible instruction is a second table structure obtained after executing the first instruction on the first table structure, and the instruction of the first table structure can be uniquely calculated through the second table structure and the reverse instruction of the first instruction; and obtaining the reverse instruction corresponding to the specified instruction based on the determination result to obtain the corresponding target instruction.
[0006] Optionally, based on the judgment result, obtaining the reverse instruction corresponding to the specified instruction and obtaining the corresponding target instruction includes: when the specified instruction is the reversible instruction, setting the reverse instruction of the specified instruction as the target instruction; when the specified instruction is the non-reversible instruction, obtaining the metadata information of the specified event in the incremental log from the first time to the second time; and determining the target instruction by combining the reverse instruction corresponding to the specified instruction with the metadata information of the specified event at the corresponding time.
[0007] Optionally, the specified event includes table mapping events.
[0008] Optionally, obtaining the third metadata information based on the second metadata information and the target instruction includes: arranging the target instructions in reverse chronological order to obtain a target instruction set; setting the table structure in the second metadata information as the initial table structure; executing the following iterative steps until the number of columns in the target table structure and the table structure in the first metadata information are consistent: setting the initial table structure to execute the target instruction in the target instruction set to obtain the target table structure, wherein the target instruction is the first instruction in the target instruction set; and using the target table structure and the next unused instruction of the target instruction as input for the next iterative step.
[0009] Optionally, verifying the first metadata information with the second metadata information stored locally at the second time includes: obtaining the number of columns in the first table of the first metadata information and the number of columns in the second table of the second metadata information; and verifying the number of columns in the first table with the number of columns in the second table.
[0010] Optionally, an error is issued when the specified instruction cannot generate the corresponding reverse instruction.
[0011] Optionally, in the event of a verification failure, the method further includes: setting the operation of obtaining metadata information from the source database to enter a waiting state, and determining the duration of the waiting state; and issuing an error when the duration exceeds a preset threshold.
[0012] Optionally, after parsing the log of the first time period using the first metadata information, the method further includes: synchronizing the first metadata information to the target database based on the parsing result.
[0013] According to one aspect of this application, a data synchronization system is also provided, comprising: a pull module for parsing logs of a source database using the method described above; an intermediate data storage module for storing data obtained by the pull module; and a write module for processing the data from the intermediate data storage module and writing it to a target database.
[0014] According to another aspect of this application, an electronic device is also provided, including a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the above-described method steps.
[0015] According to another aspect of this application, a readable storage medium is also provided, on which computer instructions are stored, wherein the computer instructions, when executed by a processor, implement the above-described method steps.
[0016] In this embodiment, first metadata information for a first time period is obtained from the source database, and the first metadata information is verified against second metadata information for a second time period stored locally. The second metadata information is the information after the first metadata information is executed with a specified instruction. If the verification fails, the incremental log from the first time period to the second time period is parsed to obtain the specified instruction. The reverse instruction corresponding to the specified instruction is obtained to obtain the corresponding target instruction. Third metadata information is obtained based on the second metadata information and the target instruction, and the third metadata information is used to replace the second metadata information to pass the secondary verification. The log for the first time period is parsed based on the third metadata information. That is, if the verification between historical metadata information (corresponding to the first metadata information mentioned above) and current metadata information (corresponding to the second metadata information mentioned above) fails, the data corresponding to the historical metadata (third metadata information) is calculated by using the current metadata and the reverse instruction of the corresponding instruction. The number of columns in the third metadata information is at least the same as the number of columns in the first metadata information mentioned above. This ensures that the verification passes and the historical log content is further parsed. This solves the technical problem in related technologies where the historical metadata and the current metadata actually obtained by the parsing module are inconsistent, which leads to the failure of historical log parsing. This achieves the technical effect of improving the stability of the log parsing module. Attached Figure Description
[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0018] Figure 1 This is a schematic flowchart of a DTS data synchronization method according to an embodiment of this application;
[0019] Figure 2 This is a flowchart of a log parsing method according to an embodiment of this application;
[0020] Figure 3 This is a flowchart of another log parsing method according to an embodiment of this application;
[0021] Figure 4 This is a schematic diagram of a log parsing system structure according to an embodiment of this application. Detailed Implementation
[0022] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0024] According to an embodiment of this application, an embodiment of a log parsing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0025] The data synchronization process of Data Transmission Service (DTS) is as follows: Figure 1As shown, this includes a source database (e.g., a relational database), a pull module, an intermediate data storage module, a write module, and a target database (e.g., a relational database). The pull module includes binary log (binlog) / redo log (redo) retrieval, parsing, metadata management, serialization, and data delivery. The intermediate data storage module includes start, insert, commit, and Data Definition Language (DDL) functions. The write module includes pulling intermediate data, data filtering, mapping, transaction concatenation, conflict matrix generation, DDL parsing, and Structured Query Language (SQL) assembly.The above-mentioned pulling module is responsible for pulling the synchronization logs of the source database, and then parsing them. During the synchronization process, it is necessary to maintain the metadata information of the source database table structure. When the pulling module is started for the first time, the table structure is obtained from the source database by executing SQL commands, and then serialized to local storage. When encountering DDL, it is necessary to parse information such as the database name, table name, column name, and column type to update the local metadata information. The metadata snapshot is stored locally according to a certain strategy to enable the state of the table to be restored to any specified time point. Then, the pulling module delivers Data Manipulation Language (DML) / DDL / Data Control Language (DCL) data to the intermediate data storage module. However, in some scenarios of real-time migration and synchronization of relational databases, problems of missing metadata often occur when parsing historical transaction logs. The source database log parsing module starts at the current time Tnow. When attempting to pull and parse incremental logs starting from time T0, problems of metadata mismatch with the Write-Ahead Log (WAL) may occur, mainly including the following two situations: Situation 1: T0 is the starting point, and Tnow is the time when the source database metadata is actually obtained after the log parsing module is started. For general scenarios, T0 < Tnow, and the difference between them is a relatively small value, usually at the second level, which depends on the speed of external control scheduling (which can be understood as a scheduler selecting a machine and starting the DTS process on that machine) and the initialization time of the log parsing module. In some extreme cases, between T0 and Tnow, the table structure of the source database may have changed. At this time, the log parsing module may fail to parse or have data errors due to the inconsistency between historical metadata and current metadata. Situation 2: Some users' requirements are to expect to parse incremental logs starting from a specified historical point T1. Tnow is the time when the source database metadata is actually obtained after the log parsing module is started. T1 < Tnow, and the difference may be a relatively large value, usually at the minute or even hour level. The probability of the table structure of the source database changing is higher than that described in Situation 1, and similar problems to those in Situation 1 may also be encountered.
[0026] Based on the above content, the embodiments of this application propose the following Figure 2 log parsing method, including:
[0027] S202: Obtain first-time metadata information from the source database and verify it against second-time metadata information stored locally. The second-time metadata information is the information obtained after executing a specified instruction on the first-time metadata information. It should be noted that the first time is shorter than the second time. During the period between the first and second times, the first-time metadata information may have executed one or more specified instructions, thus potentially transforming into second-time metadata information. The first and second-time metadata information includes, but is not limited to, table names, column names, column types, and encodings. For example, if the first-time metadata information includes table structure A, and by the second time, table structure A may have added a column, becoming table B, or removed a column, becoming table C, then verifying table structure A against table structure B or table structure C will result in a verification failure. Furthermore, it should be noted that the specified instructions include, but are not limited to: Data Manipulation Language (DML) instructions, Database Schema Definition Language (DDL) instructions, and Data Control Language (DCL) instructions.
[0028] S204, In the event of a verification failure, the incremental log from the first time point to the second time point is parsed to obtain the specified instruction. It should be noted that the specified instruction includes, but is not limited to: instructions to add columns, instructions to remove columns, instructions to change table names, and instructions to change column names. For example, from the first time point to the second time point, table structure A executed an instruction to add columns, becoming table structure B; therefore, the specified instruction from the first time point to the second time point is the instruction to add columns. As another example, from the first time point to the second time point, table structure A underwent instruction 1 to add columns, becoming table structure B, and then underwent instruction 2 to add columns, resulting in table structure C; therefore, the specified instructions from the first time point to the second time point are instruction 1 to add columns and instruction 2 to add columns.
[0029] S206, Obtain the reverse instruction corresponding to the specified instruction to get the corresponding target instruction;
[0030] Solving the aforementioned reverse instruction requires introducing two concepts: reversible instruction and reverse instruction. Taking a relational database as an example, it has a table structure A, which is transformed into table structure B after an instruction. Obviously, given A and the instruction, B can always be calculated. If B and the instruction are known, whether table structure A can be uniquely calculated requires case-by-case discussion. An instruction that can uniquely calculate table structure A from table structure B and the instruction is called a reversible instruction. If the instruction is reversible, it can be expressed by the formula: A + instruction = B is equivalent to B - instruction = A. For a reversible instruction, there is a corresponding reverse instruction, which is the -instruction in the above formula. This -instruction is called the reverse instruction, i.e., B + reverse instruction = A.
[0031] S208: Obtain third metadata information based on the second metadata information and the target instruction, and replace the second metadata information with the third metadata information to pass the secondary verification. It should be noted that the number of table columns in the third metadata information and the first metadata information can be consistent. Of course, for more accurate verification results, verification can also be performed by combining the third metadata information with other information from the first metadata information. In this step, since the second metadata information is replaced with the third metadata information, the number of table columns in the third metadata information and the first metadata information are consistent, thus passing the verification.
[0032] Furthermore, since historical metadata information is required to parse historical logs, this application embodiment also proposes S210, which parses the log of the first time according to the third metadata information. The parsed content includes, but is not limited to, DML (insert, update, delete), DDL, DCL, etc. in the log.
[0033] Through the above steps S202-S210, first metadata information for a first time period is obtained from the source database, and this first metadata information is verified against second metadata information for a second time period stored locally. The second metadata information is the information obtained after executing a specified instruction from the first metadata information. If the verification fails, the incremental log from the first time period to the second time period is parsed to obtain the specified instruction. The reverse instruction corresponding to the specified instruction is obtained to obtain the corresponding target instruction. Third metadata information is obtained based on the second metadata information and the target instruction, and this third metadata information is used to replace the second metadata information to pass the secondary verification. The log for the first time period is parsed based on the third metadata information. That is, if the verification between historical metadata information (corresponding to the first metadata information mentioned above) and current metadata information (corresponding to the second metadata information mentioned above) fails, the data corresponding to the historical metadata (third metadata information) is calculated by using the current metadata and the reverse instruction of the corresponding instruction. The number of columns in the third metadata information is at least the same as the number of columns in the first metadata information mentioned above. This ensures that the verification passes and the historical log content is further parsed. This solves the technical problem in related technologies where the historical metadata and the current metadata actually obtained by the parsing module are inconsistent, which leads to the failure of historical log parsing. This achieves the technical effect of improving the stability of the log parsing module.
[0034] In an optional implementation, obtaining the reverse instruction corresponding to the specified instruction and thus obtaining the corresponding target instruction includes: determining whether the specified instruction is a reversible instruction, wherein the reversible instruction is an instruction that, after executing the first instruction on the first table structure, a second table structure is obtained, and the instruction of the first table structure can be uniquely calculated through the second table structure and the reverse instruction of the first instruction; and, based on the determination result, obtaining the reverse instruction corresponding to the specified instruction and thus obtaining the corresponding target instruction. Specifically, obtaining the reverse instruction corresponding to the specified instruction and thus obtaining the corresponding target instruction based on the determination result may include: when the specified instruction is a reversible instruction, setting the reverse instruction of the specified instruction as the target instruction; when the specified instruction is an irreversible instruction, obtaining the metadata information of the specified event in the incremental log from the first time to the second time; and determining the target instruction by combining the reverse instruction corresponding to the specified instruction with the metadata information of the specified event at the corresponding time. It should be noted that the specified event may include a table mapping event. For example, table structure A has two columns, F1 and F2. If the specified instruction is to change table A by adding column F3, then table structure B will have three columns: F1, F2, and F3. If B and the specified instruction are known, table structure A can be uniquely calculated. Therefore, the specified instruction to change table A by adding column F3 is a reversible instruction. Its reverse instruction is to change the table by removing column F3. However, in some cases, knowing B and the instruction is not enough to uniquely calculate A. In this case, the instruction is called a non-reversible instruction. For example, if table structure A has two columns, F1 and F2, and the instruction is to change table A by removing column F2, table structure B will have one column F1. However, the specific type of F2 or its position (before or after F1) cannot be determined from the instruction. Non-reversible instructions are not entirely useless. In this embodiment, although it is impossible to infer the specific type of column F2 before removal from the instruction to change table A by removing column F2, some auxiliary information in the log can help with the analysis. For example, metadata information in the `table_map_event` event of the MySQL binary log (binlog) can be used to infer the fuzzy type and location of F2. Since F2 will eventually be removed, from the perspective of eventual consistency, parsing F2 based on the fuzzy type is acceptable. Therefore, in this embodiment, irreversible instructions may also infer key information of reverse instructions under certain conditions, which can be used to restore the incremental parsing system. Thus, through the above two methods, one or more reverse instructions can be effectively obtained, providing a foundation for restoring historical metadata.
[0035] In this embodiment, obtaining the third metadata information based on the second metadata information and the target instruction may include: arranging the target instructions in reverse chronological order to obtain a target instruction set; setting the table structure in the second metadata information as the initial table structure; executing the following iterative steps until the number of columns in the target table structure and the table structure in the first metadata information are consistent; setting the initial table structure to execute the target instruction in the target instruction set to obtain the target table structure, wherein the target instruction is the first instruction in the target instruction set; using the target table structure and the next unused instruction of the target instruction as input for the next iterative step. That is, one or more reverse instructions are traversed from back to front, combined with the aforementioned second metadata information, to calculate the metadata information corresponding to the first time (i.e., the aforementioned third metadata information), so as to update the third metadata information in local storage.
[0036] Optionally, in this embodiment, the verification mainly focuses on metadata column verification. Specifically, it obtains the number of columns in the first table of the first metadata information and the number of columns in the second table of the second metadata information; then it verifies the number of columns in the first table and the number of columns in the second table. For example, if the table structure in the first metadata information includes one column and the table structure in the second metadata information includes one column, then the number of columns in the table structures of the first and second metadata information is the same, and the verification passes. As another example, if the table structure in the first metadata information includes one column and the table structure in the second metadata information includes two columns, then the number of columns in the table structures of the first and second metadata information is different, and the verification fails.
[0037] Of course, if the specified instruction cannot generate the corresponding reverse instruction, this embodiment of the application will also issue an error. In the case of verification failure, the operation of obtaining metadata information from the source database is set to enter a waiting state, and the duration of the waiting state is determined; if the duration exceeds a preset threshold, an error will also be issued.
[0038] After obtaining the initial log parsing results using the methods described above, the initial metadata information can be synchronized to the target database based on the parsing results, achieving data synchronization between the source and target databases. Then, the log parsing methods described above can be used to sequentially obtain logs from other points in time in the source database.
[0039] The embodiments of this application will be illustrated below with specific examples.
[0040] like Figure 3As shown, taking the relational database MySQL as an example, the specified commands are explained using DDL commands. Specifically, the source relational database has a table t1 with the table structure c1 int. At time ts1, table t1 has only one column, c1int. At time ts2, a DDL command to add a column occurs. At time ts3, table t1 has two columns, c2int, which is added to table t1. The incremental fetch module starts at time ts3, aiming to fetch the binary log (binlog) from time ts1. At time ts3, the incremental fetch module retrieves the structure of table t1 from the source database and caches it in memory. Therefore, the snapshot time of table t1 is ts3. At time ts3, table t1 has two columns, c1 and c2. However, the binlog to be fetched is from time ts1, and the binlog contains data from only one column of table t1. Therefore, according to the original logic, the incremental fetch module will fail to parse due to column validation failure. After introducing the reverse instructions corresponding to DDL instructions, a DDL instruction parsing thread will be started simultaneously when the incremental log fetching thread is started. The DDL instruction parsing thread reads the binlog from the starting point, mainly to obtain the DDL instructions in the binlog. Its parsing point is continuously updated as it fetches binlog. If the source database is MySQL, it will also collect the table mapping event table_map_event content of each table to assist in reasoning about the type and position of columns. When the fetch thread encounters a mismatch between the binlog and the table structure, it enters a loop waiting phase. During this phase, it continuously acquires the position of the DDL instruction parsing thread until the position of the DDL instruction parsing thread exceeds the snapshot time of the table structure that caused the error. Then, the fetch thread iterates through the DDL instructions parsed by the DDL instruction thread from back to front. If these DDL instructions can generate their corresponding reverse instructions, the table structure in memory will be updated sequentially to obtain the table structure at time ts1. The fetch thread can obtain a table structure that matches the binlog and enters the normal synchronization phase. If the DDL instructions cannot generate their corresponding reverse instructions, it will exit with an error. If the fetch thread exceeds the maximum waiting time, it will also exit with an error.
[0041] In summary, this embodiment proposes the concept of reverse instructions, providing reverse mapping capabilities for instructions. It can, based on the characteristics of different relational databases, such as MySQL's `table_map_event`, resolve historical log parsing issues with a high probability. While performing recovery tasks, it can achieve eventual consistency in data migration and synchronization. This embodiment is applicable to scenarios where historical data needs to be pulled back during log parsing in relational databases, and is a general solution. In DTS customer support tickets, there have indeed been several scenarios requiring the pulling back of historical data. The instructions to pull back to the current time point mainly involve changing tables and adding columns. The solution of constructing historical metadata information based on the reverse instructions proposed in this embodiment has successfully helped customers parse logs.
[0042] This invention also provides a data synchronization system, such as... Figure 4 As shown, it includes: a pull module 42, used to parse the logs of the source database using the above-mentioned log parsing method; an intermediate data storage module 44, used to store the data obtained by the pull module; and a write module 46, used to process the data from the intermediate data storage module and write it to the target database.
[0043] The system retrieves first-time metadata information from the source database and verifies it against second-time metadata information stored locally. The second-time metadata information is the result of executing a specified instruction on the first-time metadata information. If verification fails, the incremental log from the first time to the second time is parsed to obtain the specified instruction. The inverse instruction corresponding to the specified instruction is obtained to arrive at the corresponding target instruction. Third-time metadata information is obtained based on the second-time metadata information and the target instruction, and this third-time metadata information replaces the first-time metadata information to pass a second verification. Finally, the log from the first time is parsed based on the third-time metadata information. That is, if the verification between historical metadata information (corresponding to the first metadata information mentioned above) and current metadata information (corresponding to the second metadata information mentioned above) fails, the data corresponding to the historical metadata (third metadata information) is calculated by using the current metadata and the reverse instruction of the corresponding instruction. The number of columns in the third metadata information is at least the same as the number of columns in the first metadata information mentioned above. This ensures that the verification passes and the historical log content is further parsed. This solves the technical problem in related technologies where the historical metadata and the current metadata actually obtained by the parsing module are inconsistent, which leads to the failure of historical log parsing. This achieves the technical effect of improving the stability of the log parsing module.
[0044] In an optional implementation, the above-mentioned pull module 42 is further configured to obtain the reverse instruction corresponding to the specified instruction and obtain the corresponding target instruction, including: determining whether the specified instruction is a reversible instruction, wherein the reversible instruction is that the first table structure is executed after the first instruction is executed to obtain the second table structure, and the instruction of the first table structure can be uniquely calculated through the second table structure and the reverse instruction of the first instruction; and obtaining the reverse instruction corresponding to the specified instruction based on the determination result to obtain the corresponding target instruction. Obtaining the reverse instruction corresponding to the specified instruction and obtaining the corresponding target instruction based on the determination result may include: when the specified instruction is the reversible instruction, setting the reverse instruction of the specified instruction as the target instruction; when the specified instruction is the non-reversible instruction, obtaining the metadata information of the specified event in the incremental log from the first time to the second time; and determining the target instruction by combining the reverse instruction corresponding to the specified instruction with the metadata information of the specified event at the corresponding time. It should be noted that the specified event may include a table mapping event. For example, table structure A has two columns, F1 and F2. If the specified instruction is to change table A by adding column F3, then table structure B will have three columns: F1, F2, and F3. If B and the specified instruction are known, table structure A can be uniquely calculated. Therefore, the specified instruction to change table A by adding column F3 is a reversible instruction. Its reverse instruction is to change the table by removing column F3. However, in some cases, knowing B and the instruction is not enough to uniquely calculate A. In this case, the instruction is called a non-reversible instruction. For example, if table structure A has two columns, F1 and F2, and the instruction is to change table A by removing column F2, table structure B will have one column F1. However, the specific type of F2 or its position (before or after F1) cannot be determined from the instruction. Non-reversible instructions are not entirely useless. In this embodiment, although it is impossible to infer the specific type of column F2 before removal from the instruction to change table A by removing column F2, some auxiliary information in the log can help with the analysis. For example, metadata information in the `table_map_event` event of the MySQL binary log (binlog) can be used to infer the fuzzy type and location of F2. Since F2 will eventually be removed, from the perspective of eventual consistency, parsing F2 based on the fuzzy type is acceptable. Therefore, in this embodiment, irreversible instructions may also infer key information of reverse instructions under certain conditions, which can be used to restore the incremental parsing system. Thus, through the above two methods, one or more reverse instructions can be effectively obtained, providing a foundation for restoring historical metadata.
[0045] In this embodiment, the fetch module 42 is further configured to: arrange the target instructions in reverse chronological order to obtain a target instruction set; set the table structure in the second metadata information as the initial table structure; execute the following loop steps until the number of columns in the target table structure and the table structure in the first metadata information are consistent: set the initial table structure to execute the target instruction in the target instruction set to obtain the target table structure, wherein the target instruction is the first instruction in the target instruction set; and use the target table structure and the next unused instruction of the target instruction as input for the next loop step. Obtaining third metadata information based on the second metadata information and the target instruction may include: arranging the target instructions in reverse chronological order to obtain a target instruction set; setting the table structure in the second metadata information as the initial table structure; executing the following loop steps until the number of columns in the target table structure and the table structure in the first metadata information are consistent: set the initial table structure to execute the target instruction in the target instruction set to obtain the target table structure, wherein the target instruction is the first instruction in the target instruction set; and use the target table structure and the next unused instruction of the target instruction as input for the next loop step. That is, one or more reverse instructions are traversed from back to front, and combined with the aforementioned second metadata information, the metadata information corresponding to the first time (i.e., the aforementioned third metadata information) is calculated, so as to update the third metadata information in the local storage.
[0046] Optionally, in this embodiment, the verification mainly focuses on metadata column verification. Specifically, the retrieval module 42 is also used to obtain the number of columns in the first table of the first metadata information and the number of columns in the second table of the second metadata information; and to verify the number of columns in the first table and the number of columns in the second table. For example, if the table structure in the first metadata information includes one column and the table structure in the second metadata information includes one column, then the number of columns in the table structures of the first and second metadata information is the same, and the verification passes. As another example, if the table structure in the first metadata information includes one column and the table structure in the second metadata information includes two columns, then the number of columns in the table structures of the first and second metadata information is different, and the verification fails.
[0047] Of course, if the specified instruction cannot generate the corresponding reverse instruction, this embodiment of the application will also issue an error. In the case of verification failure, the operation of obtaining metadata information from the source database is set to enter a waiting state, and the duration of the waiting state is determined; if the duration exceeds a preset threshold, an error will also be issued.
[0048] This invention also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the steps of any of the above methods.
[0049] This invention also provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of any of the above methods.
[0050] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0051] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0052] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0053] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0054] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0055] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A log parsing method, comprising: First metadata information is obtained from the source database at a first time, and the first metadata information is verified with second metadata information stored locally at a second time, wherein the second metadata information is the information after the first metadata information is executed with a specified instruction; In the event of a verification failure, the incremental log from the first time to the second time is parsed to obtain the specified instruction; Obtain the reverse instruction corresponding to the specified instruction to obtain the corresponding target instruction; The target instructions are arranged in reverse chronological order to obtain a set of target instructions, and the table structure in the second metadata information is set as the initial table structure. Execute the following iterative steps until the number of columns in the target table structure and the table structure in the first metadata information are consistent, and obtain the third metadata information: set the initial table structure to execute the first target instruction in the target instruction set to obtain the target table structure; use the target table structure and the next unused instruction of the first target instruction as the input for the next iterative step; Replace the second metadata information with the third metadata information to pass the secondary verification; The logs from the first time period are parsed based on the third metadata information.
2. The method according to claim 1, wherein, Obtaining the reverse instruction corresponding to the specified instruction, and thus obtaining the corresponding target instruction, includes: Determine whether the specified instruction is a reversible instruction, wherein the reversible instruction is the second table structure obtained after the first instruction is executed from the first table structure, and the instruction of the first table structure can be uniquely calculated through the second table structure and the reverse instruction of the first instruction; Based on the judgment result, the reverse instruction corresponding to the specified instruction is obtained, and the corresponding target instruction is obtained.
3. The method according to claim 2, wherein, Based on the judgment result, the reverse instruction corresponding to the specified instruction is obtained, and the corresponding target instruction is obtained, including: When the specified instruction is the reversible instruction, the inverse instruction of the specified instruction is set as the target instruction; When the specified instruction is an irreversible instruction, the metadata information of the specified event in the incremental log from the first time to the second time is obtained; the target instruction is determined by combining the reverse instruction corresponding to the specified instruction with the metadata information of the specified event at the corresponding time.
4. The method according to claim 3, wherein, The specified events include table mapping events.
5. The method according to claim 1, wherein, Verifying the first metadata information with the second metadata information stored locally at the second time includes: Obtain the number of columns in the first table containing the first metadata information and the number of columns in the second table containing the second metadata information; The number of columns in the first table is compared with the number of columns in the second table.
6. The method according to claim 1, wherein, An error is issued when the specified instruction cannot generate the corresponding reverse instruction.
7. The method according to claim 1, wherein, In the event of a verification failure, the method further includes: The operation of retrieving metadata information from the source database is set to enter a waiting state, and the duration of the waiting state is determined. An error is issued when the duration exceeds a preset threshold.
8. The method according to claim 1, wherein, After parsing the log from the first time period using the first metadata information, the method further includes: Based on the parsing results, the first metadata information is synchronized to the target database.
9. A data synchronization system, comprising: A pull module is used to parse logs from a source database using the method described in any one of claims 1 to 8; An intermediate data storage module is used to store the data obtained by the pull module; The writing module is used to process the data from the intermediate data storage module and write it to the target database.
10. An electronic device comprising a memory and a processor; wherein, The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the steps of the method according to any one of claims 1 to 8.
11. A readable storage medium having computer instructions stored thereon, wherein, When executed by a processor, the computer instructions implement the steps of the method described in any one of claims 1 to 8.