Log analysis method and device, electronic equipment, storage medium and program product
By generating a dynamic data dictionary to parse the write-ahead log file, the problem that static data dictionaries cannot adapt to dynamic table structure changes is solved, achieving high-precision restoration of SQL statements and improving operational efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CETC JINCANG (BEIJING) TECH CO LTD
- Filing Date
- 2026-03-02
- Publication Date
- 2026-06-05
AI Technical Summary
When parsing the Write-Ahead Log (WAL), existing technologies cannot adapt the static data dictionary to dynamic table structure changes, resulting in missing fields, type mismatches, and primary key constraint errors in SQL statements, which affects the accuracy of data recovery and operational efficiency.
By parsing the write-ahead log file to obtain the table structure update content, a dynamic data dictionary is generated. Based on the initial data dictionary and the table structure update content, the data operation statements of the transaction processing log are restored, avoiding the problem of mismatch between the static data dictionary and the dynamic table structure.
It significantly improves the automation and accuracy of write-ahead log file parsing, supports high-precision restoration of SQL statements before and after table structure changes, and solves the problem that the data dictionary cannot adapt to dynamic changes in table structure.
Smart Images
Figure CN122152602A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of database technology, and in particular to a log parsing method, apparatus, electronic device, storage medium, and program product. Background Technology
[0002] In core business scenarios such as database operation and maintenance, data recovery, and compliance auditing, the Write-Ahead Log (WAL) is a core mechanism of relational databases, recording detailed information about all transaction operations. For example, when database data becomes abnormal due to accidental operations, such as accidentally deleting table data or accidentally updating fields, operation and maintenance personnel need to parse the WAL log to reconstruct the specific SQL statements (data manipulation statements) for reverse execution or data rollback to restore the data. In data security auditing scenarios, it is necessary to trace the SQL corresponding to all data modification operations within a specific time period to meet compliance requirements. When troubleshooting master-slave replication anomalies, it is necessary to analyze the operation records in the WAL log to locate the root cause of replication interruption. In addition, WAL log parsing is also widely used in scenarios such as database migration, data consistency verification, and historical data version comparison.
[0003] However, existing technologies rely on static table structures or fixed data dictionaries when parsing WAL logs to restore SQL. When the database table structure changes dynamically, such as when table definitions are modified through DDL operations (Data Definition Language) like CREATE TABLE and ALTER TABLE, the static data dictionary cannot be updated in time. This results in parsed SQL statements containing issues such as missing fields, type mismatches, and incorrect primary key constraints, leading to poor adaptability and ultimately affecting the accuracy of data recovery, audit compliance, and operational efficiency. Summary of the Invention
[0004] This application provides a log parsing method, apparatus, electronic device, storage medium, and program product to solve the SQL restoration error problem caused by the mismatch between static data dictionary and dynamic table structure.
[0005] In a first aspect, embodiments of this application provide a log parsing method, including:
[0006] Load the initial data dictionary;
[0007] Parse the write-ahead log file to generate table structure update content;
[0008] Generate a dynamic data dictionary based on the table structure update content and the initial data dictionary;
[0009] Based on a dynamic data dictionary, the data manipulation statements of the transaction processing log in the write-ahead log file are restored.
[0010] In one embodiment, parsing the write-ahead log file to generate table structure update content includes:
[0011] Based on the transaction commit records in the write-ahead log file, obtain the table structure information; the table structure information includes data field names, field types, and constraint information.
[0012] Based on the table structure information, determine the content to be updated in the table structure.
[0013] In one embodiment, the content to be updated in the structure is determined based on the table structure information, including:
[0014] Based on the object identifier in the table structure information, determine whether table structure information exists in the initial data dictionary;
[0015] If the initial data dictionary table structure does not contain object identifiers, then the data field names, field types, and constraint information will be used as the table structure update content.
[0016] If object identifiers exist in the table structure of the initial data dictionary, then the change records in the table structure information will be used as the table structure update content.
[0017] In one embodiment, before parsing the write-ahead log file, the method further includes:
[0018] Create a hash table; the hash table is used to cache the mapping relationship between physical file names and object identifiers.
[0019] When parsing a write-ahead log file, a hash table is used to determine the target write-ahead log file.
[0020] In one embodiment, the target write-ahead log file is determined using a hash table, specifically including:
[0021] When the physical filename of the table changes, or the mapping between the physical filename of the table and the object identifier changes, the target write-ahead log file is determined and the mapping in the hash table is updated.
[0022] In one embodiment, a dynamic data dictionary is generated based on the table structure update content and the initial data dictionary, specifically including at least one of the following:
[0023] Only update the field definitions in the initial data dictionary that are related to the table structure update content;
[0024] Only write-ahead log entries related to table structure changes are parsed.
[0025] Secondly, embodiments of this application provide a log parsing apparatus, including:
[0026] The loading module is used to load the initial data dictionary;
[0027] The parsing module is used to parse the write-ahead log file and generate table structure update content;
[0028] The generation module is used to generate a dynamic data dictionary based on the table structure update content and the initial data dictionary;
[0029] The restore module is used to restore the data manipulation statements of the transaction log in the write-ahead log file based on the dynamic data dictionary.
[0030] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;
[0031] The memory stores instructions that the computer executes;
[0032] The processor executes computer execution instructions stored in memory, causing the processor to perform any of the methods described above.
[0033] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement any of the methods described above.
[0034] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements any of the methods described above.
[0035] This application provides a log parsing method, apparatus, electronic device, storage medium, and program product. The method includes: loading an initial data dictionary; parsing a write-ahead log file to generate table structure update content; generating a dynamic data dictionary based on the table structure update content and the initial data dictionary; and restoring the data operation statements of the transaction log in the write-ahead log file based on the dynamic data dictionary. This application obtains the table structure update content by parsing the structure change records in the write-ahead log file, and generates a dynamic data dictionary based on the initial data dictionary and the table structure update content. The dynamic data dictionary is used to parse the data operation statements in the transaction log, converting binary data into data operation statements (SQL) that conform to the current table structure. The dynamic data dictionary avoids problems such as missing fields, type mismatches, or incorrect primary key constraints caused by the mismatch between the static data dictionary and the dynamic table structure. This significantly improves the automation level and accuracy of write-ahead log file parsing, supporting high-precision restoration of SQL statements before and after table structure changes. Furthermore, the dynamic data dictionary generation mechanism solves the problem in the prior art where the data dictionary cannot adapt to dynamic changes in the table structure. Attached Figure Description
[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0037] Figure 1 A flowchart illustrating a log parsing method provided in an embodiment of this application;
[0038] Figure 2 A logical diagram illustrating the generation of a dynamic data dictionary provided in an embodiment of this application;
[0039] Figure 3 A flowchart illustrating a log parsing method provided in an embodiment of this application;
[0040] Figure 4 This is a schematic diagram of the structure of a log parsing device provided in an embodiment of this application;
[0041] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.
[0042] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0044] First, let me explain the terms used in this application:
[0045] WAL: Write-Ahead Log (WAL) is a transaction log mechanism widely used in relational databases (such as PostgreSQL, MySQL, etc.).
[0046] Data dictionary: A file that records table structure information, and also needs to record metadata about the data dictionary's generation: timeline, database, and corresponding WAL log sequence number. Walminer relies on the data dictionary to convert the binary data in the WAL records into the corresponding format when parsing WAL files.
[0047] Walminer: A WAL log parsing tool designed to extract all the useful information from WAL logs, thereby supporting data recovery.
[0048] Timeline: A logical time marker for a database instance, used to distinguish WAL files at different points in time.
[0049] WAL segment number: The segment number of the WAL file, used to locate the physical location of the log.
[0050] Existing technologies rely on static table structures or fixed data dictionaries when parsing WAL logs to restore SQL. For example, if a static data dictionary is not updated promptly after an ALTER TABLE statement adds a field, the DML (Data Manipulation Language) statements corresponding to the added field in the subsequent WAL log parsing may fail to restore correctly due to missing fields or syntax errors caused by type mismatches. Furthermore, manually configuring the data dictionary requires user intervention, increasing operational complexity. In scenarios with dynamic table structures, the parsing accuracy and adaptability of existing solutions decrease significantly, failing to meet the requirements for high-reliability data recovery and auditing.
[0051] To address the shortcomings of existing technologies, the inventors of this solution have creatively designed a new approach. This solution provides a log parsing method to solve the problem that static data dictionaries cannot adapt to dynamic table structure changes. The specific application scenario for this application is the operation and maintenance of relational databases, especially database environments involving frequent table structure changes (such as DDL operations like CREATETABLE and ALTER TABLE). For example, in scenarios such as database fault recovery, data auditing, and troubleshooting master-slave replication anomalies, maintenance personnel need to recover DML statements from WAL logs to locate problems or repair data. When the database table structure changes frequently due to business needs or system upgrades, the static data dictionary cannot be updated in a timely manner, resulting in SQL statements parsed from WAL logs containing problems such as missing fields and type mismatches.
[0052] This application provides a log parsing method that obtains table structure update content by parsing structure change records in a write-ahead log file. Based on an initial data dictionary and the updated table structure content, a dynamic data dictionary is generated. This dynamic data dictionary is then used to parse data operation statements in the transaction log, converting binary data into SQL statements that conform to the current table structure. The dynamic data dictionary avoids problems such as missing fields, type mismatches, or primary key constraint errors caused by mismatches between the static data dictionary and the dynamic table structure. This significantly improves the automation and accuracy of write-ahead log file parsing, supporting high-precision reconstruction of SQL statements before and after table structure changes. Furthermore, the dynamic data dictionary generation mechanism solves the problem in existing technologies where data dictionaries cannot adapt to dynamic table structure changes.
[0053] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.
[0054] like Figure 1 As shown, Figure 1 A flowchart illustrating a log parsing method according to an embodiment of this application. The log parsing method includes the following steps:
[0055] Step S101: Load the initial data dictionary.
[0056] Specifically, database structure information is loaded as an initial data dictionary to extract table structures, such as table names, field names, and types, from the system tables in the initial data dictionary. In one embodiment, if the user specifies a data dictionary, that dictionary is directly loaded as the initial data dictionary; if the user only specifies a directory, the most suitable dictionary is loaded as the initial data dictionary based on the timeline and segment number of the WAL file. For example, if the WAL file to be parsed is located at segment 45 of timeline 123, the system searches the directory for a dictionary file generated at timeline 123 with a corresponding WAL segment number of 45. If no match is found, the most recently generated dictionary in timeline 123 is selected as the initial data dictionary. This ensures that the initial data dictionary matches the WAL file to be parsed in time and logic, avoiding parsing failures due to inconsistencies in timelines or segment numbers. By using the timeline and segment number matching rules, the most suitable dictionary can be automatically selected as the parsing starting point, laying the foundation for subsequent dynamic updates.
[0057] Step S102: Parse the write-ahead log file and generate table structure update content.
[0058] In one embodiment, step S102 includes the following steps:
[0059] Based on the transaction commit records in the write-ahead log file, obtain the table structure information; the table structure information includes data field names, field types, and constraint information.
[0060] Based on the table structure information, determine the content to be updated in the table structure.
[0061] Specifically, when parsing the write-ahead log (WAL) file, only the WAL records from correctly committed transactions are parsed, such as those from the Commit operation. From these transactions, field names, field types, and constraint information related to table structure changes are extracted. Constraint information refers to the rules and restrictions imposed on data columns in the table structure to ensure data integrity, consistency, and validity. For example, the field definition corresponding to the ALTER TABLE ADD COLUMN operation. The extracted field information is then used to update the initial data dictionary, generating updated table structure content. For example, if the initial data dictionary does not contain the field `new_col`, but the transaction commit record contains its definition, then updated content including that field is generated. This example achieves accurate parsing of table structure change records through the identification of transaction commit records and the extraction of field information, ensuring that the updated content of the dynamic data dictionary strictly matches the actual table structure changes. By comparing the extracted field information with the definitions in the initial data dictionary, it ensures that the updated content of the dynamic data dictionary only includes the actually changed fields, avoiding redundant updates. For example, when the database executes the ALTER TABLE ADD COLUMN operation, the system only extracts the information of the newly added fields and generates the updated content, rather than scanning the entire table structure. This improves the accuracy of dynamic data dictionary updates and ensures that the restoration of subsequent data operation statements is always based on the latest table structure definition.
[0062] Step S103: Based on the table structure update content and the initial data dictionary, generate a dynamic data dictionary.
[0063] Specifically, the extracted table structure update content is used to update the initial data dictionary, for example, after adding a field or modifying the field type, to generate a dynamic data dictionary.
[0064] Step S104: Based on the dynamic data dictionary, restore the data operation statements of the transaction processing log in the write-ahead log file.
[0065] Specifically, based on the updated dynamic data dictionary, the DML operations (INSERT, UPDATE, or DELETE) in the transaction commit log are parsed to generate SQL statements that match the current table structure.
[0066] This application obtains table structure update content by parsing structure change records in write-ahead log files, and generates a dynamic data dictionary based on the initial data dictionary and the table structure update content. The dynamic data dictionary is then used to parse data operation statements in the transaction log, converting binary data into data operation statements (SQL) conforming to the current table structure. The dynamic data dictionary avoids problems such as missing fields, type mismatches, or primary key constraint errors caused by mismatches between static data dictionaries and dynamic table structures. This significantly improves the automation and accuracy of write-ahead log file parsing, supporting high-precision reconstruction of SQL statements before and after table structure changes. Furthermore, the dynamic data dictionary generation mechanism solves the problem in existing technologies where data dictionaries cannot adapt to dynamic table structure changes.
[0067] like Figure 2 As shown, Figure 2 This is a logical diagram illustrating the generation of a dynamic data dictionary according to an embodiment of this application. The WAL file is parsed to obtain table structure information, and the table structure of the initial data dictionary is obtained, i.e. Figure 2 The old data dictionary in the WAL file, for example, if the first record is parsed as tuple(1,1), test1, insert, then based on the current data dictionary, the constructed SQL statement would be insert into test1(a,b) values(1,1). If the third record is parsed as tuple(b), insert, at, then this belongs to table structure information, and the dynamic data dictionary, i.e., the new data dictionary, is updated, as shown in the table below:
[0068]
[0069] The fourth record in the WAL file, tuple(1,3), test2, insert, will change (a,b) in test2 in the dynamic data dictionary to (1,3).
[0070] In one embodiment, determining the table structure update content based on table structure information includes the following steps:
[0071] Based on the object identifier in the table structure information, determine whether table structure information exists in the table structure of the initial data dictionary.
[0072] Specifically, the initial data dictionary is traversed and compared with the table structure information. Please refer to [link / reference]. Figure 2 , Figure 2 The first record, wal record1, and the second record, wal record2, contain table structure information parsed from the wal file, which is present in the initial data dictionary; the third record, wal record3, does not exist.
[0073] If the initial data dictionary table structure does not contain object identifiers, then the data field names, field types, and constraint information will be added to the table structure.
[0074] Specifically, the field type refers to the storage format of the database field; the constraint information refers to the constraint rules of the field, such as primary key constraints, uniqueness constraints, or NOT NULL constraints. For example, if a new table or field is added in the table structure information, the added content will be directly used as the table structure update content.
[0075] If object identifiers exist in the table structure of the initial data dictionary, then the change records in the table structure information will be used as the table structure update content.
[0076] Specifically, based on the extracted table structure information and the content parsed from the WAL records, the content is directly added to the data dictionary or the existing content in the data dictionary is updated. For example, adding fields and constraints is directly written to the data dictionary; if it is an update, such as changing VARCHAR(100) to VARCHAR(255) or modifying table field names, the data dictionary is traversed to find the corresponding field record in the table and the record is updated directly. For example, when the database modifies a field type, the system writes the updated type to the dynamic data dictionary through this step, ensuring that subsequent parsed data operation statements are based on the latest field definition. This example achieves accurate updates of field definitions in the dynamic data dictionary through traversal logic; secondly, it determines whether a new field exists by checking the field name and updates the field definition by comparing the type and constraints, avoiding parsing errors caused by duplicate field definitions or type mismatches. For example, when the database modifies a field type, the system only updates the type definition of that field, rather than replacing the entire table structure, thereby improving the efficiency and accuracy of dynamic data dictionary maintenance.
[0077] In one embodiment, the following steps are included before parsing the write-ahead log file:
[0078] Create a hash table; the hash table is used to cache the mapping between physical file names and object identifiers.
[0079] Specifically, during the parsing process, the system creates a hash table to cache the mapping between the physical filenames of the tables and their object identifiers (OIDs). For example, if the physical filename of the table 'users' is 12346 and its object identifier (OID) is 12345, the system stores the mapping between 12346 and 12345 in the hash table.
[0080] When parsing a write-ahead log file, a hash table is used to determine the target write-ahead log file.
[0081] Specifically, by parsing the WAL records of system tables in real time, metadata related to table structure changes is extracted, and hash tables are used to cache mapping relationships to filter records from non-target tables. This process ensures that the data dictionary always contains the latest table structure information when parsing the WAL logs, thereby avoiding parsing errors caused by table structure changes.
[0082] In one embodiment, determining the target write-ahead log file using a hash table specifically includes the following steps:
[0083] When the physical filename of the table changes, or the mapping between the physical filename of the table and the object identifier changes, the target write-ahead log file is determined and the mapping in the hash table is updated.
[0084] Specifically, when a change in the physical filename or OID of a table is detected (such as a vacuum full operation), the hash table dynamically updates the mapping relationship to ensure that the filtering rules are consistent with the current table structure. This not only reduces unnecessary parsing overhead but also avoids filtering errors caused by old mappings, thereby improving overall parsing efficiency and accuracy.
[0085] During the parsing process, there is only one data dictionary, namely the dynamic data dictionary. Each update is performed on this data dictionary, and the latest data dictionary is always used for parsing when restoring the SQL.
[0086] In one embodiment, step S103 specifically includes at least one of the following:
[0087] Only update the field definitions in the initial data dictionary that are related to the table structure update.
[0088] Only write-ahead log entries related to table structure changes are parsed.
[0089] Specifically, when generating a dynamic data dictionary, the system only updates the field definitions related to table structure changes in the initial data dictionary, such as adding fields or modifying field types, rather than performing a full scan of the entire data dictionary. For example, when a new_col field is added to the table structure, the system only updates the definition of that field, while retaining the original definitions of other fields. Simultaneously, only write-ahead log records related to table structure changes (such as records corresponding to ALTER TABLE operations) are parsed, avoiding the parsing of irrelevant records (such as records corresponding to INSERT operations). This example avoids redundant operations and reduces memory usage and CPU overhead through partial updates and record filtering. For example, when the database executes an ALTER TABLE operation, the system only updates the definitions of the affected fields, rather than performing a full scan of the entire data dictionary, thereby improving the efficiency of dynamic data dictionary maintenance.
[0090] An example will be used to illustrate this, such as Figure 3 As shown, Figure 3 This is a flowchart illustrating a log parsing method provided in one embodiment of this application. First, a basic data dictionary is loaded, or a directory for loading the data dictionary is provided. If no data dictionary is loaded, Walminer automatically loads a basic data dictionary (initial data dictionary) in the specified directory based on the WAL file to be parsed. Each WAL log record is read sequentially, and information such as the data, transaction number, table (user data table and system table associated with the data dictionary), and SQL type of the current log record is obtained and cached. Data generated within a transaction is cached together. If a WAL record corresponds to data from a system table where table information is stored in the data dictionary, a hash table is used to temporarily cache the mapping relationship between the table's physical filename and table OID in that record. When updating the data dictionary, filtering out WAL records that do not belong to or cannot be parsed can be done based on the data dictionary and hash table content (in cases where no corresponding mapping relationship is found), improving parsing efficiency and avoiding filtering out WAL records that could originally be parsed (such as when the table's physical filename changes or a new table is added).
[0091] If the parsing involves transaction commit logs, the SQL is restored on a transaction-by-transaction basis, parsing only the correctly committed SQL statements. During the actual SQL restoration process, based on the metadata in the parsed data (which database table it belongs to), it is determined whether the data belongs to the initial data dictionary. If so, the data is updated in the dynamic data dictionary. If the data is related to the data dictionary, the currently used data dictionary is directly updated. For addition types (such as adding a table or table field), the data is directly written to the data dictionary. For update types (such as updating a table name or table field name), the corresponding content in the original data dictionary is updated. For deletion types (such as deleting a table field), the corresponding content in the original data dictionary is deleted. This data dictionary can be immediately applied to the restoration of DML statements and written to the result table.
[0092] If the data is from a user table, the DML restoration process is executed. Based on the parsed transaction number, data type, and SQL type, and the dynamic data dictionary, the corresponding INSERT, UPDATE, and DELETE statements are restored. During this process, the data dictionary and the data to be restored are always matched. Finally, the results are written to the result table. The result table clearly shows the DML statements before and after the table definition changes.
[0093] Walminer selects its base data dictionary according to the following rules: If the user provides the required data dictionary, it uses that dictionary as the basis for parsing the WAL file. If only a data dictionary directory is provided, Walminer finds a suitable data dictionary based on the WAL file to be parsed, following these rules: the timeline of the data dictionary generation must be consistent with the timeline of the WAL file; the first WAL file segment to be parsed must be consistent with the WAL file segment generated when the data dictionary was generated; if no consistent data dictionary exists, it finds the data dictionary closest to the current WAL file for parsing.
[0094] like Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of a log parsing device provided in an embodiment of this application. The log parsing device 400 includes: a loading module 401 for loading an initial data dictionary; a parsing module 402 for parsing a write-ahead log file and generating table structure update content; a generation module 403 for generating a dynamic data dictionary based on the table structure update content and the initial data dictionary; and a restoration module 404 for restoring the data operation statements of the transaction log in the write-ahead log file based on the dynamic data dictionary.
[0095] In one embodiment, the parsing module 402 is used to obtain table structure information based on transaction commit records in the write-ahead log file; the table structure information includes data field names, field types and constraint information; and determine the table structure update content based on the table structure information.
[0096] In one embodiment, the parsing module 402 is used to determine whether table structure information exists in the table structure of the initial data dictionary based on the object identifier in the table structure information; if the object identifier does not exist in the table structure of the initial data dictionary, the data field name, field type and constraint information are used as the table structure update content; if the object identifier exists in the table structure of the initial data dictionary, the change record in the table structure information is used as the table structure update content.
[0097] In one embodiment, the parsing module 402 is used to create a hash table; wherein the hash table is used to cache the mapping relationship between physical file names and object identifiers; when parsing a write-ahead log file, the target write-ahead log file is determined by the hash table.
[0098] In one embodiment, the parsing module 402 is used to determine the target write-ahead log file and update the mapping in the hash table when the table physical file name or the mapping relationship between the table physical file name and the object identifier changes.
[0099] In one embodiment, the generation module 403 is used to update only the field definitions in the initial data dictionary that are related to the table structure update content; and to parse only the write-ahead log records related to the table structure change.
[0100] The log parsing device 400 provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.
[0101] This application provides an electronic device, including: a memory and a processor;
[0102] The memory stores instructions that the computer executes;
[0103] The processor executes computer execution instructions stored in memory, causing the processor to perform any of the methods described above.
[0104] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device 500 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 500 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.
[0105] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.
[0106] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0107] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0108] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0109] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0110] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0111] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0112] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0113] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0114] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0116] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0117] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0118] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0119] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.
Claims
1. A log parsing method, characterized in that, include: Load the initial data dictionary; Parse the write-ahead log file to generate table structure update content; Based on the table structure update content and the initial data dictionary, a dynamic data dictionary is generated; Based on the dynamic data dictionary, the data manipulation statements of the transaction processing log in the write-ahead log file are restored.
2. The method according to claim 1, characterized in that, The process of parsing the write-ahead log file and generating table structure update content includes: Based on the transaction commit records in the write-ahead log file, obtain the table structure information; the table structure information includes data field names, field types, and constraint information. Based on the table structure information, determine the content to be updated in the table structure.
3. The method according to claim 2, characterized in that, The step of determining the table structure update content based on the table structure information includes: Based on the object identifier in the table structure information, determine whether the table structure information exists in the table structure of the initial data dictionary; If the table structure information does not exist in the table structure of the initial data dictionary, then the data field name, the field type, and the constraint information are used as the table structure update content; If the table structure information exists in the table structure of the initial data dictionary, then the change records in the table structure information are used as the table structure update content.
4. The method according to claim 1, characterized in that, Before parsing the write-ahead log file, the following steps are also included: Create a hash table; wherein the hash table is used to cache the mapping relationship between table physical file names and object identifiers; When parsing a write-ahead log file, a hash table is used to determine the target write-ahead log file.
5. The method according to claim 4, characterized in that, The step of determining the target write-ahead log file using a hash table specifically includes: When the physical file name of the table, or the mapping relationship between the physical file name of the table and the object identifier, changes, the target write-ahead log file is determined, and the mapping relationship in the hash table is updated.
6. The method according to claim 1, characterized in that, The step of generating a dynamic data dictionary based on the updated content of the table structure and the initial data dictionary specifically includes at least one of the following: Only update the field definitions in the initial data dictionary that are related to the table structure update content; Only write-ahead log entries related to table structure changes are parsed.
7. A log parsing device, characterized in that, include: The loading module is used to load the initial data dictionary; The parsing module is used to parse the write-ahead log file and generate table structure update content; The generation module is used to generate a dynamic data dictionary based on the updated content of the table structure and the initial data dictionary; The restore module is used to restore the data operation statements of the transaction processing log in the write-ahead log file based on the dynamic data dictionary.
8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.
10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-6.