Database log processing methods, database change query methods, and related devices

Abstract

This application discloses a database log processing method, a database table change query method, and related apparatus to address the problems of low efficiency and accuracy in existing database change retrieval schemes. The database log processing method includes: parsing the data change logs of a business database to obtain event information of change events in the business database. Each change event represents a data change operation performed on the business database, and the event information includes the execution time of the change event and the metadata information of the business data table changed by the change event; creating an index node corresponding to the change event in the business database, whereby the index node stores the metadata information of the business data table changed by the corresponding change event; and creating an event index table based on the index node and the execution time of the change event.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a database log processing method, a database change query method, and related apparatus. Background Technology

[0002] With the rapid development of internet applications, various databases have emerged and been widely adopted. Database administrators (DBAs) maintain numerous database instances within business databases. Therefore, they not only need to monitor the operational status of each database instance but also respond to various data processing needs from business lines, such as exporting change event data within a specific time range or synchronizing data from a certain period to other databases. In their daily management work, DBAs frequently encounter tasks such as retrieving data change operation data from business databases over a past period to facilitate backup and restore operations.

[0003] Currently, retrieving data change operation data requires DBAs to manually filter and retrieve a large amount of business database logs using relevant database tools, and then parse and restore the filtered logs to obtain the data change operation data of the business database for a certain period of time. However, this manual retrieval and parsing method is highly dependent on human experience, resulting in low efficiency and accuracy. Summary of the Invention

[0004] The purpose of this application is to provide a database log processing method, a database table change query method, and related apparatus to solve the problems of low efficiency and accuracy in existing database change retrieval schemes.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, embodiments of this application provide a database log processing method, including:

[0007] Parse the data change log of the business database to obtain the event information of the change events of the business database. The change events are used to indicate the data change operations performed on the business database. The event information of the change events includes the execution time of the change events and the metadata information of the business data tables changed by the change events.

[0008] Create an index node corresponding to the change event of the business database. The index node is used to store the metadata information of the business data table changed by the corresponding change event.

[0009] An event index table is created based on the index node and the execution time of the change event.

[0010] As can be seen, in this embodiment, since the data change log of the business database records the data change operations performed on the business database in the form of events, by parsing the data change log of the business database, the execution time of the change event and the metadata information of the business data table changed by the change event can be obtained. Furthermore, by creating corresponding index nodes for the change events to store the metadata information of the business data tables changed by the change events, and creating an event index table based on the index nodes and the execution time of the change events corresponding to the index nodes, the change events executed at a certain time in the past and the metadata information of the business data tables they changed can be quickly queried based on the event index table. Then, based on the queried change events and the metadata information of the business data tables they changed, the data change operation performed on the business database can be quickly determined. It is evident that the entire process requires no manual intervention, nor does it require additional logical processing or log data restoration processing, making it simple, fast, efficient, and accurate.

[0011] Secondly, embodiments of this application provide a database change query method, including:

[0012] The event index table is queried based on the target time to obtain the target index node corresponding to the target time. The event index table is created based on the index node corresponding to the change event in the business database and the execution time of the change event. The index node is used to store the metadata information of the business data table changed by the corresponding change event. The execution time of the change event and the metadata information of the business data table changed by the change event are obtained by parsing the data change log of the business database.

[0013] Based on the event type of the target change event corresponding to the target index node and the metadata information of the target business data table changed by the target change event, the data operation statement corresponding to the target change event is determined.

[0014] As can be seen from the embodiments of this application, since the index nodes in the event index table store the metadata information of the business data table changed by the corresponding change event, the target change event executed at the target time and the metadata information of the business data table it changes can be determined by querying the event index table. Then, based on the metadata information of the target change event and the business data table it changes, the data operation statement executed on the business database can be quickly determined. It can be seen that the whole process does not require manual intervention, nor does it require additional logic processing and log data restoration processing, etc., making it simple, fast, efficient and accurate.

[0015] Thirdly, embodiments of this application provide a database log processing apparatus, comprising:

[0016] The parsing unit is used to parse the data change log of the business database to obtain the event information of the change events of the business database. The change events are used to represent the data change operations performed on the business database. The event information of the change events includes the execution time of the change events and the metadata information of the business data tables changed by the change events.

[0017] The first creation unit is used to create an index node corresponding to the change event of the business database. The index node is used to store the metadata information of the business data table changed by the corresponding change event.

[0018] The second creation unit is used to create an event index table based on the index node and the execution time of the change event.

[0019] Fourthly, embodiments of this application provide a database change query device, comprising:

[0020] The query unit is used to query the event index table based on the target time to obtain the target index node corresponding to the target time. The event index table is created based on the index node corresponding to the change event in the business database and the execution time of the change event. The index node is used to store the metadata information of the business data table changed by the corresponding change event. The execution time of the change event and the metadata information of the business data table changed by the change event are obtained by parsing the data change log of the business database.

[0021] The first statement determination unit is used to determine the data operation statement corresponding to the target change event based on the event type of the target change event corresponding to the target index node and the metadata information of the target business data table changed by the target change event.

[0022] Fifthly, embodiments of this application provide an electronic device, including:

[0023] processor;

[0024] Memory used to store the processor's executable instructions;

[0025] The processor is configured to execute the instructions to implement the method as described in the first or second aspect.

[0026] In a sixth aspect, embodiments of this application provide a computer-readable storage medium that, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method described in the first or second aspect. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0028] Figure 1 A schematic diagram illustrating an application scenario to which the method provided in the embodiments of this application is applicable;

[0029] Figure 2 A schematic diagram illustrating another application scenario to which the method provided in the embodiments of this application is applicable;

[0030] Figure 3 A schematic flowchart of a database log processing method is provided in one embodiment of this application;

[0031] Figure 4 A schematic diagram of the structure of an event index table provided in one embodiment of this application;

[0032] Figure 5 A schematic flowchart illustrating a database change query method provided in one embodiment of this application;

[0033] Figure 6 A flowchart illustrating a database change query method provided for another embodiment of this application;

[0034] Figure 7 A schematic diagram of the structure of a database log processing device provided in one embodiment of this application;

[0035] Figure 8 A schematic diagram of a database change query device provided in one embodiment of this application;

[0036] Figure 9 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] The terms "first," "second," etc., used in this specification and claims are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in this specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0039] Explanation of some concepts:

[0040] MySQL is one of the most popular relational database systems in the industry, characterized by its open-source nature, excellent performance, and flexible system function expansion.

[0041] DML: Data Manipulation Language, which mainly includes statements for adding, deleting, updating, and querying data records.

[0042] BINLOG: MySQL records transaction changes during database transactions. BINLOG records all Data Definition Language (DDL) and Data Manipulation (DML) statements, and records them as events.

[0043] CDC: Change Data Capture. It can retrieve incremental change data from the database non-intrusively, based on BINLOG logs.

[0044] As mentioned earlier, currently, retrieving data change operation data requires DBAs to manually filter and retrieve a large number of business database logs using relevant database tools, and then parse and restore the filtered logs to obtain the data change operation data of the business database for a certain period of time. However, this manual retrieval and parsing method is highly dependent on human experience, resulting in low efficiency and accuracy.

[0045] To address the low efficiency and accuracy issues of existing database change retrieval schemes, this application proposes a database log processing scheme and a database table change query scheme. The database log processing scheme proposed in this application records data change operations performed on the business database in the form of events. By parsing the data change logs, the execution time of the change events and the metadata information of the business data tables modified by the events can be obtained. Furthermore, by creating corresponding index nodes for change events to store the metadata information of the business data tables modified by the events, and creating an event index table based on the index nodes and the execution time of the corresponding change events, a change event executed at a certain time in the past and the metadata information of the business data tables it modified can be quickly retrieved based on the event index table. Then, based on the retrieved change event and the metadata information of the modified business data tables, the data change operation performed on the business database can be quickly determined. It is evident that the entire process requires no manual intervention, nor does it require additional logical processing or log data restoration processing, making it simple, fast, efficient, and accurate.

[0046] Based on the event index table created by the database log processing scheme proposed in this application embodiment, this application embodiment also proposes a database change query method. Since the index nodes in the event index table store the metadata information of the business data table changed by the corresponding change event, the target change event executed at the target time and the metadata information of the business data table it changed can be determined by querying the event index table. Then, based on the metadata information of the target change event and the business data table it changed, the data operation statement executed on the business database can be quickly determined. It can be seen that the whole process does not require manual intervention, nor does it require additional logic processing and log data restoration processing, etc., making it simple, fast, efficient and accurate.

[0047] It should be understood that the database log processing method and database change query method provided in the embodiments of this application can both be executed by electronic devices or software installed in electronic devices. The term "electronic device" here may include terminal devices, such as smartphones, tablets, laptops, desktop computers, smart voice interaction devices, smart home appliances, smartwatches, vehicle terminals, aircraft, etc.; or, the electronic device may also include servers, such as independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing cloud computing services, etc.

[0048] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0049] The database log processing method and database change query method provided in the embodiments of this application can both be applied to... Figure 1 and Figure 2 The application scenario shown includes: database system 1 and database management platform 2.

[0050] Database system 1 includes a business database 11. Business database 11 stores at least one business data table, which records various business data during the business processing. For example, a user information table records authorized user information (such as username, user ID, user's region, and user's consumption level). During business processing, the business process provider will inevitably perform data operations on business database 11, such as data addition (adding data to the business data table), data deletion (deleting data from the business data table), data update (updating data from the business data table), and data query (querying data from the business data table).

[0051] Database system 1 generates a corresponding data change log 12 based on the data operations performed on business database 11. The data change log 12 records the data operation information performed on business database 11 in the form of events, such as the changed business data tables and the data operation statements that implement the change operations.

[0052] In practical applications, database system 1 can be of any suitable type, and can be configured according to actual needs. This application embodiment does not limit this. For example, business database system 1 can be MySQL, and correspondingly, the generated data change log 12 can be BINLOG, and the data operation statements can be Structured Query Language (SQL), etc.

[0053] like Figure 1 As shown, the database management platform 2 can be deployed in an electronic device. The electronic device may include terminal devices, or it may also include servers, such as a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services, etc.

[0054] Specifically, the database management platform 2 can have any suitable architecture, which can be configured according to actual needs, and this application embodiment does not limit this. In an optional implementation, the database management platform 2 may include a DB_CDC component, an INDEX component, a FILE_SAVER component, and a QUERY component.

[0055] The DB_CDC component can be used to retrieve the data change log 12 of the business database 11 from the database system 1, and parse the data change log 12 to obtain the event information of the change events in the business database 11. For example, the event information of the change events may include, but is not limited to, the execution time of the change event, the event type of the change event, the data change log information to which the change event belongs, the offset of the change event in its respective data change log, the length of the change event, and the metadata information of the business data table changed by the change event. The event types include data addition events, data deletion events, and data change events, etc.

[0056] The INDEX component can be used to create an event index table based on event information from change events in the business database 11. Specifically, such as... Figure 2 As shown, the event index table includes multiple index nodes (e.g., Figure 2 The system contains index nodes (E0, E1, E2, etc.) and their index values ​​(index_val). Each index node corresponds one-to-one with a change event, and each index node stores the event information of its corresponding change event. For example, the index value of each index node can be the execution time of the change event corresponding to each index node. Each index node can include an Index Entry table and a Table Entry table. The Index Entry table can include, but is not limited to, fields such as index_val, binlog_file, size, table_schema, level[], and next[]. The index_val field records the index value of the index node, the binlog_file field records the data change log to which the change event corresponding to the index node belongs, the size field records the length of the change event corresponding to the index node, the table_schema field records the business data table changed by the change event corresponding to the index node, level[] records the number of index layers of the index node, and next[] records the next index node pointed to by the index node. The Entry table can include, but is not limited to, multiple fields such as tabname, col_id, col_name, col_type, and col_value. The tabname field is used to record the name of the business data table that was changed by the change event corresponding to the index node. The col_id field is used to record the field identifier of the field in the business data table. The col_name field is used to record the field name of the field in the business data table. The col_type field is used to record the field type of the field in the business data table. The col_value field is used to record the field size of the field in the business data table, etc.

[0057] In practical applications, the event index table can have any suitable structure, which can be selected according to actual needs, such as an ordered linked list, a skip list, etc. This application embodiment does not limit this. Of course, it should be understood that if the event index table is a skip list, then each index node can also store its own index level (level[]) and the next index node pointed to by each index level (next[]), etc.

[0058] The FILE_SAVER component can be used to perform local persistent storage of the data change log 12 of the business database 11, and to generate a corresponding log list based on the storage result. This log list is used to record relevant information about the data change log 12 of the business database 11. For example, such as... Figure 2 As shown, the log list can include multiple fields such as file_name, file_size, and file_local_path. The file_name field is used to record the file name of the stored data change log 12, the file_size field is used to record the size of the stored data change log 12, and the file_local_path field is used to record the storage location of the stored data change log 12 in the local persistent storage area.

[0059] The QUERY component can be used to query the change events and event information of the business database 11 at a specified time from the event index table, and then determine the data change operation performed on the business database 11 at the specified time based on the change events and event information at the specified time.

[0060] The database log processing method provided in this application embodiment will describe in detail how the database management platform 2 establishes an event index table, and the database change query method provided in this application embodiment will describe in detail how the database management platform 2 determines the data operations performed on the business database 11 at a specified time, etc.

[0061] Please refer to Figure 3 This is a flowchart illustrating a database log processing method according to an embodiment of this application. This method can be applied to electronic devices, specifically to a database management platform deployed in an electronic device (e.g.,...). Figure 1 and Figure 2 In the database management platform 2), the method may include the following steps:

[0062] S302, parse the data change log of the business database to obtain the event information of the change events in the business database.

[0063] In this embodiment, a change event is used to represent a data change operation performed on the business database, such as a data deletion operation, a data update operation, and a data addition operation. The event information of a change event includes the execution time of the change event and the metadata information of the business data table modified by the change event. Optionally, by parsing the data change log of the business database, the event information of the change event may also include the event type of the change event, the length of the change event, the data change log information to which the change event belongs, and the offset of the change event in its respective data change log. Specifically, the metadata information of the business data table modified by the change event may include, but is not limited to, the table name of the business data table modified by the change event, the field identifier, field name, field type, and field size of the fields in the business data table that have been changed.

[0064] Typically, a business database's data change log contains a log file header and a series of change events. Each change event in the data change log contains an event header and a data portion specific to the event type (i.e., the event body). The event header records the execution time (timestamp), event type (event_type), source database system host identifier (serverid), event size (event_size), and the position of the next change event (log_pos). The event body records the position of the first change event recorded in the next data change log, the filename of the next data change log, and the metadata information of the business data table to which the change event targets. Based on this, by parsing the data change log, the event information of the business database's change events can be obtained quickly and accurately.

[0065] Specifically, in one alternative implementation, S302 may include the following steps:

[0066] S321, parse the data change log of the business database to obtain the event header and event body of the change events recorded in the data change log.

[0067] For example, using the DB_CDC component in the aforementioned database management platform 2, based on the type of the business database, the event headers of a series of change events recorded in the data change log of the business database are parsed out. For example, in the data change log of a MySQL database, the first 23 bytes from the offset of each change event in the data change log are usually the event header. Then, based on the event body length recorded in the event header, the corresponding number of bytes are extracted from the data change log as the event body of the change event. The event header and its corresponding event body constitute a complete change event.

[0068] S322, parse the event header of the change event to obtain the execution time of the change event.

[0069] S323, parse the event body of the change event to obtain the business data table that was changed by the change event.

[0070] After obtaining the event header and event body of a change event through the DB_CDC component, the database management platform can send the event header and event body of the change event to the INDEX component. The INDEX component parses the event header of the change event to obtain the execution time of the change event recorded therein, and parses the event body of the change event to obtain the table identifier (${table}) of the business data table changed by the change event recorded therein.

[0071] S324. Based on the business data table changed by the change event, query the business database to obtain the metadata information of the business data table changed by the change event.

[0072] The database management platform can use the INDEX component to query the business database based on the table identifier of the business data table that has been changed by the change event, and obtain the metadata information of the business data table that the change event targets.

[0073] This application embodiment illustrates one specific implementation of S302 described above. Of course, it should be understood that S302 can also be implemented in other ways, and this application embodiment does not limit this implementation.

[0074] In practical applications, optionally, the database management platform can obtain the data change logs of the business database from the database system when needed, and then parse the data change logs of the business database through the above S302 to obtain the event information of the change events of the business database.

[0075] Optionally, to achieve dynamic monitoring of the business database and ensure the real-time nature of subsequently created event index tables, thereby providing reliable assurance for subsequent backup and recovery of the business database, the database management platform can also obtain real-time data change logs generated by the business database from the database system. Specifically, the database management platform can use the DB_CDC component to send the DUMP_GTID command to the database system according to the database system's communication protocol requirements, instructing the database system to push the business database's data change logs to the database management platform in real time.

[0076] Furthermore, to ensure the long-term retention of data change logs for the business database, thereby providing reliable support for subsequent operation and maintenance of the business database, the data change logs can be stored in a local persistent storage area after they are obtained. For example, the local persistent storage area can be a designated region on the disk of the data management platform. The database management platform can use the FILE_SAVER component and the FDATASYNC model to store the data change logs of the business database in the designated region on its disk.

[0077] More specifically, regarding the scenario where the database management platform retrieves real-time data change logs generated by the business database from the database system, the platform stores each retrieved data change log in the local persistent storage area using the FILE_SAVER component. For example, the database management platform can store the retrieved data change logs in the local persistent storage area according to the order in which they were retrieved or by their filename.

[0078] After storing the acquired data change logs to the local persistent storage area via FILE_SAVER, the database management platform can determine the storage location of the data change logs in the local persistent storage area. Furthermore, to facilitate quick and accurate retrieval of the data change logs from the local persistent storage area, after storing the currently acquired data change logs in the local persistent storage area, the database log processing method provided in this application embodiment may further include: updating the content of the log list based on the attribute information of the currently acquired data change logs and their storage location in the local persistent storage area. The attribute information of the data change logs may include, for example, but not limited to, the filename, size, and generation time of the data change logs.

[0079] The log list can be pre-created by the database management platform using the FILE_SAVER component before retrieving data change logs from the business database, and updated after the retrieved data change logs are stored. The log list can be used to record stored data change logs.

[0080] The log list can have any suitable form, and can be set according to actual needs. This application embodiment does not limit this. For example, such as... Figure 2As shown, the log list can include multiple fields such as file_name, file_size, and file_local_path. The file_name field records the filename of the stored data change log, the file_size field records the size of the stored data change log, and the file_local_path field records the storage location of the stored data change log in the local persistent storage area. Alternatively, the log list can also be as shown in Table 1 below, where / home / data / represents the storage location of the stored data change log in the local persistent storage area, binlog.000001 and binlog.000017 represent the filenames of the stored data change logs, 0000001F000000F represents the size of data change log binlog.000001, and 00000020F45470F represents the size of data change log binlog.000017.

[0081] Table 1

[0082]

[0083] It is understandable that by writing the attribute information of the stored data change log and the storage location of the data change log in the local persistent storage area into the log list, the required data change log can be quickly retrieved from the local persistent storage area by querying the log list later.

[0084] In practical applications, database management platforms can use various storage methods to persistently store the currently acquired data change logs locally.

[0085] Optionally, the database management platform can directly store the currently acquired data change logs to the local persistent storage area. In this case, after storing the data change logs, the database management platform can directly write the attribute information of the data change logs and their storage location in the local persistent storage area into the log list.

[0086] Optionally, to fully utilize the storage space of the local persistent storage area, the database management platform can also store the currently acquired data change log in the local persistent storage area based on the size of the currently acquired data change log. For example, if the size of the currently acquired data change log exceeds a preset threshold, the currently acquired data change log is split into multiple sub-logs, and these sub-logs are further stored in the local persistent storage area. In this case, to ensure the accuracy of the log list content, so that the specified data change log can be accurately retrieved from the local persistent storage area through the log list, the database management platform can obtain the attribute information of each sub-log and the storage location of each sub-log in the local persistent storage area, and write the attribute information and storage location of each sub-log into the log list.

[0087] For example, when splitting the currently acquired data change log, the database management platform can divide the data change log into multiple sub-logs according to the execution order of the change events recorded in the data change log, with each sub-log recording one or more change events. Correspondingly, when updating the log list, the database management platform can determine the recording order of each sub-log according to the execution order of the change events recorded in each sub-log, and then write the attribute information of each sub-log and its storage location in the local persistent storage area into the log list according to the recording order of each sub-log.

[0088] S304, Create the index node corresponding to the change event of the business database.

[0089] In this embodiment, there is a one-to-one correspondence between index nodes and change events. The index node is used to store the metadata information of the business data table changed by the corresponding change event. Optionally, the index node can also be used to store other event information such as the event type, execution time, length, data change log to which the change event belongs, and the offset of the change event in its data change log.

[0090] In this embodiment of the application, the index node can have any suitable structure, for example Figure 2 The index nodes shown can be set according to actual needs, and this application embodiment does not limit them.

[0091] S306, Create an event index table based on the index node and the execution time of the change event.

[0092] Specifically, to facilitate candidate queries for event information of change events performed on the business database at a specified time, an index order is created between the index nodes corresponding to the change events in the business database based on the execution time of the change events, thereby obtaining the event index table.

[0093] In this embodiment, the event index table can have any suitable structure, and can be set according to actual needs. This embodiment does not limit this. In an optional implementation, S306 above includes:

[0094] S361 creates pointers between the index nodes corresponding to the change events in the business database based on the execution time of the change events.

[0095] Optionally, to enable quick querying of event information for change events at a specified time, the event index table can be an ordered linked list, with index nodes connected by pointers. Accordingly, in S361 above, the execution time of the change event corresponding to each index node can be used as the index value of each index node, and the index nodes can be sorted in ascending order of index value; then, pointers are created between adjacent index nodes, with the pointer pointing from the index node with the smaller index value to the index node with the larger index value.

[0096] Optionally, considering that querying change events based on an ordered linked list requires traversing the list from the first index node until the desired change event is found, which is complex and inefficient, the event index table can be a skip list to reduce the complexity of the query process and improve query efficiency. Each index node can have one or more index layers, and the index layers of each index node are connected by pointers. Accordingly, in S361 above, the index value and number of index layers of the index node corresponding to each change event can be determined based on the execution time of each change event in the business database. Furthermore, based on the index value and number of index layers of each index node, pointers are created between different index nodes that meet the preset index conditions. Specifically, if two index nodes have the same number of index layers, then these two index nodes are determined to meet the preset index conditions. It should be noted that for each index node, when there are multiple index layers, the order of each index layer is arranged from bottom to top. For example, if the number of index layers of an index node is 3, then the bottom index layer is the first index layer, the middle index layer is the second index layer, and the top index layer is the third index layer.

[0097] For example, the execution time of the change event corresponding to each index node can be used as the index value of each index node, and the number of index layers for each index node can be obtained using a random function. Figure 4The skip list shown is an example of an event index table. Initially, an empty linked list is created. Then, for the index node with index value 3 (hereinafter referred to as index node 3), a random number of index layers (1) is generated. Index node 3 is added between the beginning node and the end node NIL of the index table, and a pointer to index node 3 is created between the first index layer (bottom) of the beginning node and index node 3. Next, for the index node with index value 6 (hereinafter referred to as index node 6), a random number of index layers (2) is generated. Index node 6 is added after index node 3, and a pointer to index node 6 is created between index node 3 and the first index layer of index node 6, and a pointer to index node 6 is also created between the beginning node and the second index layer of index node 6. This process continues, resulting in the following... Figure 4 The pointers between the various index nodes are shown.

[0098] S362 creates an event index table based on the index nodes and the pointers between them.

[0099] The event index table can include the index nodes corresponding to change events in the business database and pointers between these index nodes. Thus, when querying a change event for a specific time, one can traverse the index nodes in the event index table based on the pointers between them, and obtain the event information for the change event at the specified time based on the chronological order between the execution time of the change event corresponding to the currently traversed index node and the specified time.

[0100] This application embodiment illustrates one specific implementation of S306 described above. It should be understood that S306 can also be implemented in other ways, and this application embodiment does not limit this implementation.

[0101] The database log processing method provided in this application records data change operations performed on the business database in the form of events. By parsing the data change logs, the execution time of the change events and the metadata information of the business data tables changed by the change events can be obtained. Furthermore, by creating corresponding index nodes for the change events to store the metadata information of the business data tables changed by the change events, and creating an event index table based on the index nodes and the execution time of the change events corresponding to the index nodes, the change events executed at a certain time in the past and the metadata information of the business data tables they changed can be quickly queried based on the event index table. Then, based on the queried change events and the metadata information of the business data tables they changed, the data change operations performed on the business database can be quickly determined. It can be seen that the entire process does not require manual intervention, nor does it require additional logical processing or log data restoration processing, making it simple, fast, efficient, and accurate.

[0102] Based on the event index table created using the database log processing method described above, this application embodiment also provides a database change query method. Please refer to... Figure 5 This is a flowchart illustrating a database change query method according to an embodiment of this application. This method can be applied to electronic devices, specifically to a database management platform deployed in the electronic device (e.g.,...). Figure 1 and Figure 2 In the database management platform 2), the method may include the following steps:

[0103] S502, query the event index table based on the target time to obtain the target index node corresponding to the target time.

[0104] The event index table is created based on the index nodes corresponding to change events in the business database and the execution time of the change events. The index nodes store the metadata information of the business data tables changed by the corresponding change events. The execution time of the change events and the metadata information of the business data tables changed by the change events are obtained by parsing the data change logs of the business database.

[0105] In this embodiment, the target time refers to the specified time to be queried. Specifically, in S502 above, the index node whose index value matches the target time can be found in the event index table according to the pointers between the index layers of each index node in the event index table, and this index node is used as the target index node corresponding to the target time.

[0106] For example, with Figure 4Taking the event index table shown as an example, if the target time = 12, then starting from the third index level (the top index level) of the first node in the event index table, we find the index node pointed to by the third index level of the first node, which is index node 9 (i.e., the index node with an index value of 9), and compare the target time with the index value of index node 9. Since the target time is greater than the index value of index node 9, we then start from the third index level of index node 9 and find the third index level of index node 21 (i.e., the index node with an index value of 21), and compare the target time with the index value of index node 21. Since the target time is smaller... Given the index value of index node 21, we then start from the second index layer (intermediate index layer) of index node 9 and find the second index layer that points to index node 17 (i.e., the index node with index value 17). We compare the target time with the index value of index node 17. Since the target time is less than the index value of index node 17, we then start from the third index layer (bottom index layer) of index node 9 and find the index layer that points to index node 12 (i.e., the index node with index value 12). Since the target time is equal to the index value of index node 12, we determine index node 12 as the target index node corresponding to the target time.

[0107] S504. Based on the event type of the target change event corresponding to the target index node and the metadata information of the target business data table changed by the target change event, determine the data operation statement corresponding to the target change event.

[0108] After the target index node is determined, the event information of the target change event executed at the target time can be obtained from the target index node. Specifically, it can include the event type of the target change event and the metadata information of the target business data table that was changed by the target change event.

[0109] For example, you can query the field in the Index Entry table of the target index node that records the event type to obtain the event type of the target change event, and query the field in the Index Entry table of the target index node that records the business data table changed by the change event (e.g., ... Figure 2The `table_schema` field shown indicates the target business data table that was changed by the target change event corresponding to the target index node. Then, in the `Table Entry` table of the target index node, the metadata information such as the table name, field identifier, field name, field type, and field size of the target business data table are queried from multiple fields such as `tabname`, `col_id`, `col_name`, `col_type`, and `col_value`.

[0110] Since data manipulation statements executed on business data tables in a business database typically have a specific syntax structure, and these statements usually specify the metadata information of the business data table to be modified, for example, the SQL statement used to add data to a business data table is usually 'INSERT INTO business data table_name(field_name) VALUES(field_value)', where tabname represents the table name of the business data table, and columns 1 and 2 represent the field names respectively. Therefore, based on the event type of the target change event and the metadata information of the target business data table that the target change event modifies, the corresponding data manipulation statement can be derived in reverse.

[0111] For example, taking SQL statements as an example, if the event type of the target change event is a data insertion operation, then the corresponding data operation statement can be deduced as 'INSERT INTO table_name(field_name) VALUES(field_value)'; if the event type of the target change event is a data deletion operation, then the corresponding data operation statement can be deduced as 'DELETE FROM table_nameWHERE field_name=field_value'; if the event type of the target change event is a data update operation, then the corresponding data operation statement can be deduced as 'UPDATE table_nameSET field_name=new_field_valueWHERE field_name=original_field_value', and so on.

[0112] To facilitate the restoration of the business database, in another embodiment of this application, such as Figure 6 As shown, after S504 above, the database change query method provided in this application embodiment may further include: determining a rollback statement for restoring the target business data table based on the data operation statement corresponding to the target change event. In this way, by executing the rollback statement on the business database, the business database can be restored to its original state corresponding to the target time.

[0113] Specifically, the corresponding rollback statement can be determined based on the business data table modified by the data manipulation statement, the modification operation performed on the business data table, and the statement syntax followed by the data manipulation statement. For example, if the data manipulation statement adds a field value to a business data table, then the corresponding rollback statement can be determined as adding the field value to that field from the business data table, and the rollback statement has the same statement syntax as the data manipulation statement; if the data manipulation statement deletes a field value from a business data table, then the corresponding rollback statement can be determined as adding the deleted field value to the business data table, and the rollback statement has the same statement syntax as the data manipulation statement; if the data manipulation statement updates the field value 1 of a field in the business data table to the field value 2, then the corresponding rollback statement can be determined as updating the field value 2 of a field in the business data table to the field value, and the rollback statement has the same statement syntax as the data manipulation statement.

[0114] The database change query method provided in this application allows for the determination of the target change event executed at the target time and the metadata information of the business data table that was changed by the corresponding change event by storing the metadata information of the business data table in the index node of the event index table. Based on the metadata information of the target change event and the business data table that was changed, the data operation statement executed on the business database can be quickly determined. It can be seen that the whole process does not require manual intervention, nor does it require additional logic processing or log data restoration processing, and is simple, fast, efficient and accurate.

[0115] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0116] With the above Figure 3 Corresponding to the database log processing method shown, this application also provides a database log processing apparatus. Please refer to... Figure 7 The diagram below illustrates the structure of a database log processing apparatus 700 according to an embodiment of this application. The apparatus 700 includes:

[0117] The parsing unit 710 is used to parse the data change log of the business database to obtain the event information of the change event of the business database. The change event is used to represent the data change operation performed on the business database. The event information of the change event includes the execution time of the change event and the metadata information of the business data table changed by the change event.

[0118] The first creation unit 720 is used to create an index node corresponding to the change event of the business database. The index node is used to store the metadata information of the business data table changed by the corresponding change event.

[0119] The second creation unit 730 is used to create an event index table based on the index node and the execution time of the change event.

[0120] Optionally, the second creation unit includes:

[0121] A pointer creation subunit is used to create pointers between the index nodes based on the execution time of the change event in the business database;

[0122] The index table creation sub-unit is used to create the event index table based on the index node and the pointers between the index nodes.

[0123] Optionally, the pointer creation subunit is specifically used for:

[0124] Based on the execution time of the change event, determine the index value and the number of index layers of the index node;

[0125] Based on the index value of the index node and the number of index layers, pointers are created between the index layers of different index nodes that meet the preset index conditions.

[0126] Optionally, the parsing unit includes:

[0127] The first parsing subunit is used to parse the data change log of the business database to obtain the event header and event body of the change events recorded in the data change log;

[0128] The second parsing subunit is used to parse the event header of the change event to obtain the execution time of the change event;

[0129] The third parsing subunit is used to parse the event body of the change event to obtain the business data table changed by the change event;

[0130] The metadata query subunit is used to query the business database based on the business data table to obtain the metadata information of the business data table.

[0131] Optionally, the device 700 further includes:

[0132] The log acquisition unit is used to acquire the real-time data change logs generated by the business database before the parsing unit parses the data change logs of the business database and obtains the event information of the change events of the business database.

[0133] A storage unit is used to store the currently acquired data change log to a local persistent storage area.

[0134] Optionally, the device 700 further includes:

[0135] The list update unit is used to update the contents of the log list based on the attribute information of the currently acquired data change log and the storage location of the currently acquired data change log in the local persistent storage area after the storage unit stores the currently acquired data change log in the local persistent storage area. The log list is used to record the stored data change log.

[0136] Optionally, the storage unit is specifically used for:

[0137] If the size of the currently acquired data change log exceeds a preset threshold, the data change log is split into multiple sub-logs, and the multiple sub-logs are stored in the local persistent storage area.

[0138] The inventory update unit is specifically used for:

[0139] Obtain the attribute information of each sub-log and the storage location of each sub-log in the local persistent storage area;

[0140] The log list records the attribute information of each sub-log and the storage location of each sub-log in the local persistent storage area.

[0141] Obviously, the database log processing device provided in this application embodiment can be used as the above-mentioned... Figure 3 The entity executing the database log processing method shown above is therefore able to implement the database log processing method as described above. Figure 3 The functions implemented are the same, so they will not be described in detail here.

[0142] The database log processing apparatus provided in this application records data change operations performed on the business database in the form of events through its data change logs. By parsing these logs, event information such as the execution time of change events and the metadata information of the business data tables modified by those events can be obtained. Furthermore, by creating corresponding index nodes for each change event to store the metadata information of the business data tables modified by the event, and by creating an event index table based on the index nodes and the execution time of the corresponding change events, a change event executed at a specific time in the past and its modified metadata information can be quickly retrieved using the event index table. Based on this retrieved change event and its modified metadata information, the data change operation performed on the business database can be quickly determined. Therefore, the entire process requires no manual intervention, no additional logical processing, and no log data restoration processing, making it simple, fast, efficient, and accurate.

[0143] With the above Figure 5 Corresponding to the database change query method shown, this application embodiment also provides a database change query device. Please refer to... Figure 8 The diagram below illustrates the structure of a database change query device 800 according to an embodiment of this application. The device 800 includes:

[0144] The query unit 810 is used to query the event index table based on the target time to obtain the target index node corresponding to the target time. The event index table is created based on the index node corresponding to the change event in the business database and the execution time of the change event. The index node is used to store the metadata information of the business data table changed by the corresponding change event. The execution time of the change event and the metadata information of the business data table changed by the change event are obtained by parsing the data change log of the business database.

[0145] The first statement determination unit 820 is used to determine the data operation statement corresponding to the target change event based on the event type of the target change event corresponding to the target index node and the metadata information of the target business data table changed by the target change event.

[0146] Optionally, the device 800 further includes:

[0147] The second statement determination unit is used to determine a rollback statement for restoring the target business data table based on the data operation statement after the first statement determination unit determines the data operation statement corresponding to the target change event based on the event type of the target change event corresponding to the target index node and the metadata information of the target business data table changed by the target change event.

[0148] Obviously, the database change query device in this application embodiment can be used as described above. Figure 5 The entity executing the database change query method shown above is therefore able to implement the database change query method in the above... Figure 5 The functions implemented are the same, so they will not be described in detail here.

[0149] The database change query device provided in this application embodiment stores the metadata information of the business data table changed by the corresponding change event in the event index table. By querying the event index table, the target change event executed at the target time and the metadata information of the business data table it changed can be determined. Then, based on the metadata information of the target change event and the business data table it changed, the data operation statement executed on the business database can be quickly determined. It can be seen that the whole process does not require manual intervention, nor does it require additional logic processing and log data restoration processing, etc., making it simple, fast, efficient and accurate.

[0150] Figure 9 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 9 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0151] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0152] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0153] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a database log processing unit at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0154] Parse the data change log of the business database to obtain the event information of the change events of the business database. The change events are used to indicate the data change operations performed on the business database. The event information of the change events includes the execution time of the change events and the metadata information of the business data tables changed by the change events.

[0155] Create an index node corresponding to the change event of the business database. The index node is used to store the metadata information of the business data table changed by the corresponding change event.

[0156] An event index table is created based on the index node and the execution time of the change event.

[0157] Alternatively, the processor reads the corresponding computer program from non-volatile memory into memory and runs it, forming a database change query mechanism at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0158] The event index table is queried based on the target time to obtain the target index node corresponding to the target time. The event index table is created based on the index node corresponding to the change event in the business database and the execution time of the change event. The index node is used to store the metadata information of the business data table changed by the corresponding change event. The execution time of the change event and the metadata information of the business data table changed by the change event are obtained by parsing the data change log of the business database.

[0159] Based on the event type of the target change event corresponding to the target index node and the metadata information of the target business data table changed by the target change event, the data operation statement corresponding to the target change event is determined.

[0160] The above is as stated in this application. Figure 3 The illustrated embodiment discloses a method performed by a database log processing apparatus, or as described in this application. Figure 5The method executed by the database change query device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0161] The electronic device can also perform Figure 3 The method, and implement the database log processing device in Figure 3 The illustrated embodiment may also perform the functions of the electronic device, or the electronic device may also perform the functions of the embodiment. Figure 5 The method, and implement the database change query device in Figure 5 The functions of the embodiments shown are not described in detail here.

[0162] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0163] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figure 3The method of the illustrated embodiment is specifically used to perform the following operations:

[0164] Parse the data change log of the business database to obtain the event information of the change events of the business database. The change events are used to indicate the data change operations performed on the business database. The event information of the change events includes the execution time of the change events and the metadata information of the business data tables changed by the change events.

[0165] Create an index node corresponding to the change event of the business database. The index node is used to store the metadata information of the business data table changed by the corresponding change event.

[0166] An event index table is created based on the index node and the execution time of the change event.

[0167] Alternatively, when executed by a portable electronic device that includes multiple applications, the instruction can enable the portable electronic device to perform... Figure 5 The method of the illustrated embodiment is specifically used to perform the following operations:

[0168] The event index table is queried based on the target time to obtain the target index node corresponding to the target time. The event index table is created based on the index node corresponding to the change event in the business database and the execution time of the change event. The index node is used to store the metadata information of the business data table changed by the corresponding change event. The execution time of the change event and the metadata information of the business data table changed by the change event are obtained by parsing the data change log of the business database.

[0169] Based on the event type of the target change event corresponding to the target index node and the metadata information of the target business data table changed by the target change event, the data operation statement corresponding to the target change event is determined.

[0170] In summary, the above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0171] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0172] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0173] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0174] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A database log processing method, characterized in that, include: Parse the data change log of the business database to obtain the event information of the change events of the business database. The change events are used to indicate the data change operations performed on the business database. The event information of the change events includes the execution time of the change events and the metadata information of the business data tables changed by the change events. Create an index node corresponding to the change event of the business database. The index node is used to store the metadata information of the business data table changed by the corresponding change event. The index value of the index node is determined based on the execution time of the change event in the business database; Create pointers between adjacent index nodes in the index nodes arranged according to the size of the index value; Alternatively, based on the index value and the number of randomly generated index layers, pointers can be created between index layers of the same number of layers in different index nodes; An event index table is created based on the index node and the pointers between the index nodes.

2. The method according to claim 1, characterized in that, Parsing the data change logs of the business database yields event information for the change events in the business database, including: Parse the data change log of the business database to obtain the event header and event body of the change events recorded in the data change log; Parse the event header of the change event to obtain the execution time of the change event; Parse the event body of the change event to obtain the business data table that was changed by the change event; Based on the business data table, the business database is queried to obtain the metadata information of the business data table.

3. The method according to claim 1, characterized in that, Before parsing the data change logs of the business database to obtain the event information of the change events in the business database, the method further includes: Obtain the real-time data change logs generated by the business database; Store the currently acquired data change log to the local persistent storage area.

4. The method according to claim 3, characterized in that, After storing the currently acquired data change log to the local persistent storage area, the method further includes: Based on the attribute information of the currently acquired data change log and the storage location of the currently acquired data change log in the local persistent storage area, update the contents of the log list, which is used to record the stored data change log.

5. The method according to claim 4, characterized in that, Storing the currently acquired data change log to the local persistent storage area includes: If the size of the currently acquired data change log exceeds a preset threshold, the data change log is split into multiple sub-logs, and the multiple sub-logs are stored in the local persistent storage area. The update log list content, based on the attribute information of the currently acquired data change log and the storage location of the currently acquired data change log in the local persistent storage area, includes: Obtain the attribute information of each sub-log and the storage location of each sub-log in the local persistent storage area; The log list records the attribute information of each sub-log and the storage location of each sub-log in the local persistent storage area.

6. A database change query method, characterized in that, include: The event index table is queried based on the target time to obtain the target index node corresponding to the target time. The event index table is created as follows: the index value of the index node corresponding to the change event is determined based on the execution time of the change event in the business database; pointers are created between adjacent index nodes arranged according to the index value; or, pointers are created between index layers of the same number of layers in different index nodes based on the index value and a randomly generated number of index layers; the event index table is created based on the index nodes and the pointers between them. The index nodes are used to store metadata information of the business data table changed by the corresponding change event. The execution time of the change event and the metadata information of the business data table changed by the change event are obtained by parsing the data change log of the business database. Based on the event type of the target change event corresponding to the target index node and the metadata information of the target business data table changed by the target change event, the data operation statement corresponding to the target change event is determined.

7. The method according to claim 6, characterized in that, After determining the data operation statement corresponding to the target change event based on the event type of the target change event corresponding to the target index node and the metadata information of the target business data table changed by the target change event, the method further includes: Based on the data manipulation statements, determine the rollback statement used to restore the target business data table.

8. A database log processing device, characterized in that, include: The parsing unit is used to parse the data change log of the business database to obtain the event information of the change events of the business database. The change events are used to represent the data change operations performed on the business database. The event information of the change events includes the execution time of the change events and the metadata information of the business data tables changed by the change events. The first creation unit is used to create an index node corresponding to the change event of the business database. The index node is used to store the metadata information of the business data table changed by the corresponding change event. The second creation unit is used to determine the index value of the index node based on the execution time of the change event of the business database; and to create pointers between adjacent index nodes in the index nodes arranged according to the size of the index value. Alternatively, based on the index value and the number of randomly generated index layers, pointers can be created between index layers of the same number of layers in different index nodes; An event index table is created based on the index node and the pointers between the index nodes.

9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1 to 7.

Images