Method, device and equipment for restoring binary log file and medium

Abstract

The application discloses a binary log file recovery method, device, equipment and medium, and belongs to the field of data processing. The method comprises the following steps: searching for an initial event and a tail event in a data block device; acquiring log file basic information from the initial event and the tail event in the case that the initial event passes error detection check and the tail event passes error detection check; searching for a data block matched with the log file basic information in the data block device, splicing and error detection checking the matched data block according to the block information of the matched data block, and determining a damaged data block; repairing the damaged data block, splicing the successfully repaired damaged data block to a corresponding position, cutting the damaged event in the damaged data block which fails to be repaired, and splicing the damaged data block after the damaged event is cut to the corresponding position, so that a binary log file is recovered. According to the embodiment of the application, the recovery of the binary log file can be realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method, apparatus, device and medium for recovering binary log files. Background Art

[0002] A database is a collection of large amounts of data stored and managed uniformly on hardware devices over a long period of time. Binary log files, or binlogs, record changes to table structures and modifications to table data within the database. These files are crucial logs for database operations and are closely related to operations and maintenance. During database operations and maintenance, errors can corrupt binary log files, leading to data loss and significant business losses. Therefore, a method for recovering binary log files is urgently needed. Summary of the Invention

[0003] The embodiments of the present application provide a binary log file recovery method, apparatus, device, and medium, which can implement binary log file recovery.

[0004] In a first aspect, an embodiment of the present application provides a method for recovering a binary log file, comprising: searching for an initial event and an end event in a data block device; when the initial event passes error detection and the end event passes error detection, obtaining basic information of the log file from the initial event and the end event, the basic information of the log file including a server identifier, a file start time, and a file end time; searching for data blocks matching the basic information of the log file in the data block device, splicing and error-checking the matching data blocks based on block information of the matching data blocks, and determining damaged data blocks, the block information including a server identifier, a first event timestamp, a first event file position, and a last event file position; repairing the damaged data blocks, splicing the successfully repaired damaged data blocks to corresponding positions, trimming the damaged events from the damaged data blocks that failed to be repaired, and splicing the damaged data blocks after the trimmed damaged events to corresponding positions, to recover the binary log file.

[0005] In a second aspect, an embodiment of the present application provides a binary log file recovery device, comprising: a search module for searching for an initial event and an end event in a data block device; an acquisition module for obtaining basic log file information from the initial event and the end event when the initial event passes the error detection check and the end event passes the error detection check, the log file basic information including a server identifier, a file start time, and a file end time; a processing module for searching for data blocks matching the log file basic information in the data block device, performing splicing and error detection on the matching data blocks based on the block information of the matching data blocks, and determining damaged data blocks, the block information including a server identifier, a first event timestamp, a first event file position, and a last event file position; a repair module for repairing damaged data blocks, splicing the successfully repaired damaged data blocks to corresponding positions, trimming damaged events from damaged data blocks that failed to be repaired, and splicing the damaged data blocks after the trimmed damaged events to corresponding positions to recover the binary log file.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the binary log file recovery method of the first aspect is implemented.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the binary log file recovery method of the first aspect is implemented.

[0008] The embodiments of the present application provide a method, apparatus, device and medium for recovering a binary log file, which can first search for the initial event and the end event in the data block device, obtain the basic information of the log file from the initial event that has passed the error check and the end event that has passed the error check, and then search for the matching data blocks in the data block device according to the basic information of the log file, and the data blocks required to recover the binary log file include the matching data blocks. According to the block information of the data blocks, the matching data blocks are spliced ​​and error-checked to determine the damaged data blocks in the matching data blocks, and the damaged data blocks are repaired. If the repair is successful, the successfully repaired damaged data blocks are spliced ​​to the corresponding position; if the repair fails, the damaged event of the damaged data block is cropped, and the cropped damaged data blocks are spliced ​​to the corresponding position. The data blocks required to recover the binary log file obtained by matching from the data block device can be spliced ​​in sequence to achieve the recovery of the binary log file. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0010] Figure 1 A flowchart of a method for recovering binary log files provided in one embodiment of the present application;

[0011] Figure 2 A flowchart of an example of a process for recording block information of a first data block and block information of a last data block into an information set table provided in an embodiment of the present application;

[0012] Figure 3 A flowchart of a method for recovering binary log files provided in another embodiment of the present application;

[0013] Figure 4 A flowchart of a method for recovering a binary log file provided in another embodiment of the present application;

[0014] Figure 5 A flowchart of an example of the process of splicing, error checking, and repairing data blocks provided in an embodiment of the present application;

[0015] Figure 6 A schematic diagram of the structure of a binary log file recovery device provided in one embodiment of the present application;

[0016] Figure 7 A schematic diagram of the structure of a model publishing device in a distributed system provided in one embodiment of the present application. DETAILED DESCRIPTION

[0017] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating examples of the present application. It should be noted that the acquisition, storage, use, processing, etc. of information and data in the embodiments of the present application are authorized by the user or relevant agencies and comply with the relevant provisions of national laws and regulations.

[0018] A database is a collection of large amounts of data stored and managed uniformly on hardware devices over a long period of time. Binary log files, or binlogs, record changes to table structures and modifications to table data within the database. They are crucial logs for the database and are closely related to operations and maintenance. During database operations and maintenance, errors can corrupt the binary log files, leading to data loss and significant business losses. Therefore, a method for recovering binary log files is urgently needed. Currently, some related technologies use file system information to repair files. However, if file system information, such as file index nodes, partition tables, and file system format information, is lost, or if some data blocks are overwritten, corrupting the data block content, the file cannot be repaired. Other related technologies use file carving methods to repair files. However, file carving methods are suitable for recovering files stored in contiguous storage space on storage devices. However, the data blocks stored in log files are not distributed contiguously and are scattered across different storage spaces depending on file system usage. Therefore, file carving methods are difficult to recover log files and cannot repair binary log files.

[0019] The present application provides a binary log file recovery method, apparatus, device, and medium, which can be applied to binary log file recovery scenarios, specifically for recovering the binary log file binlog of a MySQL database. The binary log file recovery method, apparatus, device, and medium can repair the binary log file according to the characteristics of the MySQL database. In the scenario where some data blocks are overwritten, the binary log file can be restored as much as possible, thereby improving the data recovery rate of the binary log file. The basic information of the binary log file can be obtained in the data block device through the initial event and the end event. Based on the basic information, the data blocks required to recover the binary log file can be found in the data block device. Based on the block information of the data blocks, the data blocks are spliced ​​in sequence and error-checked. If damaged data blocks are found in the process, the damaged data blocks are repaired and the repaired data blocks are spliced ​​to the corresponding positions, thereby obtaining a recovered binary log file. Using this binary log file recovery method, even if the file system information is damaged and the distribution of the data blocks stored in the binary log file is not continuous, the binary log file can be recovered, thereby improving the recovery rate of the binary log file.

[0020] The following describes a binary log file recovery method, apparatus, device, and medium provided by the present application.

[0021] In a first aspect, the present application provides a binary log file recovery method, which can be applied to binary log file recovery scenarios and can be specifically executed by a binary log file recovery device, equipment, etc. Figure 1 A flowchart of a method for recovering binary log files provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the binary log file recovery method may include steps S101 to S104.

[0022] In step S101 , an initial event and an end event are searched in a data block device.

[0023] The initial event is the first event in a binary log file and appears only once. For example, the initial event might be FORMAT_DESCRIPTION_EVENT. The tail event is the last event in a binary log file. When the binary log file reaches the maximum size or a flush command is executed, the binary log file is switched. A tail event is added to the current binary log file. The tail event specifies the name and location of the next binary log file. For example, the tail event might be ROTATE_EVENT. The initial and tail events provide basic information about the binary log files to be restored. The number of binary log files to be restored can be one or more, and there is no limit here.

[0024] In some examples, a regular expression generated based on initial event filtering information can be used to match initial events in a data block device; a regular expression generated based on final event filtering information can be used to match final events in a data block device. Initial event filtering information includes the desired recovery time period and the event type of the initial event. Depending on requirements, the initial event filtering information may also include other information, such as the desired server identifier. Final event filtering information includes the desired recovery time period and the event type of the final event. Depending on requirements, the final event filtering information may also include other information, such as the desired server identifier. The desired recovery time period is the time period within which the user wishes to restore binary log files. This can be set based on scenarios and requirements and is not limited here. For example, if a user wishes to restore binary log files for two months before November 2023, the desired recovery time period may be from September 1 to October 31, 2023. The event type represents the type of event. Different events have different event types and can be represented by different identifier values. For example, the event type range may include, but is not limited to, \x00 to \x26. The event type of the initial event may be x0f, and the event type of the final event may be x04. The desired server identifier identifies the server for which the user wishes to restore the binary log file. A regular expression can be generated using the desired recovery time period and event type. This regular expression can be used to match the block device to retrieve the initial and final events. For example, the regular expression "[TIMESTAMP_RANGE]\x0f....\x" can be used to match the initial event within the time period indicated by TIMESTAMP_RANGE. Similarly, the regular expression "[TIMESTAMP_RANGE]\x04....\x" can be used to match the final event within the time period indicated by TIMESTAMP_RANGE. A regular expression can also be generated using the desired recovery time period, event type, and desired server identifier. This regular expression can be used to match the initial and final events within the block device to retrieve the initial and final events. For example, a regular expression consisting of the desired recovery time [TIMESTAMP_RANGE] + event type [TYPE_CODE] + [server identifier] can be used to match the initial or final event within the time period indicated by TIMESTAMP_RANGE for the server corresponding to the server identifier.

[0025] After obtaining the initial event, an error check is performed on the initial event to determine whether it contains errors. If the initial event contains errors, it may be damaged. Similarly, after obtaining the final event, an error check is performed on the final event to determine whether it contains errors. If the final event contains errors, it may be damaged. Error checking can be performed using a cyclic redundancy check 32 (CRC32). When obtaining the initial event and the final event, a data block including the initial event and a data block including the final event can also be obtained. The data block including the initial event can be the first data block in the binary log file, and the data block including the final event can be the last data block in the binary log file. An information collection table can be set. If the initial event passes the error check, the block information of the data block including the initial event is recorded in the information collection table as the block information of the first data block; if the final event passes the error check, the block information of the data block including the final event is recorded in the information collection table as the block information of the last data block.

[0026] For example, Figure 2 This is a flowchart of an example of a process of recording the block information of the first data block and the block information of the last data block into the information set table provided in an embodiment of the present application, as shown in FIG. Figure 2 As shown, the process of recording the block information of the first data block and the block information of the last data block into the information set table may include steps a1 to a9.

[0027] In step a1, the data block size (BS) is obtained. Specifically, the data block size can be read from the file system information of the data block device, or the basic unit of data recovery, namely the data block size, can be determined according to user input.

[0028] In step a2, the data block containing the initial event is searched in the data block device.

[0029] In step a3, an error check is performed on the initial event; if the error check passes, jump to step a4; if the error check fails, jump to step a2.

[0030] In step a4, the block information of the data block containing the initial event is added to the information set table as the first data block information.

[0031] In step a5, it is determined whether the data blocks containing the initial event in the data block device have been traversed; if the traversal is completed, jump to step a6; if the traversal is not completed, jump to step a2.

[0032] In step a6, the data block containing the end event is searched in the data block device.

[0033] In step a7, an error check is performed on the last event; if the error check passes, jump to step a8; if the error check fails, jump to step a6.

[0034] In step a8, the block information of the data block containing the last event is added to the information set table as the last data block information.

[0035] In step a9, it is determined whether the data block containing the end event in the data block device has been traversed; if the traversal is completed, the process ends; if the traversal is not completed, the process jumps to step a6.

[0036] Through the above steps a1 to a9, the block information of the first data block and the block information of the last data block that can be found in the data block device can be recorded in the information collection table, which is convenient for subsequent search of the block information of the first data block and the block information of the last data block.

[0037] In step S102 , when the initial event passes the error detection check and the last event passes the error detection check, basic log file information is obtained from the initial event and the last event.

[0038] The initial event is checked for error to indicate that the initial event is not damaged, and the final event is checked for error to indicate that the final event is not damaged. The basic information of the log file can be obtained from the undamaged initial event and final event. The events in the two-level log file may include elements such as timestamp, event type, type_code, server ID, event length, event_len, position of the event in the file, log_position, flag, error check code, etc., and the lengths of the above elements are all pre-set, so the offset of each element, offset, can be obtained. For example, the elements contained in the event can be shown in Table 1:

[0039] Table 1

[0040] elements Offset: The length of the feature Timestamp 0：4 Event Type 4：1 Server ID 5：4 Event length 9：4 The position of the event in the file 13：4 Flag 17：2 Error checking code Event Length - 4:4

[0041] When the user sets an expected recovery time, the event can be matched based on whether the timestamp is within the expected recovery time period, thereby obtaining the data block within the expected recovery time period. The error check code is used for error checking. If the error check code is a CRC32 code, it can be used for CRC32 verification.

[0042] Basic log file information includes the server identifier, file start time, and file end time. The server identifier indicates the server. The file start time indicates the start time of the binary log file containing the initial event. The file end time indicates the end time of the binary log file containing the final event. Basic log file information may also include, but is not limited to, server version information, binary log file name, and binary log file size. Basic log file information such as the server identifier, file start time, and server version information can be obtained from the initial event, while basic log file information such as the binary log file name, binary log file size, and file end time can be obtained from the final event.

[0043] In step S103, data blocks matching the basic information of the log file are searched in the data block device, and according to the block information of the matching data blocks, the matching data blocks are spliced ​​and error-checked to determine the damaged data blocks.

[0044] The data block device stores multiple data blocks. The data blocks that match the log file basic information include the data blocks required for this binary log file recovery. Data blocks whose server ID in the data block device matches the server ID in the log file basic information and whose timestamps fall within the time period formed by the file start and end times are identified as matching data blocks. For example, the regular expression "[TIMESTAMP][\x00-\x26](server ID)" generated by the time period formed by the file start and end times, supported event types, and the server ID can be used to perform a match in the data block device to filter out matching data blocks. The block information of these matching data blocks can be recorded in the information collection table.

[0045] The block information may include the server identifier, the first event timestamp, the first event file position, and the last event file position. The first event timestamp of a data block is the timestamp of the first event in the data block. The first event file position of a data block is the position of the first event in the data block in the binary log file. The last event file position of a data block is the position of the last event in the data block in the binary log file. Based on the first event timestamp, the first event file position, and the last event file position of the data block, the arrangement order of the above-mentioned matching data blocks can be obtained, so that the matching data blocks can be spliced ​​according to the arrangement order. During the splicing process, the matching data blocks need to be error-checked. During the splicing process, damaged data blocks can be determined. Damaged data blocks are data blocks with errors, that is, they are considered to be damaged data blocks. In the subsequent process, the data blocks are first tried to be repaired to try to ensure the integrity of the restored binary log file.

[0046] In step S104, the damaged data blocks are repaired, the successfully repaired damaged data blocks are spliced ​​to the corresponding positions, the damage events are trimmed from the damaged data blocks that failed to be repaired, and the damaged data blocks after the damage events are trimmed are spliced ​​to the corresponding positions to restore the binary log file.

[0047] If there is no damaged data block among the matching data blocks, the matching data blocks are spliced ​​in order to recover the binary log file.

[0048] If there are corrupted data blocks among the matching data blocks, different corrupted data blocks may have different damage modes. Different repair strategies can be used to repair corrupted data blocks with different damage modes. A corrupted data block can be repaired successfully or unsuccessfully. If the repair is successful, the repaired data block can be spliced ​​into the corresponding position of the damaged data block to ensure the integrity of the recovered binary log file. If the repair fails, it is likely that only some events were damaged. The damaged events can be pruned as damaged events. The pruned damaged data block includes undamaged events and no longer includes damaged events. The pruned damaged data block is then spliced ​​into the corresponding position of the damaged data block to ensure the integrity of the binary log file. Even if there are corrupted data blocks that fail to be repaired, the pruned damaged data block after pruned can retain most of the data in the damaged data block, making the recovered binary log file usable. In some examples, the probability of failed repair of the corrupted data block is very small, and the recovery rate of the binary log file is very high.

[0049] In some examples, after the binary log file is recovered, the recovered binary log file can be replayed and verified to further verify whether the binary log file is successfully recovered.

[0050] In an embodiment of the present application, the initial event and the end event can be searched in the data block device first, and the basic information of the log file can be obtained from the initial event that has passed the error check and the end event that has passed the error check. Then, the matching data blocks can be searched in the data block device based on the basic information of the log file, and the data blocks required to restore the binary log file include the matching data blocks. According to the block information of the data blocks, the matching data blocks are spliced ​​and error-checked to determine the damaged data blocks in the matching data blocks, and the damaged data blocks are repaired. If the repair is successful, the successfully repaired damaged data blocks are spliced ​​to the corresponding position; if the repair fails, the damaged event of the damaged data blocks is trimmed, and the trimmed damaged data blocks are spliced ​​to the corresponding position. The data blocks required to restore the binary log file matched from the data block device can be spliced ​​in sequence to achieve the recovery of the binary log file. In the embodiment of the present application, the recovery process of the binary log file does not rely on the file system information. Even if the file system information is lost or some data blocks are overwritten, resulting in the destruction of the data block content, the binary log file can still be recovered.

[0051] The binary log file recovery method provided in the embodiments of the present application is highly effective for recovering log files whose stored data blocks are discontinuously distributed. Testing has shown that, for binary log files damaged by file system misoperation, the data recovery rate using the ext4magic tool, which relies on file system information, is 60%. Using the binary log file recovery method provided in the embodiments of the present application, this data recovery rate can be increased to 100%. For the same relational database, UPSQL, the binary log file recovery method provided in the embodiments of the present application can also increase the data recovery rate of UPSQL's damaged binary log files from 93.1% using other recovery methods to 100%.

[0052] In some embodiments, the damaged data blocks may include a first type of damaged data blocks and a second type of damaged data blocks. The first type of damaged data blocks includes data blocks damaged due to missing data blocks between two adjacent data blocks in the matching data blocks. The second type of damaged data blocks includes data blocks in the matching data blocks that fail error detection. Figure 3 A flowchart of a method for recovering binary log files provided in another embodiment of the present application is provided. Figure 3 and Figure 1 The difference is that Figure 1 Step S103 in the above example can be specifically broken down into Figure 3 Steps S1031 to S103

[0053] In step S1031 , a data block matching the basic information of the log file is searched in the data block device.

[0054] In step S1032 , the matching data blocks are sorted according to the server identifier, the first event timestamp, and the first event file location of the matching data blocks.

[0055] Data blocks with the same server ID are grouped together and sorted by their first event timestamps from earliest to latest and their first event file locations from smallest to largest. Data blocks are retrieved sequentially according to this sorting. Each retrieved data block serves as the current data block, and the previous data block is checked for corruption.

[0056] In step S1033 , based on the tail event file position of the current data block, the order of the current data block in the sorting, and the data block size, it is determined whether the current data block is a first-category damaged data block.

[0057] Based on the tail event file position of the current data block, its order in the sort, and its size, it can be determined whether there are missing data blocks between the current data block and the next data block in the sort. If missing data blocks are determined, this indicates that some of the data blocks required to recover the binary log file have not been matched. The current data block cannot be directly spliced ​​with the next data block in the sort and must be repaired or trimmed before it can be spliced ​​with the next data block.

[0058] In some examples, the expected file position of the current data block can be determined based on the order of the current data block in the sort and the data block size; the difference between the tail event file position of the current data block and the expected file position of the current data block is calculated; and based on the size relationship between the difference and the preset standard file position difference, it is determined whether the current data block is a first-class damaged data block.

[0059] The expected file position is the file position of the tail event if there is no missing data block after the current data block. The product of the order of the current data block in the sort and the data block size can be determined as the expected file position of the current data block. The standard file position difference is the file position difference used to determine whether there is no missing data block after the current data block. The standard file position difference can be twice the size of the data block. The size relationship between the difference between the tail event file position of the current data block and the expected file position of the current data block and the standard file position difference can indicate whether there is a missing data block between the current data block and the next data block in the sort. If the difference is greater than or equal to the standard file position difference, it means that there is a missing data block between the current data block and the next data block in the sort; if the difference is less than the standard file position difference, it means that there is no missing data block between the current data block and the next data block in the sort. In the case that there is a missing data block between the current data block and the next data block in the sort, it is necessary to first try to find the missing data block in the data block device. Specifically, when the difference is greater than or equal to the standard file position difference, the number of missing data blocks can be determined based on the difference and the data block size; the candidate data blocks of the missing number can be found based on the block number of the current data block in the data block device; the candidate data blocks are spliced ​​with the current data block and error checking is performed; if the error checking fails, the current data block is determined to be a first-class damaged data block. The quotient of the above difference and the data block size can be used as the number of missing data blocks. For example, the missing number can be calculated according to the following formula (1):

[0060] N=(log_pos - current_block_count * BS) / BS (1)

[0061] Where N is the number of missing records; log_pos is the tail event file position of the current data block; current_block_count is the order of the current data block in the sort; and BS is the data block size.

[0062] The block number of the current data block in the data block device can be determined from the block information of the current data block, and the number of missing data blocks can be retrieved near the block number of the current data block. For example, starting from the block number of the current data block, the number of missing data blocks can be retrieved forward as candidate data blocks; or starting from the block number of the current data block, the number of missing data blocks can be retrieved backward as candidate data blocks; or, the block number of the next data block in the order can be obtained from the data block device, and starting from the block number of the next data block, the number of missing data blocks can be retrieved forward as candidate data blocks; or, the block number of the next data block in the order can be obtained from the data block device, and starting from the block number of the next data block, the number of missing data blocks can be retrieved backward as candidate data blocks. Error checking can be used to determine whether the candidate data block is a missing data block between the current data block and the next data block in the order. If the error checking passes, the candidate data block can be determined to be a missing data block between the current data block and the next data block in the order, and the candidate data block can be spliced ​​between the current data block and the next data block in the order. If the error detection fails, it can be determined that the candidate data block is not the missing data block between the current data block and the next data block in the sort, and the missing data block between the current data block and the next data block in the sort cannot be obtained. Since the data block is missing between the current data block and the next data block in the sort, the current data block needs to be repaired as a first-class damaged data block.

[0063] In some examples, a damage information set may be set to record the block information of damaged data blocks. When it is determined that the current data block is a first-category damaged data block, a preset damage flag may be set, and the block information of the current data block may be added to the damage information set; the next data block in the sorting is obtained, and when it is detected that the damage flag is set, the block information of the next data block is added to the damage information set and associated with the block information of the current data block, and the damage flag is reset. The damage flag is used to trigger the entry of the block information of the current data block as a damaged data block into the damage information set. Specifically, when the current data block is a first-category damaged data block, the block number, tail event file position, and tail event length of the current data block may be recorded in the damage information set. When obtaining the next data block in the sorting, if it is detected that the damage flag is set, the first event file position and first event length of the next data block may be recorded in the damage information set and associated with the block information of the current data block.

[0064] In step S1034, an error check is performed on the current data block. If the error check fails, the current data block is determined to be a second type damaged data block.

[0065] If the error check of the current data block fails, the current data block has a damage event, and the current data block can be determined as a second type of damaged data block. Block information of the second type of damaged data block can be recorded in the damage information set.

[0066] The above method can be used to identify damaged data blocks in the matching data blocks, so that the damaged data blocks can be repaired in the subsequent process and the data in the binary log file can be restored as much as possible.

[0067] In some embodiments, the data blocks required to recover each binary log file can be determined based on the first data block containing the initial event, the last data block containing the last event, and the data blocks ordered before the first and last data blocks. However, it is possible that the matching data blocks include a damaged data block of the initial event, or that the damaged first data block of the initial event is not matched. Therefore, some of the matching data blocks may not have a corresponding first data block. These data blocks need to be processed to recover the binary log file using these data blocks as much as possible. Figure 4 A flowchart of a method for recovering binary log files provided in another embodiment of the present application is provided. Figure 4 and Figure 3 The difference is that Figure 4 The binary log file recovery method shown may further include step S1035.

[0068] In step S1035, if the data block following the data block including the last event in the sorting does not include the initial event, the data block with the smallest first event file position following the data block including the last event is determined as the third type of damaged data block.

[0069] The damaged data blocks include the third type of damaged data blocks, which include data blocks that lack corresponding first data blocks and need to be repaired. The information of the third type of damaged data blocks can be recorded in the damage information set. If the first data block is not missing or the first data block is not damaged in the matching data blocks, the matching data blocks can be divided into one or more data block groups required for binary log file recovery based on the first data module including the initial event that passed the error check and the tail data block including the last event that passed the error check; after sorting, the data block following the data block including the last event should be the data block including the initial event, or there is no data block following the data block including the last event, that is, the data block including the last event is the last data block in the sort. If the first data block is missing from the matching data blocks or is damaged, the data block including the last event has the next data block in the sort order, but the next data block does not include the initial event. In order to recover as much data as possible from the binary log file, the data block with the smallest position in the first event file behind the data block including the last event can be repaired to obtain the corresponding first data block. In this way, the binary log file can be restored using the data blocks behind the data block including the last event, thereby recovering the data in the binary log file to the greatest extent possible.

[0070] In some embodiments, matching data blocks may be retrieved sequentially in sorted order. If the retrieved matching data blocks represent the end of a binary log file, damaged data blocks corresponding to the ended binary log file are repaired. If an end event is present in the retrieved matching data blocks or the sum of the sizes of the retrieved matching data blocks and the previous data blocks reaches the file size limit, it may be determined that the retrieved matching data blocks represent the end of a binary log file. The end of a binary log file indicates that the data blocks required to recover a binary log file have been obtained, so damaged data blocks within the data blocks required to recover the binary log file are repaired.

[0071] For ease of understanding, an example is used here to illustrate the process of data block splicing, error checking, and repair. Figure 5 This is a flow chart of an example of the process of splicing, error checking and repairing data blocks provided in an embodiment of the present application, such as Figure 5 As shown, the process of splicing, error checking and repairing the data block may include steps b1 to b10.

[0072] In step b1, the matching data blocks are sorted according to the server identifier, the first event timestamp, and the first event file location.

[0073] In step b2, determine whether the first data block can be obtained from the information set table in the sorted order; if so, jump to step b3; if not, jump to step b7.

[0074] In step b3, the block information of the next data block is obtained from the information set table in the sorted order.

[0075] In step b4, the next data block is concatenated with the previous data block, and an error check is performed to determine whether the error check passes; if so, jump to step b3; if not, jump to step b5.

[0076] In step b5, the data blocks that fail the error detection check are determined as damaged data blocks, and the damaged data blocks are recorded.

[0077] In step b6, it is determined whether the data block represents the end of the binary log file; if so, jump to step b2; if not, jump to step b3.

[0078] In step b7, determine whether there are any unrecovered data blocks in the information collection table; if so, jump to step b8; if not, jump to step b9.

[0079] In step b8, the data block with the smallest first event file position among the unrecovered data blocks is obtained as the damaged data block, the loss of the first data block is recorded, and the process goes to step b3.

[0080] In step b9, determine whether there is a record of damaged data blocks; if so, jump to step b10; if not, end.

[0081] In step b10, the damaged data block is repaired.

[0082] Through the above steps b1 to b10, the matching data blocks can be spliced, error-checked, and repaired, thereby restoring the data in the binary log file to the greatest extent possible, so that the restored binary log file is available and can be opened and viewed normally.

[0083] Different damaged data blocks may be damaged in different ways and may require different repair methods. Based on the event type of the event in the damaged data block, the damaged data block can be repaired using a repair method corresponding to the event type. Specifically, the event type of the event in the damaged data block is obtained; the event in the damaged data block is repaired using a repair strategy corresponding to the event type. The event file location of the last event in the damaged data block and the event file location of the first event in the next data block after the damaged data block in the damage information set can be used to determine the length of the damaged event that needs to be repaired, facilitating subsequent repair of the damaged event. A correspondence between event type and repair strategy can be established in advance. After determining the event type in the damaged data block, the repair strategy corresponding to the event type can be searched within this correspondence.

[0084] In some examples, a fixed template repair method can be used to repair damaged events in a damaged data block. In the case where the event type is a fixed format event type, the repair strategy includes: obtaining the timestamp of the last event in the previous binary log file of the binary log file to which the damaged data block belongs in chronological order, and repairing the damaged data block based on the timestamp of the last event in the previous binary log file and the template corresponding to the event type. For events of the fixed format event type, even if the data therein is completely destroyed, the timestamp of the start time of the binary log file to which the damaged data block belongs can be determined based on the timestamp of the last time in the previous binary log file. Except for the timestamp, the other elements of the fixed format event type are relatively fixed. The fixed format event type can be repaired using a pre-set template corresponding to the event type, thereby completing the repair of the fixed format event type event in the damaged data block. For example, the initial event FORMAT_DESCRIPTION_EVENT and the last event ROTATE_EVENT are both fixed format event types and can be repaired using the fixed template repair method.

[0085] In some examples, the damaged event can be repaired using contextual events associated with the damaged data block's damage event. If the event type is a query event, the repair strategy includes repairing the damaged data block based on events with event timestamps before and / or after the timestamp of the damaged event. Events belonging to the query event type (query_event) include, but are not limited to, write events (WRITE events), update events (UPDATE events), and delete events (DELETE events). If a query event type event is corrupted, the corrupted event (corruption event) can be repaired using events in the preceding binary log file that share the same primary key as the corrupted event. For example, if the damaged event is an UPDATE_ROWS_EVENT or DELETE_ROW_EVENT, data recovery can be performed based on the order of the event timestamps. If the damaged event is a WRITE_ROWS_EVENT, the damaged WRITE_ROWS_EVENT can be recovered using DELETE and UPDATE events with event timestamps that precede the timestamp of the damaged event and share the same primary key (KEY). In some cases, in order to repair data errors, events with error detection codes such as CRC32 codes can be automatically repaired; for events that lack error detection codes such as CRC32 codes, information such as the block number of the data block where the event is located and the event file location of the event can be recorded in the repair confirmation log, and then merged after manual confirmation by staff to improve the accuracy of data repair.

[0086] A second aspect of the present application provides a binary log file recovery device. Figure 6 A schematic diagram of the structure of a binary log file recovery device provided in an embodiment of the present application is shown as follows: Figure 6 As shown, the binary log file recovery device 200 may include a search module 201 , an acquisition module 202 , a processing module 203 and a repair module 204 .

[0087] The search module 201 may be configured to search for an initial event and an end event in a data block device.

[0088] The acquisition module 202 may be configured to acquire basic log file information from the initial event and the last event when the initial event passes the error detection check and the last event passes the error detection check.

[0089] The basic information of the log file includes the server ID, file start time, and file end time.

[0090] The processing module 203 may be used to search for data blocks matching the basic information of the log file in the data block device, perform splicing and error checking on the matching data blocks according to the block information of the matching data blocks, and determine damaged data blocks.

[0091] The chunk information includes the server ID, the first event timestamp, the first event file location, and the last event file location.

[0092] The repair module 204 can be used to repair the damaged data blocks, splice the successfully repaired damaged data blocks to the corresponding positions, trim the damage events in the damaged data blocks that failed to be repaired, and splice the damaged data blocks after the damage events are trimmed to the corresponding positions to recover the binary log file.

[0093] In an embodiment of the present application, the initial event and the end event can be searched in the data block device first, and the basic information of the log file can be obtained from the initial event that has passed the error check and the end event that has passed the error check. Then, the matching data blocks can be searched in the data block device based on the basic information of the log file, and the data blocks required to restore the binary log file include the matching data blocks. According to the block information of the data blocks, the matching data blocks are spliced ​​and error-checked to determine the damaged data blocks in the matching data blocks, and the damaged data blocks are repaired. If the repair is successful, the successfully repaired damaged data blocks are spliced ​​to the corresponding position; if the repair fails, the damaged event of the damaged data blocks is trimmed, and the trimmed damaged data blocks are spliced ​​to the corresponding position. The data blocks required to restore the binary log file matched from the data block device can be spliced ​​in sequence to achieve the recovery of the binary log file. In the embodiment of the present application, the recovery process of the binary log file does not rely on the file system information. Even if the file system information is lost or some data blocks are overwritten, resulting in the destruction of the data block content, the binary log file can still be recovered.

[0094] In some embodiments, the search module 201 can be specifically used to: use a regular expression generated based on the initial event filtering information to match the initial event in the data block device, and the initial event filtering information includes the expected recovery time period and the event type of the initial event; use a regular expression generated based on the last event filtering information to match the last event in the data block device, and the last event filtering information includes the expected recovery time period and the event type of the last event.

[0095] In some embodiments, the damaged data blocks include first-category damaged data blocks and second-category damaged data blocks. The processing module 203 may be specifically configured to: sort the matched data blocks according to the server identifier, first-event timestamp, and first-event file location of the matched data blocks; determine whether the current data block is a first-category damaged data block based on the last-event file location of the current data block, the order of the current data block in the sorting, and the data block size; and perform an error check on the current data block. If the error check fails, determine that the current data block is a second-category damaged data block.

[0096] In some embodiments, the processing module 203 can be specifically used to: determine the expected file position of the current data block based on the order of the current data block in the sort and the data block size; calculate the difference between the tail event file position of the current data block and the expected file position of the current data block; and determine whether the current data block is a first-class damaged data block based on the size relationship between the difference and the preset standard file position difference.

[0097] In some examples, the processing module 203 can be specifically used to: determine the number of missing data blocks based on the difference and the data block size when the difference is greater than or equal to the standard file position difference; search for the candidate data blocks with the missing number based on the block number of the current data block in the data block device; splice the candidate data blocks with the current data block and perform error detection; if the error detection fails, determine that the current data block is a first-class damaged data block.

[0098] In some embodiments, the processing module 203 may be further configured to: if the error detection check passes, splice the candidate data block between the current data block and the next data block in the sorting.

[0099] In some embodiments, the processing module 203 can also be used to: when it is determined that the current data block is a first-type damaged data block, set a preset damage flag and add the block information of the current data block to the damage information set; obtain the next data block in the sorting, and when it is detected that the damage flag is set, add the block information of the next data block to the damage information set and associate it with the block information of the current data block, and reset the damage flag.

[0100] In some embodiments, the damaged data blocks further include a third type of damaged data blocks. The processing module 203 may further be specifically configured to: if the data blocks following the data block including the last event in the sorting do not include the initial event, determine the data block with the smallest file position of the first event following the data block including the last event as the third type of damaged data block.

[0101] In some embodiments, the repair module 204 may be specifically configured to: when the matching data block represents the end of a binary log file, repair the damaged data block corresponding to the ended binary log file.

[0102] In some embodiments, the repair module 204 may be specifically configured to: obtain an event type of an event in a damaged data block; and repair the event in the damaged data block using a repair strategy corresponding to the event type.

[0103] In some examples, when the event type is a fixed-format event type, the repair strategy includes: obtaining the timestamp of the last event in the previous binary log file of the binary log file to which the damaged data block belongs in chronological order, and repairing the damaged data block based on the timestamp of the last event in the previous binary log file and a template corresponding to the event type. When the event type is a query event type, the repair strategy includes: repairing the damaged data block based on events whose event timestamps are before the timestamp of the damaged event of the damaged data block and / or events whose event timestamps are after the timestamp of the damaged event of the damaged data block.

[0104] The third aspect of the present application also provides an electronic device. Figure 7 This is a schematic diagram of the structure of a model publishing device in a distributed system provided by an embodiment of the present application. Figure 7 As shown, the electronic device 300 includes a memory 301 , a processor 302 , and a computer program stored in the memory 301 and executable on the processor 302 .

[0105] In some examples, the processor 302 may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0106] The memory 301 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Therefore, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method for recovering a binary log file according to an embodiment of the present application.

[0107] The processor 302 reads the executable program code stored in the memory 301 to run a computer program corresponding to the executable program code, so as to implement the binary log file recovery method in the above embodiment.

[0108] In some examples, the electronic device 300 may further include a communication interface 303 and a bus 304. Figure 7 As shown, the memory 301 , the processor 302 , and the communication interface 303 are connected via a bus 304 and communicate with each other.

[0109] The communication interface 303 is mainly used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. Input devices and / or output devices can also be connected through the communication interface 303.

[0110] Bus 304 includes hardware, software, or both that couples components of electronic device 300 to each other. By way of example, and not limitation, bus 304 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of the above. Bus 304 may include one or more buses, where appropriate. Although embodiments herein describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

[0111] A fourth aspect of the present application further provides a computer-readable storage medium having computer program instructions stored thereon. When executed by a processor, the computer program instructions can implement the binary log file recovery method described in the above embodiment and achieve the same technical effect. To avoid repetition, the above-mentioned computer-readable storage medium may include a non-transitory computer-readable storage medium, such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., and is not limited here.

[0112] An embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the binary log file recovery method in the above embodiment and can achieve the same technical effect. To avoid repetition, it is not described here.

[0113] It should be understood that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For device embodiments, equipment embodiments, computer-readable storage medium embodiments, and computer program product embodiments, the relevant parts can be referred to the description section of the method embodiment. This application is not limited to the specific steps and structures described above and shown in the figures. Those skilled in the art can make various changes, modifications and additions, or change the order of the steps after understanding the spirit of this application. In addition, for the sake of brevity, a detailed description of known method technologies is omitted here.

[0114] Aspects of the present application have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed via the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. This processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or the flowchart and the combination of the boxes in the block diagram and / or the flowchart can also be implemented by the dedicated hardware that performs the specified function or action, or can be implemented by the combination of dedicated hardware and computer instructions.

[0115] Those skilled in the art should understand that the above embodiments are illustrative rather than restrictive. Different technical features appearing in different embodiments can be combined to achieve beneficial effects. Based on a study of the drawings, the specification and the claims, those skilled in the art should be able to understand and implement other variations of the disclosed embodiments. In the claims, the term "comprising" does not exclude other devices or steps; the quantifier "one" does not exclude a plurality; the terms "first" and "second" are used to identify names rather than to indicate any specific order. Any figure marks in the claims should not be understood as limiting the scope of protection. The functions of multiple parts appearing in the claims can be implemented by a separate hardware or software module. The fact that certain technical features appear in different dependent claims does not mean that these technical features cannot be combined to achieve beneficial effects.

Claims

1. A method for recovering a binary log file, characterized in that: include: Find the initial and last events in the block device; When the initial event passes the error detection check and the final event passes the error detection check, obtaining basic information of the log file from the initial event and the final event, the basic information of the log file including the server identifier, the file start time and the file end time; Searching the data block device for data blocks that match the basic information of the log file, performing splicing and error checking on the matching data blocks based on block information of the matching data blocks to determine damaged data blocks, where the block information includes a server identifier, a first event timestamp, a first event file location, and a last event file location; Repair the damaged data blocks, splice the successfully repaired damaged data blocks to the corresponding positions, trim the damage events in the damaged data blocks that failed to be repaired, and splice the damaged data blocks after the damage events are trimmed to the corresponding positions to recover the binary log file.

2. The method according to claim 1, characterized in that The step of searching for the initial event and the last event in the data block device includes: Matching an initial event in the data block device using a regular expression generated based on initial event screening information, wherein the initial event screening information includes an expected recovery time period and an event type of the initial event; The end event is matched in the data block device using a regular expression generated based on the end event screening information, wherein the end event screening information includes an expected recovery time period and an event type of the end event.

3. The method according to claim 1, characterized in that The damaged data blocks include first-category damaged data blocks and second-category damaged data blocks. The step of performing splicing and error checking on the matched data blocks according to the block information of the matched data blocks to determine the damaged data blocks includes: Sort the matching data blocks according to their server identifiers, first event timestamps, and first event file locations; Determining whether the current data block is the first type of damaged data block based on the tail event file position of the current data block, the order of the current data block in the sorting, and the size of the data block; An error check is performed on the current data block. If the error check fails, it is determined that the current data block is the second type of damaged data block.

4. The method according to claim 3, characterized in that The determining whether the current data block is the first type of damaged data block based on the tail event file position of the current data block, the order of the current data block in the sorting, and the data block size includes: Determine the expected file position of the current data block based on the order of the current data block in the sorting and the size of the data block; Calculate the difference between the tail event file position of the current data block and the expected file position of the current data block; Based on the size relationship between the difference value and a preset standard file position difference, it is determined whether the current data block is the first type of damaged data block.

5. The method according to claim 4, characterized in that The determining whether the current data block is the first type of damaged data block based on the size relationship between the difference value and a preset standard file position difference includes: When the difference is greater than or equal to the standard file position difference, determining the number of missing data blocks according to the difference and the size of the data blocks; Searching for the missing number of candidate data blocks according to the block number of the current data block in the data block device; Splicing the candidate data block with the current data block and performing error checking; If the error detection fails, it is determined that the current data block is the first type of damaged data block.

6. The method according to claim 5, characterized in that Also includes: If the error detection is passed, the candidate data block is spliced ​​between the current data block and the next data block in the sort.

7. The method according to claim 5, characterized in that Also includes: When it is determined that the current data block is the first type of damaged data block, a preset damage flag is set, and block information of the current data block is added to the damage information set; The next data block in the sorting is obtained, and when it is detected that the damage flag is set, the block information of the next data block is added to the damage information set and associated with the block information of the current data block, and the damage flag is reset.

8. The method according to claim 3, characterized in that The damaged data blocks also include a third type of damaged data blocks, The step of performing splicing and error checking on the matched data blocks according to the block information of the matched data blocks to determine the damaged data blocks further includes: If the data block following the data block including the last event in the sorting does not include the initial event, the data block with the smallest first event file position following the data block including the last event is determined as the third type of damaged data block.

9. The method according to claim 1, characterized in that The repairing of the damaged data block includes: When the matching data block represents the end of a binary log file, the damaged data block corresponding to the ended binary log file is repaired.

10. The method according to claim 1, characterized in that The repairing of the damaged data block includes: Obtaining an event type of an event in the damaged data block; The events in the damaged data block are repaired using a repair strategy corresponding to the event type.

11. The method according to claim 10, characterized in that If the event type is a fixed-format event type, the repair strategy includes: obtaining, in chronological order, a timestamp of a last event in a previous binary log file to which the damaged data block belongs, and repairing the damaged data block according to the timestamp of the last event in the previous binary log file and a template corresponding to the event type; In the case where the event type belongs to a query event type, the repair strategy includes: repairing the damaged data block based on events whose event timestamps are before the timestamp of the damage event of the damaged data block and / or events whose event timestamps are after the timestamp of the damage event of the damaged data block.

12. A binary log file recovery device, characterized in that: include: A search module for searching for an initial event and an end event in a data block device; An acquisition module, configured to acquire basic log file information from the initial event and the last event when the initial event passes the error detection check and the last event passes the error detection check, wherein the basic log file information includes a server identifier, a file start time, and a file end time; a processing module configured to search the data block device for data blocks that match the basic information of the log file, perform splicing and error checking on the matching data blocks based on block information of the matching data blocks, and determine damaged data blocks, where the block information includes a server identifier, a first event timestamp, a first event file location, and a last event file location; The repair module is used to repair the damaged data blocks, splice the successfully repaired damaged data blocks to the corresponding positions, trim the damage events in the damaged data blocks that failed to be repaired, and splice the damaged data blocks after the damage events are trimmed to the corresponding positions to recover the binary log file.

13. An electronic device, characterized in that: include: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the binary log file recovery method according to any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the binary log file recovery method according to any one of claims 1 to 11.

Images