Methods, apparatus, equipment, and media for recovering binary log files.

Abstract

This invention discloses a method, apparatus, device, and medium for recovering binary log files, belonging to the field of data processing. The method includes: searching for initial and final events in a data block device; if both the initial and final events pass error detection checks, obtaining basic log file information from the initial and final events; searching for data blocks in the data block device that match the basic log file information; performing concatenation and error verification on the matching data blocks based on their block information to identify damaged data blocks; repairing the damaged data blocks; concatenating successfully repaired damaged data blocks to their corresponding positions; trimming damaged events from damaged data blocks that failed to be repaired; and concatenating the trimmed damaged data blocks to their corresponding positions to recover the binary log file. According to embodiments of this invention, the recovery of binary log files can be achieved.

Description

Technical Field

[0001] The present invention belongs to the field of data processing, and in particular relates to a method, device, equipment and medium for recovering a binary log file. Prior Art

[0002] A database is a collection of large amounts of data that is stored in hardware devices for a long time and managed in a unified manner. Binary log files, or binlogs, record changes in table structures and table data modifications in the database. They are important logs in the database and are closely related to operation and maintenance. In the process of database operation and maintenance, incorrect operations may cause the database's binary log files to be damaged, resulting in data loss, which will cause huge data losses to the business. Therefore, a method for recovering binary log files is urgently needed. Summary of the invention

[0003] The embodiment of the present invention provides a binary log file recovery method, device, equipment and medium, which can realize the recovery of binary log files.

[0004] In a first aspect, an embodiment of the present invention provides a method for recovering a binary log file, comprising: searching for an initial event and an end event in a data block device; obtaining basic information of the log file from the initial event and the end event when the initial event passes the error detection check and the end event passes the error detection check, the basic information of the log file including a server identifier, a file start time, and a file end time; searching for a data block matching the basic information of the log file in the data block device, splicing and error checking the matching data block according to the block information of the matching data block, and determining a damaged data block, the block information including a server identifier, a first event timestamp, a first event file position, and a last event file position; repairing the damaged data block, splicing the successfully repaired damaged data block to a corresponding position, trimming the damaged event from the damaged data block that failed to be repaired, and splicing the damaged data block after the trimmed damaged event to a corresponding position, and recovering the binary log file.

[0005] In a second aspect, an embodiment of the present invention provides a recovery device for a binary log file, comprising: a search module, used to search for an initial event and an end event in a data block device; an acquisition module, used to obtain basic information of the log file from the initial event and the end event when the initial event passes the error detection check and the end event passes the error detection check, the basic information of the log file includes a server identifier, a file start time and a file end time; a processing module, used to search for a data block matching the basic information of the log file in the data block device, splice and error-check the matching data block according to the block information of the matching data block, and determine the damaged data block, the block information includes a server identifier, a first event timestamp, a first event file position and a last event file position; a repair module, used to repair the damaged data block, splice the successfully repaired damaged data block to the corresponding position, trim the damaged event from the damaged data block that failed to be repaired, and splice the damaged data block after the trimmed damaged event to the corresponding position, so as to recover the binary log file.

[0006] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the binary log file recovery method of the first aspect is implemented.

[0007] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the binary log file recovery method of the first aspect is implemented.

[0008] The embodiment of the present invention provides a method, device, equipment and medium for recovering a binary log file, which can first search for an initial event and an end event in a data block device, obtain basic information of the log file from the initial event that passes the error check and the end event that passes the error check, and then search for a matching data block in the data block device according to the basic information of the log file, and the data block required for recovering the binary log file includes the matching data block. According to the block information of the data block, the matching data block is spliced and error-checked to determine the damaged data block in the matching data block, and the damaged data block is repaired. If the repair is successful, the successfully repaired damaged data block is spliced to the corresponding position; if the repair fails, the damaged event of the damaged data block is cut, and the cut damaged data block is spliced to the corresponding position. The data blocks required for recovering the binary log file obtained by matching from the data block device can be spliced in sequence to realize the recovery of the binary log file. Simple diagram description

[0009] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the following is a brief introduction to the diagrams required for use in the embodiment of the present invention. For ordinary technicians in this field, other diagrams can be obtained based on these diagrams without making any progressive efforts. Figure 1 is a flowchart of a method for recovering a binary log file according to an embodiment of the present invention; Figure 2 is a flowchart of an example of a process of recording block information of the first data block and the tail data block into the information collection table according to an embodiment of the present invention; Figure 3 is a flowchart of a method for recovering a binary log file according to another embodiment of the present invention; Figure 4 is a flowchart of a method for recovering a binary log file provided by yet another embodiment of the present invention; Figure 5 is a flowchart of an example of the process of splicing, error checking and repairing data blocks provided in an embodiment of the present invention; Figure 6 is a schematic diagram of the structure of a binary log file recovery device according to an embodiment of the present invention; FIG. 7 is a schematic diagram of the structure of a model publishing device in a distributed system provided by an embodiment of the present invention. Implementation

[0010] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are intended only to explain the present invention, rather than to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by showing examples of the present invention. It should be noted that the acquisition, storage, use, and processing of information and data in the embodiments of the present invention are authorized by users or relevant institutions and comply with the relevant provisions of national laws and regulations.

[0011] A database is a collection of a large amount of data that is uniformly managed and stored in a hardware device for a long time. Binary log files, namely binlogs, record changes in table structures and modifications to table data in the database. They are important logs in the database and are closely related to operation and maintenance. In the process of operating and maintaining the database, the binary log files of the database may be damaged due to incorrect operations, resulting in data loss, which will cause huge data losses to the business. Therefore, a method for recovering binary log files is urgently needed. At present, there are also related technologies that use file system information to repair files, but if file system information such as file node information, namely index node information, partition table, file system format information, etc. is lost, or if some data blocks are copied and cause the content of the data blocks to be destroyed, the files cannot be repaired. There are also related technologies that use file carving methods, namely file carving methods, to repair files, but file carving methods are suitable for recovering files stored in continuous storage spaces in storage devices, while the distribution of data blocks stored in log files is not continuous and will be scattered in different storage spaces according to the usage of the file system. Therefore, the file carving method is difficult to recover log files and cannot repair binary log files.

[0012] The present invention provides a binary log file recovery method, device, equipment and medium, which can be applied to the binary log file recovery scenario, and can be specifically used to recover the binary log file binlog of a MySQL database. The binary log file recovery method, device, equipment and medium can repair the characteristics of the binary log file of the MySQL database, and can restore the binary log file as much as possible in the scenario where some data blocks are duplicated, thereby improving the data recovery rate of the binary log file. The basic information of the binary log file can be obtained in the data block device through the initial event and the end event, and the data block required for recovering the binary log file can be found in the data block device according to the basic information, and the data blocks are spliced in sequence and error-checked based on the block information of the data blocks. If a damaged data block is found in the process, the damaged data block is repaired, and the repaired data block is spliced to the corresponding position, thereby obtaining a recovered binary log file. By utilizing the binary log file recovery method, even if the file system information is destroyed and the distribution of the data blocks stored in the binary log file is not continuous, the binary log file can be recovered, thereby improving the recovery rate of the binary log file.

[0013] The present invention provides a method, device, equipment and medium for recovering a binary log file, which are described below.

[0014] A first aspect of the present invention provides a binary log file recovery method, which can be applied to a binary log file recovery scenario, and can be specifically performed by a binary log file recovery device, equipment, etc. FIG. 1 is a flow chart of a binary log file recovery method provided by an embodiment of the present invention. As shown in FIG. 1 , the binary log file recovery method may include steps S101 to S104.

[0015] In step S101, the initial event and the last event are searched in the data block device.

[0016] The initial event is the first event in the binary log file, and the initial event appears only once in the binary log file. For example, the initial event can be FORMAT_DESCRIPTION_EVENT. The tail event is the last event in the binary log file. When the size of the binary log file reaches the maximum requirement or a refresh instruction is executed, the binary log file will be switched, and the tail event will be added to the current binary log file. The tail event can specify the name and location of the next binary log file. For example, the tail event can be ROTATE_EVENT. The basic information of the binary log file to be restored can be obtained through the initial event and the tail event. The number of binary log files to be restored can be one or more, which is not limited here.

[0017] In some examples, the rule operation formula generated based on the initial event filtering information can be used to match the initial event in the data block device; the rule operation formula generated based on the end event filtering information can be used to match the end event in the data block device. The initial event filtering information includes the expected recovery time period and the event type of the initial event. According to the requirements, the initial event filtering information can also include other information, such as the expected server identification, etc. The end event filtering information includes the expected recovery time period and the event type of the end event. According to the requirements, the end event filtering information can also include other information, such as the expected server identification, etc. The expected recovery time period is the time period in which the user expects to restore the binary log file, which can be set according to the scenario, requirements, etc., and is not limited here. For example, if the user expects to restore the binary log files for two months before November 2023, the expected recovery time period can be from September 1 to October 31, 2023. The event type may characterize the type of the event. Different events have different event types and may be represented by different identification values. For example, the range of the event type may include but is not limited to \x00 to \x26. The event type of the initial event may be x0f, and the event type of the final event may be x04. The expected server identifier may characterize the server corresponding to the binary log file that the user expects to restore. A rule operation formula may be generated by the expected recovery time period and the event type, and the rule operation formula may be used to match in the data block device to obtain the initial event and the final event. For example, the rule operation formula "[TIMESTAMP_RANGE]\x0f....\x" may be used to match the initial event in the time period indicated by TIMESTAMP_RANGE in the data block device. Similarly, the rule operation formula "[TIMESTAMP_RANGE]\x04....\x" may be used to match the final event in the time period indicated by TIMESTAMP_RANGE in the data block device. A rule operation formula may also be generated by the expected recovery time period, the event type, and the expected server identifier, and the rule operation formula may be used to match in the data block device to obtain the initial event and the final event. For example, a rule operation formula including expected recovery time [TIMESTAMP_RANGE]+event type [TYPE_CODE]+[server identifier] may be used to match the initial event or the last event in the time period indicated by TIMESTAMP_RANGE in the data block device that is related to the server corresponding to the server identifier.

[0018] After obtaining the initial event, it is necessary to perform an error check on the initial event to determine whether there is an error in the initial event. If there is an error in the initial event, it is possible that the initial event is damaged. Similarly, after obtaining the last event, it is necessary to perform an error check on the last event to determine whether there is an error in the last event. If there is an error in the last event, it is possible that the last event is damaged. The error check can be verified using a cyclic redundancy check code (Cyclic Redundancy Check 32, CRC32). When obtaining the initial event and the last event, a data block including the initial event and a data block including the last event can also be obtained. The data block including the initial event can be the first data block in the binary log file, and the data block including the last event can be the last data block in the binary log file. An information collection table can be set. When the initial event passes the error check, the block information of the data block including the initial event obtained is recorded in the information collection table as the block information of the first data block; when the last event passes the error check, the block information of the data block including the last event obtained is recorded in the information collection table as the block information of the last data block.

[0019] For example, Figure 2 is a flowchart of an example of a process of recording the block information of the first data block and the block information of the last data block into an information collection table provided by an embodiment of the present invention. As shown in Figure 2, the process of recording the block information of the first data block and the block information of the last data block into the information collection table may include steps a1 to a9.

[0020] In step a1, the data block size (Block Size, BS) is obtained. Specifically, the data block size can be read from the file system information of the data block device, or the basic unit of data recovery, namely the data block size, can be determined according to the user's input.

[0021] In step a2, the data block containing the initial event is searched in the data block device.

[0022] In step a3, the initial event is checked for errors; if the error check passes, jump to step a4; if the error check fails, jump to step a2.

[0023] In step a4, the block information of the data block containing the initial event is added to the information collection table as the first data block information.

[0024] In step a5, determine whether the data block containing the initial event in the data block device has been traversed; if the traversal is completed, jump to step a6; if the traversal is not completed, jump to step a2.

[0025] In step a6, the data block device is searched for a data block containing the end event.

[0026] In step a7, the end event is checked for error; if the error check passes, jump to step a8; if the error check fails, jump to step a6.

[0027] In step a8, the block information of the data block containing the end event is added to the information collection table as the tail data block information.

[0028] In step a9, determine whether the data block containing the end event in the data block device has been traversed; if the traversal is completed, end; if the traversal is not completed, jump to step a6.

[0029] Through the above steps a1 to a9, the block information of the first data block and the block information of the last data block that can be found in the data block device can be recorded in the information collection table, which is convenient for subsequent searching of the block information of the first data block and the block information of the last data block.

[0030] In step S102, when the initial event passes the error detection check and the final event passes the error detection check, basic log file information is obtained from the initial event and the final event.

[0031] The initial event is checked by error detection to indicate that the initial event is not damaged, and the final event is checked by error detection to indicate that the final event is not damaged. The basic information of the log file can be obtained from the initial event and the final event that are not damaged. The events in the two-level log file may include elements such as timestamp, event type, type_code, server ID, event length, event_len, position of the event in the file, log_position, flag, error detection code, etc., and the lengths of the above elements are all pre-set, so the offset of each element can be obtained. For example, the elements contained in the event can be shown in Table 1:

[0032] When the user sets the expected recovery time, the event can be matched according to whether the timestamp is within the expected recovery time period, and the data block within the expected recovery time period can be obtained. The error check code is used for error checking. If the error check code is a CRC32 code, it can be used for CRC32 checking.

[0033] The basic information of the log file includes the server identifier, the file start time and the file end time. The server identifier may indicate the server. The file start time may indicate the start time of the binary log file where the initial event is located. The file end time may indicate the end time of the binary log file where the final event is located. The basic information of the log file may also include, but is not limited to, information such as server version information, binary log file name, binary log file size, etc. Among them, the basic information of the log file such as the server identifier, file start time, server version information, etc. may be obtained from the initial event, and the basic information of the log file such as the binary log file name, binary log file size, file end time, etc. may be obtained from the final event.

[0034] In step S103, a data block matching the basic information of the log file is searched in the data block device, and the matching data blocks are spliced and error-checked according to the block information of the matching data blocks to determine the damaged data blocks.

[0035] The data block device stores multiple data blocks, and the data blocks that match the basic information of the log file include the data blocks required for restoring the binary log file this time. The data blocks whose server identifiers in the data block device are consistent with the server identifiers in the basic information of the log file and whose timestamps are located in the time period formed by the file start time and the file end time can be determined as matching data blocks. For example, the time period formed by the file start time and the file end time, the supported event types, and the rule operation formula "[TIMESTAMP][\x00-\x26](server ID)" generated by the server identifier can be used to match in the data block device and filter out matching data blocks. The block information of the matching data blocks can be recorded in the information collection table.

[0036] The block information may include a server identifier, a first event timestamp, a first event file position, and a last event file position. The first event timestamp of a data block is the timestamp of the first event in the data block. The first event file position of a data block is the position of the first event in the data block in the binary log file. The last event file position of a data block is the position of the last event in the data block in the binary log file. According to the first event timestamp, the first event file position, and the last event file position of the data block, the arrangement order of the above-mentioned matching data blocks can be obtained, so that the matching data blocks can be spliced according to the arrangement order. In the splicing process, the matching data blocks need to be error-checked. In the splicing process, damaged data blocks can be determined. Damaged data blocks are data blocks with errors, that is, data blocks that are considered damaged. In the subsequent process, the data blocks are first tried to be repaired to try to ensure the integrity of the restored binary log file.

[0037] In step S104, the damaged data blocks are repaired, the successfully repaired damaged data blocks are spliced to the corresponding positions, the damage events are trimmed from the damaged data blocks that failed to be repaired, and the damaged data blocks after the damage events are trimmed are spliced to the corresponding positions to restore the binary log file.

[0038] If there is no damaged data block in the matched data blocks, the matched data blocks are spliced in order to recover the binary log file.

[0039] If there are damaged data blocks in the matched data blocks, different damaged data blocks may be damaged in different ways, and different repair strategies can be used to repair damaged data blocks with different damage modes. There are two possibilities for damaged data blocks: successful repair and failed repair. If the damaged data block is successfully repaired, the successfully repaired damaged data block can be spliced to the corresponding position of the damaged data block to ensure the integrity of the recovered binary log file; if the damaged data block fails to be repaired, it is possible that only part of the events are damaged, and the damaged events can be trimmed as damaged events. The damaged data block after trimming includes undamaged events and no longer includes damaged events. The damaged data block after trimming is spliced to the corresponding position of the damaged data block to try to ensure the integrity of the binary log file. Even if there are damaged data blocks that fail to be repaired, the damaged data blocks after splicing and trimming the damaged events can retain most of the data in the damaged data blocks, and the recovered binary log file is usable. In some examples, the possibility of failure to repair the damaged data block is very small, and the recovery rate of the binary log file is very high.

[0040] In some examples, after the binary log file is recovered, a replay check may be performed on the recovered binary log file to further verify whether the binary log file is successfully recovered.

[0041] In an embodiment of the present invention, the initial event and the end event can be searched in the data block device first, and the basic information of the log file can be obtained from the initial event that has passed the error check and the end event that has passed the error check, so as to search the matching data block in the data block device according to the basic information of the log file, and the data block required for recovering the binary log file includes the matching data block. According to the block information of the data block, the matching data block is spliced and error-checked to determine the damaged data block in the matching data block, and the damaged data block is repaired. If the repair is successful, the successfully repaired damaged data block is spliced to the corresponding position; if the repair fails, the damaged event of the damaged data block is cut, and the cut damaged data block is spliced to the corresponding position. The data blocks required for recovering the binary log file obtained from the data block device can be spliced in sequence to realize the recovery of the binary log file. In the embodiment of the present invention, the recovery process of the binary log file does not rely on the file system information. Even if the file system information is lost and some data blocks are copied to cause the data block content to be destroyed, the binary log file can still be recovered.

[0042] The binary log file recovery method provided by the embodiment of the present invention has a good recovery effect on log files whose data blocks are not distributed continuously. After testing, for the situation where the binary log file is damaged due to the misoperation of the file system, the data recovery rate of the binary log file is 60% when the ext4magic tool relying on the file system information is used to recover the binary log file. The data recovery rate of the binary log file can be increased to 100% when the binary log file recovery method provided by the embodiment of the present invention is used to recover the binary log file. For the same relational database UPSQL, the data recovery rate of the damaged binary log file of UPSQL can also be increased from 93.1% when recovered by other methods to 100% by using the binary log file recovery method provided by the embodiment of the present invention.

[0043] In some embodiments, the damaged data blocks may include a first type of damaged data blocks and a second type of damaged data blocks. The first type of damaged data blocks include data blocks damaged due to missing data blocks between two data blocks that are adjacent in order in the matching data blocks. The second type of damaged data blocks include data blocks that fail to pass the error check in the matching data blocks. FIG3 is a flow chart of a binary log file recovery method provided by another embodiment of the present invention. The difference between FIG3 and FIG1 is that step S103 in FIG1 can be specifically refined into steps S1031 to S1034 in FIG3.

[0044] In step S1031, a data block matching the basic information of the log file is searched in the data block device.

[0045] In step S1032, the matching data blocks are sorted according to the server identifier, the first event timestamp and the first event file position of the matching data blocks.

[0046] Data blocks with the same server ID can be grouped together, and the matching data blocks can be sorted in the order of the first event timestamp from early to late and the first event file position from small to large. Data blocks are obtained in sequence according to the order, and each obtained data block can be used as the current data block to detect and determine whether the data block before the point is a damaged data block.

[0047] In step S1033, based on the tail event file position of the current data block, the order of the current data block in the sorting, and the data block size, it is determined whether the current data block is a first-category damaged data block.

[0048] According to the tail event file position of the current data block, the order of the current data block in the sorting, and the data block size, it can be determined whether there is a lack of data blocks between the current data block and the next data block in the sorting. If it is determined that there is a lack of data blocks between the current data block and the next data block in the sorting, it means that some data blocks required to restore the binary log file have not been matched. The current data block cannot be directly spliced with the next data block in the sorting, and the current data block needs to be repaired or trimmed before it can be spliced with the next data block.

[0049] In some examples, the expected file position of the current data block can be determined based on the order of the current data block in the sort and the data block size; the difference between the tail event file position of the current data block and the expected file position of the current data block is calculated; based on the size relationship between the difference and the preset standard file position difference, it is determined whether the current data block is a first-category damaged data block.

[0050] The expected file position is the file position of the tail event if there is no missing data block behind the current data block. The expected file position of the current data block can be determined by multiplying the order of the current data block in the sorting by the data block size. The standard file position difference is the file position difference used to determine that there is no missing data block behind the current data block. The standard file position difference can be twice the size of the data block. The difference between the tail event file position of the current data block and the expected file position of the current data block and the size relationship of the standard file position difference can indicate whether there is a missing data block between the current data block and the next data block in the sorting. If the difference is greater than or equal to the standard file position difference, it indicates that there is a missing data block between the current data block and the next data block in the sorting; if the difference is less than the standard file position difference, it indicates that there is no missing data block between the current data block and the next data block in the sorting. In the case that there is a missing data block between the current data block and the next data block in the sorting, it is necessary to first try to find the missing data block in the data block device. Specifically, when the difference is greater than or equal to the standard file position difference, the number of missing data blocks can be determined based on the difference and the data block size; the candidate data blocks with the missing number can be found based on the block number of the current data block in the data block device; the candidate data blocks are spliced with the current data block and error checking is performed; if the error checking fails, the current data block is determined to be a first-class damaged data block. The quotient of the above difference and the data block size can be used as the number of missing data blocks. For example, the missing number can be calculated according to the following formula (1): N=(log_pos-current_block_count * BS) / BS (1) Among them, N is the number of missing data; log_pos is the tail event file position of the current data block; current_block_count is the order of the current data block in the sort; BS is the data block size.

[0051] The block number of the current data block in the data block device can be determined from the block information of the current data block, and the missing number of data blocks can be obtained near the block number of the current data block. For example, starting from the block number of the current data block, the missing number of data blocks are taken forward as candidate data blocks; or, starting from the block number of the current data block, the missing number of data blocks are taken backward as candidate data blocks; or, the block number of the next data block in the sorting in the data block device is obtained, and the missing number of data blocks are taken forward from the block number of the next data block as candidate data blocks; or, the block number of the next data block in the sorting in the data block device is obtained, and the missing number of data blocks are taken backward from the block number of the next data block as candidate data blocks. Error checking can be used to determine whether the candidate data block is a missing data block between the current data block and the next data block in the sorting. If the error checking passes, it can be determined that the candidate data block is a missing data block between the current data block and the next data block in the sorting, and the candidate data block is spliced between the current data block and the next data block in the sorting. If the error detection fails, it can be determined that the candidate data block is not the missing data block between the current data block and the next data block in the sort, and the missing data block between the current data block and the next data block in the sort cannot be obtained. Since the data block is missing between the current data block and the next data block in the sort, the current data block needs to be repaired as a first-class damaged data block.

[0052] In some examples, a damage information set may be set to record the block information of the damaged data block. When it is determined that the current data block is a first-class damaged data block, a preset damage flag is set, and the block information of the current data block is added to the damage information set; the next data block in the sorting is obtained, and when it is detected that the damage flag is set, the block information of the next data block is added to the damage information set and associated with the block information of the current data block, and the damage flag is reset. The damage flag is used to trigger the block information of the current data block as a damaged data block to enter the damage information set. Specifically, when the current data block is a first-class damaged data block, the block number, tail event file position, and tail event length of the current data block can be recorded in the damage information set. When the next data block in the sorting is obtained, if it is detected that the damage flag is set, the first event file position and first event length of the next data block can be recorded in the damage information set and associated with the block information of the current data block.

[0053] In step S1034, an error check is performed on the current data block. If the error check fails, the current data block is determined to be a second type damaged data block.

[0054] If the error check of the current data block fails, the current data block has a damage event, and the current data block can be determined as a second type of damaged data block. The block information of the second type of damaged data block can be recorded in the damage information set.

[0055] The damaged data blocks in the matching data blocks can be identified by the above method, so as to repair the damaged data blocks in the subsequent process and restore the data in the binary log file as much as possible.

[0056] In some embodiments, the data blocks required for restoring each binary log file can be determined based on the first data block containing the initial event, the last data block containing the last event, and the data blocks sorted before the first data block and the last data block. However, there may be a situation where the matched data block includes the data block damaged by the initial event, or there may be a situation where the first data block damaged by the initial event is not matched. There may be some data blocks in the matched data blocks that do not have corresponding first data blocks. It is necessary to process these data blocks and use these data blocks as much as possible to restore the binary log file. Figure 4 is a flowchart of a binary log file recovery method provided by another embodiment of the present invention. The difference between Figure 4 and Figure 3 is that the binary log file recovery method shown in Figure 4 can also include step S1035.

[0057] In step S1035, if the data block behind the data block including the last event in the sorting does not include the initial event, the data block with the smallest first event file position behind the data block including the last event is determined as the third type of damaged data block.

[0058] The damaged data blocks include the third type of damaged data blocks, and the third type of damaged data blocks include data blocks that need to be repaired because they lack the corresponding first data blocks. Information of the third type of damaged data blocks can be recorded in the damaged information set. If the first data block is not missing in the matched data blocks or the first data block is not damaged, the matched data blocks can be divided into one or more data block groups required for binary log file repair according to the first data module including the initial event that passes the error check and the tail data block including the tail event that passes the error check; after sorting, the data block behind the data block including the tail event should be the data block including the initial event, or there is no data block behind the data block including the tail event, that is, the data block including the tail event is the last data block in the sorting. If the first data block is missing or damaged in the matching data blocks, there is a next data block in the order of the data block including the last event, but the next data block does not include the initial event. In order to recover as much data in the binary log file as possible, the data block with the smallest position in the first event file behind the data block including the last event can be repaired to obtain the corresponding first data block, so that the binary log file can be restored using the data block behind the data block including the last event, thereby recovering the data in the binary log file to the greatest extent.

[0059] In some embodiments, matching data blocks may be obtained in sequence according to the sorting order, and when the obtained matching data blocks represent the end of a binary log file, the damaged data blocks corresponding to the ended binary log file are repaired. When there is an end event in the obtained matching data blocks or the sum of the sizes of the obtained matching data blocks and the previous data blocks reaches the file size limit, it can be determined that the obtained matching data blocks represent the end of a binary log file. The end of a binary log file indicates that the data blocks required to restore a binary log file have been obtained, so the damaged data blocks in the data blocks required to restore the binary log file are repaired.

[0060] For ease of understanding, an example is used here to illustrate the process of splicing, error checking and repairing data blocks. FIG5 is a flowchart of an example of the process of splicing, error checking and repairing data blocks provided by an embodiment of the present invention. As shown in FIG5, the process of splicing, error checking and repairing data blocks may include steps b1 to b10.

[0061] In step b1, the matching data blocks are sorted according to the server identifier, the first event timestamp, and the first event file position.

[0062] In step b2, determine whether the first data block can be obtained from the information collection table in the sorted order; if yes, jump to step b3; if not, jump to step b7.

[0063] In step b3, the block information of the next data block is obtained from the information set table in the sorted order.

[0064] In step b4, the next data block is spliced with the previous data block, and an error check is performed to determine whether the error check passes; if passed, jump to step b3; if not, jump to step b5.

[0065] In step b5, the data blocks that fail the error detection check are determined as damaged data blocks, and the damaged data blocks are recorded.

[0066] In step b6, determine whether the data block indicates the end of the binary log file; if so, jump to step b2; if not, jump to step b3.

[0067] In step b7, determine whether there are any unrecovered data blocks in the information collection table; if so, jump to step b8; if not, jump to step b9.

[0068] In step b8, the data block with the smallest first event file position in the unrecovered data blocks is obtained as the damaged data block, the missing of the first data block is recorded, and the process goes to step b3.

[0069] In step b9, determine whether there is a record of damaged data blocks; if so, jump to step b10; if not, end.

[0070] In step b10, the damaged data block is repaired.

[0071] Through the above steps b1 to b10, the matching data blocks can be spliced, error-checked and repaired, and the data in the binary log file can be restored to the greatest extent, so that the restored binary log file is available and can be opened and viewed normally.

[0072] Different damaged data blocks may be damaged in different ways and may be repaired in different ways. The damaged data block may be repaired using a repair method corresponding to the event type according to the event type of the event in the damaged data block. Specifically, the event type of the event in the damaged data block is obtained; and the event in the damaged data block is repaired using a repair strategy corresponding to the event type. The length of the damaged event to be repaired may be determined using the event file position of the last event in the damaged data block in the damaged information set and the event file position of the first event in the next data block of the damaged data block, so as to facilitate the subsequent repair of the damaged event. The correspondence between the event type and the repair strategy may be established in advance. After determining the event type in the damaged data block, the repair strategy corresponding to the event type may be searched in the correspondence.

[0073] In some examples, the method of fixed template repair can be used to repair the damaged events in the damaged data block. In the case where the event type belongs to the fixed format event type, the repair strategy includes: obtaining the timestamp of the last event in the previous binary log file of the binary log file to which the damaged data block belongs in chronological order, and repairing the damaged data block according to the timestamp of the last event in the previous binary log file and the template corresponding to the event type. For events of the fixed format event type, even if the data therein is completely destroyed, the timestamp of the start time of the binary log file to which the damaged data block belongs can be determined according to the timestamp of the last time in the previous binary log file. Except for the timestamp, the other elements of the fixed format event type are relatively fixed, and the template corresponding to the event type can be used to repair it, thereby completing the repair of the fixed format event type event in the damaged data block. For example, the initial event FORMAT_DESCRIPTION_EVENT and the last event ROTATE_EVENT are both events of the fixed format event type, and can be repaired using the method of fixed template repair.

[0074] In some examples, the damaged event can be repaired using the context-related event of the damaged event of the damaged data block. In the case where the event type belongs to the query event type, the repair strategy includes: repairing the damaged data block based on events whose event timestamps are before the timestamp of the damaged event of the damaged data block and / or events whose event timestamps are after the timestamp of the damaged event of the damaged data block. Events belonging to the query event type, i.e., query_event, may include but are not limited to write events, i.e., WRITE events, update events, i.e., UPDATE events, delete events, i.e., DELETE events, etc. If an event of the query event type is damaged, the damaged event, i.e., the damaged time, can be repaired by using an event in the previous binary log file with the same record primary key as the damaged event. For example, if the damaged event is UPDATE_ROWS_EVENT or DELETE_ROW_EVENT, the damaged event can be restored in the order of event timestamps; if the damaged event is WRITE_ROWS_EVENT, the damaged WRITE_ROWS_EVENT can be restored using the DELETE event and UPDATE event with the same primary key KEY whose event timestamp is before the event timestamp of the damaged event. In some cases, in order to repair data errors, events with error detection codes such as CRC32 codes can be automatically repaired; for events that lack error detection codes such as CRC32 codes, the block number of the data block where the event is located, the event file location of the event, and other information can be recorded in the repair confirmation log, and merged after manual confirmation by the staff to improve the accuracy of data repair.

[0075] The second aspect of the present invention provides a binary log file recovery device. FIG6 is a schematic diagram of the structure of a binary log file recovery device provided by an embodiment of the present invention. As shown in FIG6 , the binary log file recovery device 200 may include a search module 201, an acquisition module 202, a processing module 203 and a repair module 204.

[0076] The search module 201 may be used to search for an initial event and an end event in a data block device.

[0077] The acquisition module 202 may be used to acquire basic information of the log file from the initial event and the last event when the initial event passes the error detection check and the last event passes the error detection check.

[0078] The basic information of the log file includes the server ID, the file start time and the file end time.

[0079] The processing module 203 can be used to search for data blocks that match the basic information of the log file in the data block device, and perform splicing and error checking on the matching data blocks according to the block information of the matching data blocks to determine the damaged data blocks.

[0080] The block information includes the server ID, the first event timestamp, the first event file position, and the last event file position.

[0081] The repair module 204 can be used to repair the damaged data blocks, splice the successfully repaired damaged data blocks to the corresponding positions, trim the damage events in the damaged data blocks that failed to be repaired, and splice the damaged data blocks after trimming the damage events to the corresponding positions to recover the binary log files.

[0082] In an embodiment of the present invention, the initial event and the end event can be searched in the data block device first, and the basic information of the log file can be obtained from the initial event that has passed the error check and the end event that has passed the error check, so as to search the matching data block in the data block device according to the basic information of the log file, and the data block required for recovering the binary log file includes the matching data block. According to the block information of the data block, the matching data block is spliced and error-checked to determine the damaged data block in the matching data block, and the damaged data block is repaired. If the repair is successful, the successfully repaired damaged data block is spliced to the corresponding position; if the repair fails, the damaged event of the damaged data block is cut, and the cut damaged data block is spliced to the corresponding position. The data blocks required for recovering the binary log file obtained from the data block device can be spliced in sequence to realize the recovery of the binary log file. In the embodiment of the present invention, the recovery process of the binary log file does not rely on the file system information. Even if the file system information is lost and some data blocks are copied to cause the data block content to be destroyed, the binary log file can still be recovered.

[0083] In some embodiments, the search module 201 can be specifically used to: use a rule operation formula generated based on the initial event filtering information to match the initial event in the data block device, and the initial event filtering information includes the expected recovery time period and the event type of the initial event; use a rule operation formula generated based on the end event filtering information to match the end event in the data block device, and the end event filtering information includes the expected recovery time period and the event type of the end event.

[0084] In some embodiments, the damaged data blocks include first-category damaged data blocks and second-category damaged data blocks. The processing module 203 can be specifically used to: sort the matched data blocks according to the server identifier, the first event timestamp and the first event file position of the matched data blocks; determine whether the current data block is a first-category damaged data block based on the tail event file position of the current data block, the order of the current data block in the sorting and the data block size; perform error checking on the current data block, and determine that the current data block is a second-category damaged data block if the error checking fails.

[0085] In some embodiments, the processing module 203 can be specifically used to: determine the expected file position of the current data block according to the order of the current data block in the sorting and the data block size; calculate the difference between the tail event file position of the current data block and the expected file position of the current data block; and determine whether the current data block is a first-category damaged data block based on the size relationship between the difference and a preset standard file position difference.

[0086] In some examples, the processing module 203 can be specifically used to: determine the number of missing data blocks based on the difference and the data block size when the difference is greater than or equal to the standard file position difference; search for candidate data blocks with the missing number based on the block number of the current data block in the data block device; splice the candidate data blocks with the current data block and perform an error check; if the error check fails, determine that the current data block is a first-type damaged data block.

[0087] In some embodiments, the processing module 203 may also be used to: if the error detection check passes, splice the candidate data block between the current data block and the next data block in the sorting.

[0088] In some embodiments, the processing module 203 can also be used to: when it is determined that the current data block is a first-type damaged data block, set a preset damage flag and add the block information of the current data block to the damaged information set; obtain the next data block in the sorting, and when it is detected that the damage flag is set, add the block information of the next data block to the damaged information set and associate it with the block information of the current data block, and reset the damage flag.

[0089] In some embodiments, the damaged data blocks also include the third type of damaged data blocks. The processing module 203 can also be specifically used to: if the data block behind the data block including the last event in the sorting does not include the initial event, determine the data block with the smallest first event file position behind the data block including the last event as the third type of damaged data block.

[0090] In some embodiments, the repair module 204 may be specifically used to: when the matching data block indicates the end of a binary log file, repair the damaged data block corresponding to the ended binary log file.

[0091] In some embodiments, the repair module 204 can be specifically used to: obtain the event type of the event in the damaged data block; and repair the event in the damaged data block using a repair strategy corresponding to the event type.

[0092] In some examples, when the event type belongs to a fixed format event type, the repair strategy includes: obtaining the timestamp of the last event in the previous binary log file of the binary log file to which the damaged data block belongs in chronological order, and repairing the damaged data block according to the timestamp of the last event in the previous binary log file and the template corresponding to the event type. When the event type belongs to a query event type, the repair strategy includes: repairing the damaged data block based on events whose event timestamps are before the timestamp of the damaged event of the damaged data block and / or events whose event timestamps are after the timestamp of the damaged event of the damaged data block.

[0093] The third aspect of the present invention also provides an electronic device. FIG7 is a schematic diagram of the structure of a model publishing device in a distributed system provided by an embodiment of the present invention. As shown in FIG7, the electronic device 300 includes a memory 301, a processor 302, and a computer program stored in the memory 301 and executable on the processor 302.

[0094] In some examples, the processor 302 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present invention.

[0095] The memory 301 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer executable instructions, and when the software is executed (e.g., by one or more processors), it can be operated to perform the operations described with reference to the method for recovering a binary log file according to an embodiment of the present invention.

[0096] The processor 302 runs a computer program corresponding to the executable program code by reading the executable program code stored in the memory 301, so as to implement the binary log file recovery method in the above embodiment.

[0097] In some examples, the electronic device 300 may further include a communication interface 303 and a bus 304. As shown in FIG7 , the memory 301, the processor 302, and the communication interface 303 are connected via the bus 304 and communicate with each other.

[0098] The communication interface 303 is mainly used to realize the communication between the modules, devices, units and / or equipment in the embodiment of the present invention. The input device and / or output device can also be accessed through the communication interface 303.

[0099] The bus 304 includes hardware, software, or both, coupling the components of the electronic device 300 to each other. By way of example and not limitation, bus 304 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCI-E) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable busses, or a combination of two or more of the foregoing. Where appropriate, bus 304 may include one or more busses. Although embodiments of the present invention describe and illustrate particular busses, the present invention contemplates any suitable busses or interconnects.

[0100] The fourth aspect of the present invention also provides a computer-readable storage medium, on which a computer program instruction is stored. When the computer program instruction is executed by a processor, the binary log file recovery method in the above embodiment can be implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the above-mentioned computer-readable storage medium may include a non-transitory computer-readable storage medium, such as a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc., which is not limited here.

[0101] Embodiments of the present invention provide a computer program product in which the instructions in the computer program product are executed by the processor of the electronic device, such that the electronic device performs the recovery method of the binary log file in the above embodiment and achieves the same technical effect, to avoid duplication.

[0102] It needs to be made clear that each embodiment in this specification is described in a progressive manner, the same or similar parts between each embodiment can be referred to each other, each embodiment focuses on the differences from the other embodiments. For device embodiments, equipment embodiments, computer-readable storage media embodiments, and computer program product embodiments, relevant may be found in the description section of method embodiments. The invention is not limited to the particular steps and structures described above and illustrated in the figure. Those skilled in the art may, after comprehending the spirit of the invention, make various alterations, modifications and additions, or alter the sequence between steps. And, for simplicity, a detailed description of known method techniques is omitted here.

[0103] Aspects of the invention are described above with reference to flow diagrams and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the invention. It shall be understood that each box of the flowchart and / or block diagram and a combination of each box in the flowchart and / or block diagram may be implemented by computer program instructions. Such computer program instructions may be provided to a processor of a general-purpose computer, a specialized computer, or other programmable data processing device to produce a machine such that these instructions executed via a processor of a computer or other programmable data processing device enable the implementation of a function / action specified in one or more boxes of the flow diagram and / or block diagram. Such processors may be, but are not limited to, general-purpose processors, specialized processors, special application processors, or field programmable logic circuits. It is also understood that each box of the block diagram and / or flow diagram and a combination of the boxes of the block diagram and / or flow diagram may also be implemented by a specialized hardware that performs a specified function or action, or by a combination of specialized hardware and computer instructions.

[0104] Those skilled in the art should understand that the above embodiments are exemplary rather than restrictive. Different technical features appearing in different embodiments can be combined to achieve beneficial effects. Based on studying the drawings, the specification and the scope of the patent application, those skilled in the art should be able to understand and implement other variations of the disclosed embodiments. In the scope of the patent application, the term "including" does not exclude other devices or steps; the quantifier "one" does not exclude multiple; the terms "first" and "second" are used to indicate names rather than to indicate any specific order. Any diagrammatic mark in the claim should not be understood as limiting the scope of protection. The functions of multiple parts appearing in the claim can be implemented by a single hardware or software module. The appearance of certain technical features in different dependent claims does not mean that these technical features cannot be combined to achieve beneficial effects.

[0105] 200: Binary log file recovery device 201: Find module 202: Get the module 203: Processing module 204: Repair module 300: Electronic equipment 301: Memory 302: Processor 303: Communication Interface 304: Bus step

Claims

1. A method for recovering a binary log file, characterized in that it includes: The binary log file recovery device searches for the initial and final events in the data block device. The initial event is the first event in the binary log file, and the final event is the last event in the binary log file. The binary log file records changes to the table structure and modifications to table data in the database. If both the initial and final events pass error detection checks, the binary log file recovery device obtains basic log file information from the initial and final events. This basic log file information includes the server identifier, file start time, and file end time. The binary log file recovery device then searches for a data block in the data block device that matches the basic log file information. Based on the block information of the matched data block, the binary log file recovery device performs concatenation and error verification on the matched data block to determine the corrupted data block. The block information includes the server identifier, first event timestamp, first event file location, and last event file location. The binary log file recovery device repairs the damaged data blocks, splices the successfully repaired damaged data blocks to the corresponding positions, trims the damaged events from the damaged data blocks that failed to be repaired, and splices the trimmed damaged data blocks to the corresponding positions to recover the binary log file.

2. The method as described in request item 1, wherein, The binary log file recovery device searches for initial and final events in the data block device, including: The binary log file recovery device uses a rule-based algorithm generated based on initial event filtering information to match initial events in the data block device, wherein the initial event filtering information includes the desired recovery time period and the event type of the initial event; The binary log file recovery device uses a rule-based algorithm generated based on final event filtering information to match final events in the data block device, wherein the final event filtering information includes the desired recovery time period and the event type of the final event.

3. The method as described in request item 1, wherein, The corrupted data blocks include first-type corrupted data blocks and second-type corrupted data blocks. The binary log file recovery device determines the corrupted data blocks by splicing and verifying the matching data blocks according to the block information of the matching data blocks, including: the binary log file recovery device sorting the matching data blocks according to the server identifier, first event timestamp, and first event file position of the matching data blocks; the binary log file recovery device determining whether the current data block is a first-type corrupted data block based on the tail event file position of the current data block, the current data block's order in the sorting, and the data block size; and the binary log file recovery device performing error verification on the current data block. If the error verification fails, the current data block is determined to be a second-type corrupted data block.

4. The method as described in request item 3, wherein, The binary log file recovery device determines whether the current data block is a first type of damaged data block based on the tail event file position of the current data block, the current data block's position in the sorting sequence, and the data block size. This includes: the binary log file recovery device determining the expected file position of the current data block based on its position in the sorting sequence and the data block size; the binary log file recovery device calculating the difference between the tail event file position of the current data block and its expected file position; and the binary log file recovery device determining whether the current data block is a first type of damaged data block based on the relationship between the difference and a preset standard file position difference.

5. The method as described in request item 4, wherein, The binary log file recovery device determines whether the current data block is a first type of damaged data block based on the relationship between the difference and the preset standard file position difference, including: if the difference is greater than or equal to the standard file position difference, the binary log file recovery device determines the number of missing data blocks according to the difference and the data block size; the binary log file recovery device searches for candidate data blocks of the missing number according to the block number of the current data block in the data block device; the binary log file recovery device concatenates the candidate data blocks with the current data block and performs error verification; if the error verification fails, the binary log file recovery device determines that the current data block is a first type of damaged data block.

6. The method as described in claim 5, wherein, It also includes: If the error verification passes, the binary log file recovery device will concatenate the candidate data block between the current data block and the next data block in the sorting.

7. The method as described in request item 5, wherein, It also includes: when the current data block is determined to be a first type of damaged data block, the binary log file recovery device sets a preset damage flag and adds the block information of the current data block to the damage information set; the binary log file recovery device obtains the next data block in the sorting, and when it detects that the damage flag is set, the binary log file recovery device adds the block information of the next data block to the damage information set and associates it with the block information of the current data block, and resets the damage flag.

8. The method as described in request item 3, wherein, The corrupted data blocks also include a third type of corrupted data blocks. The binary log file recovery device performs splicing and error verification on the matching data blocks based on the block information of the matching data blocks to determine the corrupted data blocks. It also includes: if the data blocks following the data block that includes the end event in the sorting do not include the initial event, the binary log file recovery device determines the data block with the smallest first event file position following the data block that includes the end event as the third type of corrupted data block.

9. The method as described in claim 1, wherein, The binary log file recovery device repairs the damaged data block by: when a matching data block indicates the end of a binary log file, the binary log file recovery device repairs the damaged data block corresponding to the ended binary log file.

10. The method as described in claim 1, wherein, The binary log file recovery device repairs the damaged data block by: the binary log file recovery device acquiring the event type of the events in the damaged data block; and the binary log file recovery device using a repair strategy corresponding to the event type to repair the events in the damaged data block.

11. The method as described in claim 10, wherein, When the event type is a fixed-format event type, the repair strategy includes: the binary log file recovery device obtains the timestamp of the last event in the previous binary log file of the binary log file to which the damaged data block belongs, in chronological order; the binary log file recovery device repairs the damaged data block based on the timestamp of the last event in the previous binary log file and the template corresponding to the event type. When the event type is a query event type, the repair strategy includes: the binary log file recovery device repairs the damaged data block based on events whose timestamps precede the timestamp of the damaged event of the damaged data block and / or events whose timestamps follow the timestamp of the damaged event of the damaged data block.

12. A device for recovering a binary log file, characterized in that it comprises: The lookup module is used to find the initial event and the last event in the data block device. The initial event is the first event in the binary log file, and the last event is the last event in the binary log file. The binary log file records the table structure changes and table data modifications in the database. The system comprises the following modules: an acquisition module, used to acquire basic log file information from the initial and final events, provided that both the initial and final events pass error detection checks; a processing module, used to locate data blocks in the data block device that match the basic log file information; a processing module, used to perform concatenation and error verification on the matching data blocks based on their block information to identify corrupted data blocks, whereby the block information includes the server identifier, the first event timestamp, the first event file location, and the last event file location; and a repair module, used to repair the corrupted data blocks by concatenating successfully repaired corrupted data blocks to their corresponding positions, trimming corrupted events from corrupted data blocks that failed to be repaired, and concatenating the trimmed corrupted data blocks to their corresponding positions to restore the binary log file.

13. An electronic device, characterized in that it comprises: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the method for recovering a binary log file as described in any one of requests 1 to 11.

14. A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the method for recovering a binary log file as described in any one of claims 1 to 11.

Images