Method for quickly locating failed position of MySQL binary log incremental recovery import

Abstract

The present application relates to the technical field of database, more further, it relates to MySQL binary log incremental recovery import failure position quick positioning method, this method includes: step one: in target database server executes flush-logs command to generate a target end latest binary log file, wherein, the relative order of the conversion process keeps the event line boundary, end_log_pos and CRC32 checksum unchanged;Step two: using a co-occurrence detector to scan the target end binary log text file line by line, and the end_log_pos value determined in the stable pairing source end co-occurrence line analysis is determined as the final recovery breakpoint position;Step three: a controller generates incremental recovery parameters according to the final recovery breakpoint position, if import is successful, record the recovery breakpoint position and the line number mapping of the source end and target end co-occurrence line. The present application significantly improves the degree of automation and accuracy of incremental recovery, reduces the data loss or repeated import caused by inaccurate breakpoint positioning.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of databases, and particularly relates to a method for quickly locating a failure position of a MySQL binary log incremental recovery import. BACKGROUND

[0002] In the field of database operation and disaster recovery, binary log (Binlog) is a core mechanism provided by MySQL database, which can record all events that modify the database content, such as INSERT, UPDATE, DELETE, etc. When using binary log for recovery, the common technical route is "full backup + incremental log recovery". That is, a full library snapshot is first generated through physical or logical means, and then binary logs are replayed in chronological order from the snapshot point, so as to recover to a specified time point. This method is widely used in industries with high requirements for data consistency, such as finance, e-commerce and communication. However, in actual application, the recovery process based on binary log often faces complex technical challenges.

[0003] The common practice of the prior art is to rely on binlog tools (such as mysqlbinlog) to parse log files, and manually or through scripts to specify the offset end_log_pos of the start position and end position, so as to determine the start and end points of the import. Although this method is basically feasible, it has significant limitations: first, the continuity of the binary log file is not always reliable. For example, when log switching (flush-logs), master-slave synchronization delay, and transaction cross-file writing, log events may be split into multiple files. At this time, the offset value of the manually positioned recovery start point is extremely prone to error, and if the error is only within a range of tens of bytes, it may also cause subsequent event parsing failure or import duplication, ultimately destroying data consistency. Secondly, the existing method often uses time stamp as the main reference point to locate the import breakpoint, but the time stamp may have time zone offset or microsecond precision loss problems at different nodes, which cannot guarantee strict one-to-one correspondence. In addition, the existing incremental recovery process mostly relies on static parameter configuration, such as continuing to import at a certain fixed end_log_pos offset. This method lacks a dynamic checking mechanism, and once the import fails in the middle, it is difficult to quickly backtrack and re-determine the breakpoint position. In practice, operation and maintenance personnel often need to repeat trial and error: repeatedly adjust the start position parameter, re-import the log, and then observe whether an error is reported. This not only is inefficient, but also may cause additional burden to the target database in the process of multiple failures. Especially in the environment with huge log volume (tens of GB level) or transaction-intensive, such trial-and-error process is almost unacceptable. SUMMARY

[0004] The main purpose of the present application is to provide a method for quickly locating the failed position of MySQL binary log incremental recovery import, which establishes a co-occurrence detection mechanism between the binary log text files of the target and source ends, combines a finite state machine to accurately analyze the end_log_pos value, and introduces a three-anchor point check composed of CRC32 checksum and event header timestamp, thereby achieving high-precision confirmation of the recovery breakpoint position. When the preliminary check fails, the method can automatically find stable pairing using bidirectional neighborhood search, thereby solving the positioning difficulty problem caused by cross-file transactions or log differences. Further, the controller supports automatic resuming and rollback retry based on stable pairing during the recovery process, making the recovery process highly robust and adaptive. Compared with the existing method which relies on manual experience and fixed parameters, the present application significantly improves the automation and accuracy of incremental recovery, reduces data loss or repeated import caused by inaccurate breakpoint positioning, effectively shortens the recovery time in a large-scale database environment, improves the consistency and stability of the system, and has strong engineering application value.

[0005] To solve the above problems, the technical scheme of the present application is as follows:

[0006] The method for quickly locating the failed position of MySQL binary log incremental recovery import comprises the following steps:

[0007] Step 1: Execute the flush-logs command on the target database server to generate a target end latest binary log file, obtain the target end latest binary log file and an imported source end binary log file, and convert the two binary log files into two line-by-line readable text files respectively, thereby obtaining a target end binary log text file and a source end binary log text file, wherein the conversion process keeps the relative order of the event line boundary, end_log_pos and CRC32 checksum unchanged;

[0008] Step two: a co-occurrence detector is used to scan the target binary log text file line by line, identify the co-occurrence line containing end_log_pos and CRC32 checksum, and count the co-occurrence lines to obtain a target co-occurrence count value; the co-occurrence detector is used to locate the candidate co-occurrence line corresponding to the target co-occurrence count value in the source binary log text file; a finite state machine is used to parse the end_log_pos value in the candidate co-occurrence line and confirm the existence of the CRC32 checksum in the line; three anchor point checks are performed, and if the check passes, the parsed end_log_pos value is determined as the recovery breakpoint position; if the check fails, a bidirectional neighborhood search is performed with the candidate co-occurrence line as the center, the co-occurrence line pairs of the source and target are enumerated in the preset window, and three anchor point checks are performed, until a stable pairing is obtained, and the end_log_pos value parsed in the source co-occurrence line in the stable pairing is determined as the final recovery breakpoint position;

[0009] Step three: a controller generates an incremental recovery parameter based on the final recovery breakpoint position, imports the subsequent binary logs in the file number increasing order from the next event corresponding to the recovery breakpoint position; if the import fails, the rollback retry is performed based on the latest stable pairing; if the import is successful, the recovery breakpoint position and the line number mapping of the source and target co-occurrence lines are recorded.

[0010] Further, step one specifically includes: executing a flush-logs command on the target database server to lock the current binary log file and create a new binary log file, and locking the binary log file as the latest binary log file of the target; selecting a source binary log file that has been imported; converting the target binary log file and the source binary log file into a target binary log text file and a source binary log text file respectively; during the conversion, keeping the line boundary of the event unchanged, unifying the character encoding to UTF-8, and unifying the line feed to a single line feed, while preserving the original spelling and relative order of end_log_pos and CRC32 checksum.

[0011] Further, step one further includes text preprocessing of the target binary log text file and the source binary log text file, which includes: decoding any binary log text file into a string line by line from the first line, if single-line decoding fails, replacing it with a preset replacement character and continuing to process the subsequent line; deleting the white space at the beginning and end of each line; converting the English letters in each line to lowercase uniformly; preserving an original line copy of each preprocessed line.

[0012] Further, the process of step two of scanning the target binary log text file line by line using the co-occurrence detector includes: for each line of the target binary log text file, performing an in-line match using the two keywords of end_log_pos and CRC32 checksum; if both end_log_pos and CRC32 checksum are matched in the current line, the current line is identified as a co-occurrence line, and a target co-occurrence counter is incremented by one; after scanning all lines of the target binary log text file, the final value of the target co-occurrence counter is recorded as the target co-occurrence count value.

[0013] Further, the process of step two of locating the candidate co-occurrence line in the source binary log text file includes: scanning the source binary log text file line by line, and using the co-occurrence detector to identify co-occurrence lines; each time a co-occurrence line is identified, a source co-occurrence counter is incremented by one; when the value of the source co-occurrence counter is equal to the target co-occurrence count value for the first time, the scanning is stopped, and the currently identified co-occurrence line is determined as the candidate co-occurrence line.

[0014] Further, the process of step two of using the finite state machine to parse the end_log_pos value in the candidate co-occurrence line includes: initializing the finite state machine to an initial state; starting from the beginning of the candidate co-occurrence line, scanning character by character, and when the end_log_pos keyword is detected, the finite state machine is switched to a guide state; starting from the guide state, skipping non-digit characters, and when the first digit character is detected, the finite state machine is switched to a collection state; in the collection state, consecutive digit characters are read and combined into a decimal integer until a non-digit character is encountered, and the decimal integer is taken as the end_log_pos value; after the end_log_pos value is collected, the finite state machine is switched to a verification state, in which the existence of the CRC32 checksum keyword is confirmed in the remaining part of the candidate co-occurrence line.

[0015] Further, the three-anchor point verification in step two includes: defining the first anchor point as the existence of the end_log_pos value in the co-occurrence line; defining the second anchor point as the existence of the CRC32 checksum in the co-occurrence line; and defining the third anchor point as the timestamp in the event header associated with the co-occurrence line; wherein the event header is identified by a preset rule that requires the event header to contain both serverid and end_log_pos in the same line, and the timestamp to exist within a preset proximity range of the line; for a set of source co-occurrence lines and target co-occurrence lines, it is checked whether they meet the conditions of the first anchor point and the second anchor point, respectively, and the event header is searched in their proximity range to extract the timestamp; when both source and target co-occurrence lines meet the conditions of the first anchor point and the second anchor point, and the extracted timestamps are the same, the three-anchor point verification is determined to pass.

[0016] Further, the bidirectional neighborhood search in step two includes: defining a search window with the candidate co-occurrence line as the center in the source end binary log text file and the target end binary log text file; in the search window, combining the co-occurrence lines of the source end and the target end according to a preset enumeration strategy to form a pair of co-occurrence lines to be verified; performing three-anchor point verification on each pair of co-occurrence lines to be verified; when a pair of co-occurrence lines passes the three-anchor point verification for the first time, the pair of co-occurrence lines is determined as a stable pair, and the bidirectional neighborhood search is terminated; if all the pairs of co-occurrence lines in the search window do not pass the three-anchor point verification, the range of the search window is expanded and the foregoing enumeration and verification steps are repeated; if a stable pair is not found before the file boundary of any binary log text file is reached, the next co-occurrence line in the source end binary log text file is selected as a new candidate co-occurrence line, and the bidirectional neighborhood search is re-executed.

[0017] Further, step three specifically includes: the controller generates an incremental recovery parameter according to the final recovery breakpoint position, which specifies that the incremental import is performed starting from the event after the recovery breakpoint position; if the incremental import fails, the controller performs a rollback retry based on the position of the latest found stable pair; if the incremental import succeeds, the controller performs persistent storage of the final recovery breakpoint position, the line numbers of the source end and target end co-occurrence lines corresponding to the final recovery breakpoint position, and the results of the three-anchor point verification.

[0018] The MySQL binary log incremental recovery import failure position rapid positioning method of the application has the following beneficial effects: it can effectively solve the problems of difficult automatic determination of the recovery breakpoint, import failure caused by cross-file transactions, and low efficiency of manual repeated trial and error in the prior art. The application introduces a co-occurrence detection mechanism between the target end and the source end binary log text file, so that the positioning process of the recovery breakpoint is no longer dependent on manual judgment of the time stamp or fixed offset, thereby improving the automation level of positioning. Further, the application uses a finite state machine to analyze the end_log_pos value, and combines the CRC32 checksum and the event header timestamp to form a three-anchor point verification mechanism, realizing multi-dimensional consistency confirmation and maintaining high robustness in the scene of slight differences in log format or transaction cross-file. When the verification fails, the application also designs a bidirectional neighborhood search strategy, which automatically enumerates the source and target co-occurrence row pairs within a preset range, gradually expands the search window, and finds a stable pairing that meets the three-anchor point conditions, greatly improving the reliability of breakpoint positioning. At the same time, the controller of the application has automatic resuming and backtracking retry functions during import, and can quickly roll back and re-import based on the latest stable pairing once the import fails, avoiding complex manual intervention and reducing the risk of data loss or repeated import. In summary, the application can significantly shorten the recovery time in a large-scale database environment, improve the consistency and stability of incremental recovery, meet the high requirements of industries such as finance and e-commerce for high availability and data consistency, and has obvious engineering application value and promotion significance. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The method flowchart of the MySQL binary log incremental recovery import failure position rapid positioning method provided for the embodiments of the application is shown in the figure.

[0020] Figure 2 The co-occurrence detector performance change characteristic curve diagram provided for the embodiments of the application is shown in the figure.

[0021] Figure 3 The bidirectional neighborhood search algorithm convergence and window parameter optimization characteristic diagram provided for the embodiments of the application is shown in the figure. DETAILED DESCRIPTION

[0022] In order to enable personnel in the technical field to better understand the application scheme, the technical solutions in the embodiments of the application will be described clearly and completely below in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are only a part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the application.

[0023] The target side gets the target side latest binary log file mysql-bin.000895 after performing flush-logs, and the source side last imported binary log file is mysql-bin.000894. The two binary log files have been converted into target side binary log text file and source side binary log text file according to step one, the relative order of event's line boundary, end_log_pos and CRC32 checksum has been kept unchanged, the character encoding is unified to UTF-8, and the line break is unified to single line break. This embodiment shows the whole process from target side co-occurrence detection, source side candidate co-occurrence line positioning, finite state machine parsing, three anchor point verification, bidirectional neighborhood search to final breakpoint position determination, and from next event import and backtracking retry.

[0024] Get target side co-occurrence count value on target side binary log text file: for clear illustration, extract some lines (only show key lines; actual file contains more context lines) related to this alignment in target side binary log text file, line number is generated when converting:

[0025] T#690:# at521600

[0026] T#691:240918 15:42:08 serverid 100 end_log_pos 521760 CRC32 0x9a7b2c11

[0027] T#692:SETTIMESTAMP=1695051728;

[0028] T#693:# at521760

[0029] T#694:240918 15:42:09 serverid 100 end_log_pos 521972 CRC32 0xa2b3cc90

[0030] T#695:SETTIMESTAMP=1695051729;

[0031] T#696:# at521972

[0032] T#697:240918 15:42:10 serverid 100 end_log_pos 522112 CRC32 0x5f3a1cde

[0033] T#698:SETTIMESTAMP=1695051730;

[0034] T#699:# at522112

[0035] T#700:24091815:42:11serverid100end_log_pos522368CRC320x8e71a2b3

[0036] T#701:SETTIMESTAMP=1695051731;

[0037] T#702:#at522368

[0038] T#703:24091815:42:12serverid100end_log_pos522640CRC320x1c2d3e4f

[0039] T#704:SETTIMESTAMP=1695051732;

[0040] The co-occurrence detector identifies lines that simultaneously contain both end_log_pos and a CRC32 checksum (e.g., T#691, T#694, T#697, T#700, T#703). Note: The first binary log text file on the target end Line text. : The total number of lines in the target binary log text file. :like If it contains both end_log_pos and CRC32 checksum, then ,otherwise . Target co-occurrence count value. Scanning and counting by row yields: In the shown segment, T#691, T#694, T#697, T#700, and T#703 all satisfy the co-occurrence condition. If two other co-occurrences previously appeared within the entire scope of this document, the specific values ​​obtained at the end of this statistical analysis are: ;here This means that the target binary log text file contains a total of 7 occurrences of the corresponding common occurrence from the first line to the current line. Locating candidate common occurrences and parsing the end_log_pos value in the source binary log text file: The relevant segment in the source binary log text file (corresponding to mysql-bin.000894) is illustrated below:

[0041] S#730:#at521560

[0042] S#731:24091815:42:08serverid101end_log_pos521720CRC320x9a7b2c11

[0043] S#732: SETTIMESTAMP = 1695051728;

[0044] S#733: #at521720

[0045] S#734: 240918 15:42:09 server id 101 end_log_pos 521932 CRC32 0xa2b3cc90

[0046] S#735: SETTIMESTAMP = 1695051729;

[0047] S#736: #at521932

[0048] S#737: 240918 15:42:10 server id 101 end_log_pos 522112 CRC32 0x5f3a1cde

[0049] S#738: SETTIMESTAMP = 1695051730;

[0050] S#739: #at522112

[0051] S#740: 240918 15:42:11 server id 101 end_log_pos 522240 CRC32 0xff11aa22

[0052] S#741: SETTIMESTAMP = 1695051731;

[0053] S#742: #at522240

[0054] S#743: 240918 15:42:11 server id 101 end_log_pos 522368 CRC32 0x8e71a2b3

[0055] S#744: SETTIMESTAMP = 1695051731;

[0056] S#745: #at522368

[0057] S#746: 240918 15:42:12 server id 101 end_log_pos 522640 CRC32 0x1c2d3e4f

[0058] S#747: SETTIMESTAMP = 1695051732;

[0059] Also define: : the line number of the first line of the source binary log text file. : the line text of the first line of the source binary log text file. : the total number of lines of the source binary log text file. : the end_log_pos value of the candidate co-occurrence line. : the CRC32 checksum of the candidate co-occurrence line. , otherwise . : the minimum line number satisfying , used to locate the candidate co-occurrence line. Scan the source file line by line, and increment the source co-occurrence counter whenever a co-occurrence line is found. Stop when the source co-occurrence counter is equal to for the first time, and the current identified line is the candidate co-occurrence line. According to the above snippet, let the 7th co-occurrence line first appear at S#737 (illustrated), then: : candidate co-occurrence line : the end_log_pos value parsed from the candidate co-occurrence line (candidate value). For S#737 line: S#737:24091815:42:10 serverid101 end_log_pos 522112 CRC32 0x5f3a1cde; the result of the finite state machine is: .

[0060] Define for three-anchor point verification: : there exists a parseable end_log_pos value in the candidate co-occurrence line. : there exists a CRC32 checksum keyword in the candidate co-occurrence line, and a segment composed of hexadecimal characters can be read after it (length not less than 8). : the timestamp text of the source candidate co-occurrence line associated event header (extracted from this line or the next 3 lines). : the timestamp text of the target corresponding co-occurrence line associated event header (extracted from this line or the next 3 lines). In the vicinity of the candidate line S#737: the candidate line satisfies and . The source timestamp text is extracted from S#738: ; in the vicinity of the target co-occurrence line T#700 corresponding to the candidate area: the target timestamp text is extracted from T#701: ; compare: ; therefore, the three-anchor point verification fails, triggering bidirectional neighborhood search.

[0061] Perform a bidirectional neighborhood search using candidate co-occurrence behavior centers and obtain stable pairings: Define the bidirectional neighborhood search as follows: : Search window radius, representing the maximum number of rows included both upwards and downwards from the center row. This example initially retrieves... . : In absolute row difference The set of co-occurrence pairs that need to be checked within the layer is enumerated in order of proximity to the center. Stable pairing: A co-occurrence pair between the source and target ends that simultaneously passes the three anchor point checks. Search process overview: Centered on the candidate co-occurrence S#737 at the source end and the candidate region T#700 at the target end, first check... The combination within, check again. The combinations within, increasing sequentially. Until Stop when a stable pairing is found.

[0062] examine Source S#740 and target T#700: S#740 line: 24091815:42:11serverid101end_log_pos522240CRC320xff11aa22; parsed to obtain Its timestamp text is taken from S#741: The timestamp text in line T#700 on the target end is taken from T#701: Three-anchor point determination: If the end_log_pos and CRC32 checksum are true in the same row, and... The condition is met. The three-anchor point check is now satisfied. To improve robustness, another pair of combinations closer to the center on the same layer, S#743 and T#700, is also checked (still belonging to...). Another pair under the "nearest center priority" order): S#743:24091815:42:11serverid101end_log_pos522368CRC320x8e71a2b3; parsed to obtain Its timestamp text is taken from S#744: The target timestamp text is consistent with that of T#700 / T#701, and the CRC32 checksum is the same as that in the target T#700 line, which is 0x8e71a2b3. This pair shows stronger consistency in terms of field values ​​in addition to the three anchor points, so the latter is given priority as a stable pair.

[0063] The stable pair is thus determined: source-side co-occurrence line: S#743 (end_log_pos 522368, CRC32 checksum 0x8e71a2b3, timestamp text "24091815:42:11"). Target-side co-occurrence line: T#700 (end_log_pos 522368, CRC32 checksum 0x8e71a2b3, timestamp text "24091815:42:11"). The end_log_pos value parsed from the source-side co-occurrence line in the stable pair is determined as the final recovery breakpoint position. Note: : Final recovery breakpoint position (stable value). : Explanation: The reason for choosing S#743 instead of S#740 in the layer is that the CRC32 checksums on both sides are completely consistent and the timestamp texts are consistent, making the line-level correspondence more deterministic; based on the strategy of prioritizing the closer to the center, S#743 is best aligned with S#700, reducing the need for further searching.

[0064] Apply to incremental recovery parameter generation and import: starting file and starting offset: the starting file is located at the source's mysql-bin.000894. The starting offset is set to . Since end_log_pos identifies the end boundary of the current event, continue parsing from this position to the next event header to get the import starting point. Set the starting position of the parsed next event header as: : File internal starting position of the next event header (position). : The next event header is indeed started from here). Start from the of mysql-bin.000894 and import subsequent events and subsequent files mysql-bin.000895 in ascending order of file sequence number. When importing, group commit when encountering transaction start and transaction end markers to ensure that transaction boundaries remain consistent. During the import process, record the starting position and timestamp text of the event header for each event submitted; for example, in mysql-bin.000894, 3 events are submitted in sequence, with event header starting positions of 522368, 522640, and 523040, and corresponding timestamp texts of "24091815:42:12", "24091815:42:13", and "24091815:42:15". Note: : The event header starting position of the th successfully submitted event. : The timestamp text of the th successfully submitted event. Example record:

[0065] ​

[0066] Suppose that during the import of mysql-bin.000895, after parsing event header position 120144, the target database server returns a constraint conflict error. The controller immediately pauses the import and retryes based on the most recent stable pair. The rollback point selection: the most recent stable pair remains S#743 and T#700, corresponding to... Rewind to the next event header at that position, and start again from... The import process begins. The reason for rolling back to this position is that it has passed three-anchor point verification and field consistency checks, minimizing the probability of mismatches and avoiding duplicate imports caused by returning to uncertain historical positions. Re-import after verification: After rolling back, the controller performs consistency checks on fields near the rollback point: whether end_log_pos is 522368, whether the CRC32 checksum is 0x8e71a2b3, and whether the timestamp text is "24091815:42:11". If all three are consistent, replay continues. If the re-import fails again, the rollback strategy remains unchanged, and the same stable pair is used as the baseline rollback point for retrying until the import succeeds or structural conflicts are manually addressed.

[0067] After successful import, the controller records the following data for subsequent batches to be directly reset and audited: Restore breakpoint position: The current line number mapping between the source and target is: S#743↔T#700. Import range: [522368, end of file] in mysql-bin.000894 and [beginning of file, end of file] in mysql-bin.000895. Key snapshot: A list for quick resumption of transmission after an external interruption. Obtain the co-occurrence count value at the target end. Locate and analyze candidate coexisting entities at the source end. After the three-anchor-point verification failed, a stable pair S#743↔T#700 was obtained through bidirectional neighborhood search, and the breakpoint location was finally determined. It imports subsequent binary logs in ascending order of file number, starting from the next event corresponding to the recovery breakpoint. In case of failure, it retryes based on the most recent stable pair. After success, it records the recovery breakpoint position and line number mapping for use in subsequent batch resets and audits.

[0068] refer to Figure 1 A method for quickly locating the location of MySQL binary log incremental recovery import failure, which includes:

[0069] Step one: execute flush-logs command on target database server to generate a target latest binary log file, get the target latest binary log file and an imported source binary log file, and convert the two binary log files into two line-by-line readable text files respectively, to obtain a target binary log text file and a source binary log text file, wherein the conversion process keeps the relative order of event line boundaries, end_log_pos and CRC32 checksum unchanged.

[0070] In an embodiment, a session with the permission to execute flush-logs command and read binary log file is established on the target database server. The flush-logs command is triggered to make the binary log file currently being written immediately close, and a new file is generated in the same directory, which is the target latest binary log file. In this way, the log split point of the target can be fixed at the time boundary, avoiding the influence of subsequent conversion stages by concurrent writing, so as to ensure the stability of the text content in subsequent matching and counting. The exact file name and path of the target latest binary log file are obtained by executing an instruction for querying the current binary log state, avoiding the positioning deviation caused by the file name increment rule or time zone difference. According to the progress record of the recovery process, an imported source binary log file is located. The file has been imported in the last round of incremental import, and selecting this file as a reference can make the target binary log text file and the source binary log text file generated subsequently form comparable sections on the event time axis, so that the line number count of the co-occurrence detector has monotonicity and reproducibility.

[0071] The conversion target is to transform the binary log content into text content readable line by line without loss, to obtain the target end binary log text file and the source end binary log text file. When converting, the line boundary maintenance rule of the event is followed: the header information of each event is output per line, and the natural separation between events is maintained, and the contents of adjacent events are not combined into the same line. The advantage of this is that the co-occurrence detector can perform counting at the event granularity with a single linear scan, without the need to backtrack across events, and can maintain linear complexity and stable memory occupation under large files. When converting, the relative order maintenance rule is followed: ensure that end_log_pos and CRC32 checksum are presented in the same event header line in the original appearance order. Maintaining the relative order can reduce the branch judgment complexity of the subsequent finite state machine, reduce the possibility of ambiguous matching, and improve the consistency of breakpoint analysis. When generating the target end binary log text file and the source end binary log text file, a fixed-size buffer is used for block writing, and line integrity checking is performed at each block boundary. If it is detected that the half line of the event header appears at the block tail, the half line is retained to the next block header for splicing, until a complete line is formed and then written out. This method can reduce the memory peak in the high concurrency and large file scenario, and ensure that the event header line is not damaged by block segmentation.

[0072] The binary log file stream is decoded into a text line stream using the official binary log decoding tool. To ensure that the event header contains end_log_pos and CRC32 checksum, the option to output the event header key fields and line-level time information is enabled when calling the tool. This guarantees that each event header in the text appears as a parsable fixed token, facilitating subsequent parsing and verification. A line-by-line standardization process is established to handle the decoded text in the following order: uniform character encoding to UTF-8, uniform line terminator to a single newline character, removal of byte order mark, and avoidance of newline differences generated by different operating systems affecting line counts. The event header split-line newline is regularized: if the tool output causes the event header information to be folded into multiple lines due to terminal line breaks or line width limitations, the adjacent lines of the event header are combined into one line in the original left-to-right reading order without crossing event boundaries, ensuring that end_log_pos and CRC32 checksum are on the same line. This eliminates the noise of line breaks at the presentation level while not changing the line boundaries between events. The content of non-event headers is output as is, line by line, without changing its relative position relationship with adjacent event header lines. This allows more context to be restored when needed, but does not interfere with the judgment of the co-occurrence detector, as the co-occurrence detector only counts on the line containing end_log_pos and CRC32 checksum. The relative order is checked before each line is written: when end_log_pos and CRC32 checksum appear on the same line, verify that the token order in the line is consistent with the order in the input text; if folding and merging bring the risk of incorrect order, concatenate the input fragments in the order of their occurrence to ensure that the relative order does not change. The text stream after standardization is written to stable storage, generating the target binary log text file and the source binary log text file. Each text file is subjected to a line count and key word coverage check after writing is complete, to confirm that end_log_pos is present at least on the event header line, and CRC32 checksum is present in scenarios where checksum output is enabled. This check is used to discover tool option configuration errors or log damage in advance, so that the co-occurrence detection link can fail quickly and be corrected before entering the co-occurrence detection link.

[0073] After executing the flush-logs command and obtaining the latest binary log file of the target, a clear boundary can be formed at the moment of log switching, so that the last complete event in the binary log text file of the target and the imported range in the binary log text file of the source form a comparable stable interval, reducing the probability of event truncation caused by concurrent writing. By normalizing the event header to one line and keeping the relative order of end_log_pos and CRC32 checksum unchanged, the co-occurrence detector can complete the same occurrence judgment within a single line, and the finite state machine can complete the extraction of end_log_pos and the confirmation of the existence of CRC32 checksum within a single line, avoiding cross-line state transfer and improving the certainty and speed of parsing. Uniform character encoding and line feed are helpful to maintain consistent line numbers and matching results between different platforms and different file systems, avoiding inconsistent co-occurrence count values in the same text in different environments.

[0074] The binary log file is directly input into the decoding tool through a pipe, and the decoding output is directly written into the target binary log text file and the source binary log text file after real-time line-by-line standardization. This method is suitable for scenarios with large file sizes or limited storage space, and can reduce disk occupancy and input / output delay. In the presence of read-only slave, execute the flush-logs command on the read-only slave and obtain the latest binary log file of the target for conversion, avoiding interference with the log write path of the master instance. The target binary log text file obtained by the read-only slave still meets the requirements of event line boundary and relative order preservation, and can directly participate in subsequent co-occurrence detection and breakpoint analysis.5.4 Strict order preservation method: When the event header line output by the original tool has inconsistent field order or missing optional fields, do not rearrange or supplement the fields, only perform line folding and encoding line feed standardization, keeping the relative order unchanged. This can ensure consistency with the assumptions of the subsequent co-occurrence detector and finite state machine, avoiding matching ambiguity caused by unauthorized rewriting of field order.

[0075] No write operation is performed on the target's latest binary log file between the execution of flush-logs command and the start of the conversion, preventing tail writes due to buffer delays from occurring during the conversion, thus affecting the stability of event row boundaries. After the generation of the target binary log text file and the source binary log text file, the time of file generation, the total number of rows, the number of rows containing end_log_pos and CRC32 checksum are recorded in the audit record of the recovery procedure, so that the input quality can be quickly determined when entering the subsequent steps. When the version difference of the decoding tool causes inconsistent behavior of whether to output the CRC32 checksum, by enabling the option that can print the checksum or performing the conversion in an environment that supports the checksum, it is ensured that the CRC32 checksum field exists in the text. If the target environment does not support the output of the CRC32 checksum, the environment should be modified before entering the subsequent steps to avoid the co-occurrence detector from being unable to form an effective count.

[0076] Step two: using a co-occurrence detector to scan the target binary log text file line by line, identifying co-occurrence lines containing end_log_pos and CRC32 checksum, and counting the identified co-occurrence lines to obtain a target co-occurrence count value; using the co-occurrence detector to locate the candidate co-occurrence line in the source binary log text file corresponding to the target co-occurrence count value; using a finite state machine to parse the end_log_pos value in the candidate co-occurrence line and confirm that the CRC32 checksum exists in the line; performing three anchor point verification, if the verification passes, the parsed end_log_pos value is determined as the recovery breakpoint position; if the verification fails, a bidirectional neighborhood search is performed around the candidate co-occurrence line, enumerating the co-occurrence line pairs of the source and target in the preset window and performing three anchor point verification until a stable pairing is obtained, and the end_log_pos value parsed in the source co-occurrence line in the stable pairing is determined as the final recovery breakpoint position.

[0077] In one embodiment, the inputs are the target binary log text file and the source binary log text file. Both have been generated in Step 1, ensuring the relative order of event's line boundary, end_log_pos and CRC32 checksum is unchanged. The co-occurrence detector processes the text files in a line-by-line manner, detecting whether end_log_pos and CRC32 checksum co-occur within a single line. This line-by-line detection avoids misjudgment caused by state transfer across lines, making the recognition result only rely on the explicit character content of the current line, thus remaining stable when the log appears line wrapping, comments or optional field changes. The target binary log text file is processed line by line and the co-occurring lines are counted to obtain the target co-occurrence count value. Then the same detection strategy is used to process the source binary log text file line by line and count, and when the source count first equals the target co-occurrence count value, the current line is located as the candidate co-occurring line. In this way of aligning the count by line number, stable mapping can be achieved without relying on line numbers or timestamp order, because the co-occurring line only appears in the event header containing the key field, and its distribution in the logs on both ends has comparable consistency.

[0078] The process of line-by-line scanning and co-occurrence line identification of the co-occurrence detector includes: reading a line of text, keeping the line intact as an immutable string, and obtaining a view for matching only. This way, the original line can be directly referenced when backtracking to a specific fragment later, avoiding repeated parsing. Two keyword matches are performed within the line, targeting end_log_pos and CRC32 checksum respectively. The matching strategy uses a combination of character-by-character prefix advancement and local backtracking: starting from the beginning of the line, a keyword is advanced character by character, and when an inconsistent character is encountered, the advancement is backtracked to the longest known prefix position of the keyword and continues. The keyword is complete or the end of the line is reached. Then the same advancement process is performed for the other keyword. When both keywords appear in the same line, the line is determined as a co-occurrence line. If end_log_pos or CRC32 checksum appears multiple times in a line, the line is still counted as one co-occurrence line, because the goal of co-occurrence detection is to confirm the existence of co-occurrence in the same line, not the number of occurrences. Repeat the above process for each line of the target binary log text file. Each time a co-occurrence line is identified, the target co-occurrence counter is incremented by one. At the end of the file scan, the final value of the target co-occurrence counter is recorded as the target co-occurrence count value. This way, a robust line degree metric can be obtained without parsing the details of the value, which can be used for subsequent sequential positioning at the source. The same line-by-line scanning and co-occurrence line identification are performed on the source binary log text file. Each time a co-occurrence line is identified, the source co-occurrence counter is incremented by one. When the value of the source co-occurrence counter is equal to the target co-occurrence count value for the first time, the scanning is stopped immediately, and the current line is determined as the candidate co-occurrence line. The first-equal stopping criterion ensures that the candidate co-occurrence line and the co-occurrence line at the target are aligned in statistical order, reducing the offset caused by insertion events or optional field differences.

[0079] The finite state machine parses the end_log_pos value and confirms the existence of the CRC32 checksum in the candidate co-occurrence line, specifically including: the design goal of the finite state machine is to reliably extract the decimal integer immediately following the end_log_pos in a single line range, and to confirm the existence of the CRC32 checksum in the same line. Single-line parsing can avoid ambiguity caused by cross-line reading, while allowing the implementation to work in the form of a character stream, thereby adapting to different whitespace and punctuation styles. For the initial state: start scanning character by character from the beginning of the line, looking for a complete end_log_pos keyword. After detecting the keyword, enter the guide state. If the keyword is not detected until the end of the line, it is determined that the line does not meet the parsing conditions, and the candidate co-occurrence line needs to be repositioned. For the guide state: continue scanning after the end_log_pos keyword, skipping non-numeric characters such as spaces, tabs, colons, or equals signs in order. This way, it can accommodate different tools inserting formatting characters between the keyword and the value, while ensuring that the final captured character sequence starts with a number. For the collection state: start reading numeric characters from the first numeric character, and read the numeric sequence as a decimal integer. When the first non-numeric character is encountered, the collection is ended and the decimal integer is latched as the end_log_pos value. If no numeric characters are read, the line is marked as a failed parsing line and the candidate co-occurrence line needs to be repositioned. For the verification state: continue scanning within the same line to confirm the existence of a complete CRC32 checksum keyword. If it exists, a segment composed of hexadecimal characters can be read after it, with a length of at least 8, to be used for consistency checking in subsequent steps; if the hexadecimal segment is missing or the characters are illegal, the presence of the CRC32 checksum keyword alone is the pass condition. The existence of the verification state ensures that the parsing will not be misjudged due to the presence of unrelated numeric values within the line, and provides useful information for subsequent three-anchor point verification. For submission: after the verification state passes, the latched end_log_pos value is output as the parsing result, and the current line number, the start and end positions of the end_log_pos value within the line, and the position of the CRC32 checksum keyword within the line are recorded together as the context for subsequent steps. Recording these positions helps to track and backtrack, and when subsequent verification fails or neighborhood search is needed, the specific character range can be directly located without the need to re-scan the entire line. The reason for using a finite state machine to parse characters one by one instead of directly using whole-line matching is that character-by-character parsing is more tolerant of formatting differences in input. For example, some tools may insert different amounts of spaces between the keyword and the value, or change the separator from a colon to an equals sign. Character-by-character parsing can absorb these differences without sacrificing determinism, reducing false rejections, and improving portability.

[0080] When the target binary log text file does not identify any co-occurrence line, the exception is recorded and the current process is terminated to prompt the check of whether the conversion option in Step One enables the output containing end_log_pos and CRC32 checksum. The effective target co-occurrence count value cannot be obtained in this case. When the source binary log text file does not have a count equal to the target co-occurrence count value before the scan is completed, it indicates that there are structural differences between the logs on the selected comparison file that cannot be aligned. At this time, the maximum source co-occurrence count value and the corresponding line number that have been identified are recorded for subsequent neighborhood search or rollback strategies. When the candidate co-occurrence line fails to parse the end_log_pos value or fails to confirm the existence of the CRC32 checksum in the finite state machine stage, the line is marked as a parsing failure line, and the next co-occurrence line is scanned as a new candidate co-occurrence line from the source binary log text file, and the finite state machine parsing is repeated. Through this single-line replacement retry, the line number alignment idea can be maintained while avoiding full file rollback. To reduce the impact of occasional character damage on the results, the maximum number of skipped characters is limited between the guide state and the collection state of the finite state machine, for example, no more than 32 non-digit characters. Limiting the skip range can prevent mis-collection caused by crossing to the next field, while taking into account the reasonable variation in delimiter length in different tool outputs.

[0081] Establishing the line number alignment relationship between the two ends using co-occurrence line count can avoid direct dependence on file line numbers or timestamp ordering. When logs have deleted comments, added optional fields, and empty lines inserted by different export tools, file line numbers are not stable, while the relative distribution of event header lines containing end_log_pos and CRC32 checksum is more stable between the two ends, so using co-occurrence line count as the alignment basis is more reliable. Limiting the parsing of end_log_pos values to within the candidate co-occurrence line can ensure that the extracted values are indeed from the event header containing the key fields, avoiding the extraction of similar numbers from unstructured description lines, thereby improving the accuracy of subsequent breakpoint calculation. While confirming the existence of the CRC32 checksum keyword, the hexadecimal fragment after it is retained, which can provide information sources for subsequent consistency checks. When the logs on both ends have content shifts near the same event, the existence and readability of the CRC32 checksum value can help determine whether the candidate co-occurrence line is truly corresponding, thereby reducing false matches.

[0082] In the co-occurrence detector, the end_log_pos and CRC32 checksum are respectively preset with complete matching templates. It is first determined whether the same row hits the two templates at the same time. If it hits, the row is identified as a co-occurrence row. In the finite state machine stage, the end position template is used once to extract the decimal integer from the candidate co-occurrence row, and the checksum template is used to confirm the existence of the CRC32 checksum keyword. This method is simple and easy to migrate across languages, but attention should be paid to backtracking control on extremely long rows. Two pointers are pushed forward at the same time within the same row. The left pointer focuses on the search for end_log_pos, and the right pointer focuses on the search for CRC32 checksum. Either pointer reaches the matching end point to record the hit position. Both of them hit to determine the co-occurrence row. In the finite state machine stage, the left hit position is used to start the guide and collection, and the right hit position is used to confirm the existence of the keyword. This method can parallel process the two keywords in the same row in a multi-core environment, improving the speed of long row analysis. For super large text files, a fixed size buffer is read and scanned by row in the buffer. When a half row appears at the end of the buffer, it is spliced with the beginning of the next buffer before matching to ensure the integrity of a single row. This method can run stably in a memory-limited environment while maintaining the correctness of co-occurrence detection and finite state machine analysis.

[0083] A co-occurrence row in a target end binary log text file, a candidate co-occurrence row in a source end binary log text file, an end_log_pos value obtained by parsing in the candidate co-occurrence row, and a CRC32 checksum confirmed to exist in the candidate co-occurrence row are obtained. The three anchor points are composed of the following objects: the first anchor point is the existence of the end_log_pos value in the candidate co-occurrence row; the second anchor point is the existence of the CRC32 checksum in the candidate co-occurrence row; and the third anchor point is the timestamp text of the event header associated with the respective main body rows on both sides. By checking these three items at the same time, a consistent match across files can be established using the structural and temporal identifiers of the same event without relying on row numbers and external historical information. The first anchor point limits the analysis target to the end position field, the second anchor point limits the row to the event header context containing the checksum information, and the third anchor point is used to cross-source to collate the consistency of the event occurrence sequence, thereby reducing the risk of false matching caused by only text similarity.

[0084] At the co-occurrence line of the target binary log text file, search for the event header both upwards and downwards simultaneously. The event header is identified according to preset rules: the line containing both serverid and end_log_pos, and there exists a timestamp within the preset proximity of the line. One preferred setting of the preset proximity is the current line and no more than 3 lines after it. The reason for choosing no more than 3 lines is that the main stream export tool outputs the timestamp immediately after the event header, and the timestamp appearing beyond this range usually belongs to the next event or context annotation, which is not suitable for inclusion in the same event's time identification. After identifying the event header, extract the timestamp text in the following order: first search for obvious timestamp identifier fragments such as settimestamp or ts within the event header line, and then search for complete timestamp expressions composed of numbers and separators within the preset proximity line by line. If multiple time representations appear simultaneously, prefer the most complete expression with year, month, day, hour, minute, and second. Repeat the above identification and extraction process at the candidate co-occurrence line of the source binary log text file to obtain the timestamp text of the source. In order to avoid misjudgment caused by pure format differences, the timestamp texts on both sides are normalized before comparison: remove extra spaces, unify the fixed order of year, month, day, hour, minute, and second, and retain numbers and necessary separators. Keeping necessary separators helps to distinguish different fields and avoid ambiguity caused by connecting different time fields into a string of numbers.

[0085] Check the first anchor point: confirm that the end_log_pos value has been parsed within the candidate co-occurrence line and that the value is composed of consecutive numbers. This confirmation ensures that the recovery breakpoint position comes from the end position field, not from other miscollected numbers. Check the second anchor point: confirm that there is a CRC32 checksum keyword within the candidate co-occurrence line and read as much of the hexadecimal fragment as possible after it, with a length of no less than 8. This confirmation is used to prove that the line has checksum information, indicating that it belongs to the event header range. Even if there are length differences or case differences in the hexadecimal fragment, as long as the keyword exists, it can be considered to meet the second anchor point for compatibility with different tool output styles. Check the third anchor point: compare the timestamp text of the target with the timestamp text of the source completely. The benefit of complete consistency comparison is to avoid mistaking similar times as the same event, thereby maintaining the certainty of event-level alignment. When the timestamp texts on both sides are completely consistent, and the first anchor point and the second anchor point are both met, it is determined that the three-anchor point check is passed. When the three-anchor point check is passed, the end_log_pos value parsed within the candidate co-occurrence line is directly determined as the recovery breakpoint position, and the line numbers and timestamp texts used for comparison on both sides are recorded for subsequent audit and rollback review. If any anchor point is not met, it is determined that the check fails, providing a trigger condition for entering the bidirectional neighborhood search.

[0086] A symmetric row window is established in the source binary log text file and the target binary log text file, respectively, with the candidate co-occurrence as the center. A preferred setting is 16 rows on each side, forming a preset window of 16 rows above and below the center row. The consideration of 16 rows is that the density of event headers is usually high, and 16 rows can cover adjacent events within the same time period while keeping the combination enumeration within an engineering-acceptable range. It is ensured that only co-occurrence rows are included in the window. If a row does not meet the co-occurrence condition, the row is not involved in enumeration, thereby reducing invalid comparisons and ensuring that each check falls within the range of event headers containing key fields.

[0087] The co-occurrence row pairs of the source and the target are enumerated in order of absolute row difference from the center from small to large. For any absolute row difference d, the co-occurrence row pairs with the same offset are first attempted for comparison, i.e., the combination of the source center row d rows above or below and the target center row d rows above or below. This strategy prioritizes the combination that is most likely to align when the event rhythm on both sides is similar, and can obtain stable pairing with fewer attempts. When the combination with the same offset fails the three-anchor point check, cross combinations within the same ring layer are expanded, i.e., the source center row d rows above respectively with the target center row d rows below, and the source center row d rows below respectively with the target center row d rows above. Cross combinations are used to cover the case of inconsistent relative displacement caused by inserted or lost events. In the case of multiple co-occurrence rows within the same absolute row difference, the order of close-to-center priority is used for enumeration, i.e., the co-occurrence row closer to the center is enumerated first, and the farther co-occurrence row is enumerated second. Close-to-center priority can shorten the search path and reduce the risk of mispairing, because the candidate co-occurrence row is usually close to the area where the true breakpoint is located. For each enumerated co-occurrence row pair, the three-anchor point check of embodiment one is performed. The co-occurrence row pair that passes the three-anchor point check for the first time is defined as a stable pair, and the enumeration is immediately terminated. The definition of a stable pair requires that three constraints be met simultaneously, and can still ensure consistent alignment at the event level even in the presence of text format differences.

[0088] If all the co-occurrence line pairs in the preset window do not form a stable pairing, the window is symmetrically enlarged to twice the original size, while maintaining the constraint of only including co-occurrence lines, and then the enumeration and verification process is repeated. Equal proportion expansion can balance the search range and cost without changing the enumeration order rule, gradually covering a larger neighborhood. When either side reaches the file boundary and still does not form a stable pairing, continue to scan the next co-occurrence line as a new candidate co-occurrence line in the source binary log text file, reset the window and center, and perform a bidirectional neighborhood search again. This approach can overcome local misalignment caused by transaction cross-file or cross-event boundaries, and continue to move forward until an alignable event is found. When a stable pairing occurs, the end_log_pos value parsed from the source co-occurrence line in the stable pairing is directly determined as the final recovery breakpoint position, and the source and target line numbers, timestamp text, and window size of the pairing are recorded for subsequent rollback strategy reference. If a stable pairing is still not formed within the acceptable maximum window range and acceptable scanning span, save the largest end_log_pos value and the corresponding source line number that have been parsed so far as the initial reference point for subsequent rollback retry. This can avoid unbounded search caused by infinite expansion, while preserving the most reliable end position clues so far.

[0089] The three constraints of end position existence, verification information existence, and timestamp consistency simultaneously limit the candidate event. The end position existence constraint eliminates the risk of mis-taking numbers from non-key fields; the verification information existence constraint limits the judgment range to the event header range containing the CRC32 checksum; and the timestamp consistency constraint provides a verifiable basis for cross-source alignment. The three constraints can remain stable in the presence of format changes, optional field additions and deletions, and empty line insertions. Bidirectional neighborhood search expands around the candidate co-occurrence line, first trying the same offset, then expanding by cross combination, which can quickly cover the most likely aligned neighborhood while avoiding the exponential explosion brought by full combination. In most scenarios, the true corresponding event is close to the candidate co-occurrence line, and distance priority can reduce search cost and shorten recovery time. Equal proportion expansion helps to gradually expand the coverage range when local alignment is not possible, while preserving the order of priority near the center, ensuring that once a reliable pairing occurs, it can converge as soon as possible.

[0090] Before the complete consistent comparison, the two-side timestamp texts are uniformly formatted. In the scenario of containing time zone identification, the time is first converted to the same time zone, and then the complete consistent comparison is performed. This method is suitable for database systems deployed across time zones, and can eliminate the time difference of the same event recorded on different nodes. Within the same absolute row difference, the co-occurrence row pairs containing longer hexadecimal fragments are compared first, and then the co-occurrence row pairs containing shorter fragments are compared. Longer hexadecimal fragments are usually from complete CRC32 checksum output, and preferentially comparing these row pairs can improve the probability of a hit. The window growth is changed from equal proportion expansion to fixed step expansion, for example, increasing 8 rows on both sides each time. Fixed step is suitable for scenarios where the density of log events is known and fine control of search cost is expected. When the timestamp is not found within the preset proximity range, but the next data row of the same event contains a field that can be inferred as a timestamp, an additional 2 rows are allowed to be extended backward for supplementary search. This method is used to be compatible with some export tools that place the timestamp multiple lines after the event header, while not breaking the main judgment logic of the preset proximity range.

[0091] When the two-side timestamp texts are still inconsistent after formatting, but the first anchor point and the second anchor point in the three anchor points both meet the conditions and the end_log_pos value shows a monotonic increasing relationship between adjacent row pairs, the third anchor point condition should not be relaxed, and the search for row pairs that meet the three anchor point check should continue through bidirectional neighborhood search to avoid mispairing adjacent but asynchronous events as stable pairs. When multiple window expansions still do not produce stable pairs, the conversion settings in step one can be reviewed to see if the relative order of the event's row boundary and end_log_pos and CRC32 checksum has been maintained, and if necessary, the target binary log text file and the source binary log text file are regenerated before executing the process of this embodiment.

[0092] Step three: a controller generates incremental recovery parameters according to the final recovery breakpoint position, and imports subsequent binary logs in ascending order of file sequence number from the next event corresponding to the recovery breakpoint position; if the import fails, the rollback retry is performed based on the latest stable pair; if the import is successful, the recovery breakpoint position and the row number mapping of the co-occurrence rows of the source and target are recorded.

[0093] In one embodiment, the controller reads the final restore point position, the corresponding source and target co-occurrence lines, and the stable pair's timestamp text. The starting file and offset are determined: locate the source binary log file containing the final restore point position in the source binary log file set, and use the final restore point position as the starting offset. The reason for starting from the next event after the final restore point position is that the end_log_pos identifies the end boundary of the current event, and setting the starting offset to this boundary can avoid repeated playback of events that have already been successfully imported, while ensuring that subsequent imports are continuous playback in complete events as the minimum unit. The subsequent binary log files are sorted according to the file name sequence number in the catalog, forming an ordered list from the starting file to the last available file. The approach of importing in ascending order of file sequence number can maintain the chronological order of events consistent, preventing transaction replay failures caused by cross-file out-of-order. The incremental restore parameters are constructed, including the starting file path, the starting offset, the file ordered list, the event header parsing switch, the CRC32 checksum verification switch, and the skip strategy when encountering an unrecognized event. The event header parsing switch and the CRC32 checksum verification switch are set to enabled, so that format checking and checksum verification are performed on each event header during the import process, so that problems can be detected as early as possible and rollback triggered when input anomalies occur.

[0094] The controller opens the starting file, positions the read pointer at the starting offset, and parses the event header at the current position to confirm that the position is exactly at the starting boundary of an event. If the read pointer is not at the legal starting of an event header, it is pushed forward until a complete event header is detected, and then the import begins. Taking boundary confirmation can avoid starting from the middle of a half event, preventing the generation of incomplete statement sequences. The events in the starting file are traversed in sequential read mode, each event is converted into a target executable statement and immediately submitted to the target database server. When a transaction start identifier is encountered, it is not submitted until the transaction end identifier, which can maintain consistency between the source and target at the transaction boundary. After the current file is imported, the next binary log file is switched according to the file ordered list, starting from the first event header of the file and continuing to import until the end of the list. The list switching uses strict sequential advancement, without skipping any files, avoiding data gaps caused by omissions. During the import process, the controller records the current file name, the starting position of the current event header in the file, the timestamp text of the current event header, and the end_log_pos of the last successfully submitted event, as real-time progress. Real-time progress is used to quickly locate the rollback point when a failure occurs, and is used to write audit records when a success occurs.

[0095] When a statement execution failure, an event parsing failure or an error returned by the target database server occurs during the import process, the controller immediately suspends the import and locates the nearest stable pair. The nearest stable pair refers to the pair among the stable pairs recorded before the current failure position, whose source-side co-occurrence line number is the largest and does not exceed the source-side line number corresponding to the failure position. The reason for selecting the nearest stable pair is that the pair has passed the three-anchor point check and can provide a verified end position landmark. Backing up to the landmark can minimize the range of repeated import and avoid jumping to an unverified area. The controller resets the start file to the source-side binary log file corresponding to the nearest stable pair and resets the start offset to the end_log_pos value parsed within the source-side co-occurrence line of the stable pair. At this time, the import still starts from the next event corresponding to the end_log_pos value to maintain the idempotent protection of the successful events. The incremental recovery parameters are regenerated and the import process is executed again. If it fails again, the controller repeats selecting an earlier stable pair as a new rollback point. The earlier stable pairs are rolled back in order from near to far according to the line number until the import can continuously advance or the stable pairs are exhausted. Before each rollback retry, the controller performs an independent consistency check on the event header at the rollback point: reads the event header row near the rollback point, and checks whether the end_log_pos, CRC32 checksum, timestamp text and the previous record are consistent. This check can timely discover the situation that the input file is replaced or damaged before entering the replay, avoiding using the wrong start point for import.

[0096] When the entire file order list is imported and no unprocessed error occurs, the controller writes the final recovery breakpoint position, the start file, the last imported file and the time range into the audit record. The controller solidifies the line number mapping of the source-side and target-side co-occurrence lines to the persistent medium. The line number mapping includes the following: the candidate co-occurrence line number used to determine the final recovery breakpoint position and the corresponding target-side co-occurrence line number, the line number pairs of the stable pairs used as rollback references during the import process, and the start line number and end line number of each rollback. The solidification of the line number mapping can enable the subsequent batch to directly locate the last successful position and provide traceable evidence for fault analysis. The controller also saves the real-time progress snapshot recorded during the import process, including the current file name at each file switch and the end_log_pos of the last successful submitted event. The snapshot is used for quick resumption after abnormal power failure or external interruption without re-executing the three-anchor point check and bidirectional neighborhood search.

[0097] When the statement execution strategy of the target database server is sensitive to duplicate events, it must be ensured that the import starts from the next event after the final restore breakpoint position, and events falling before the restore breakpoint position are not allowed to be replayed. This constraint can avoid unique constraint conflicts caused by duplicate submission. When the physical naming of the source binary log file does not completely conform to the time sequence, the controller should use the naming rule containing the incremental serial number as the basis for sorting, and not use the file modification time as the basis for sorting, in order to prevent sequence errors caused by file system timestamp drift. When a long-time event-free empty file appears, the file still needs to be included in the ordered list and traversed empty to ensure that the continuity of the file serial number is verified to prevent skipping that causes the sequence of subsequent files to be offset. When multiple backtracking still cannot be advanced, the controller should stop continuing to enlarge the backtracking range, retain the backtracking point information that can stably parse the event header last time, and prompt to review whether the input of step one and step two meets the constraint of keeping the line boundary and relative sequence, to prevent invalid retries on incorrect input.

[0098] Figure 2 For the performance characteristic curve of the co-occurrence detector with the complexity of the binary log file, the horizontal axis represents the binary log complexity index, the value range is 0-100, the left side of the vertical axis represents the detection accuracy percentage, the value range is 0-100%, and the right side of the vertical axis represents the processing time, the unit is millisecond, and the value range is 0-500 ms. As shown in FIG. 6, the detection accuracy percentage is 100% when the binary log complexity index is 0, and the processing time is 0 ms. When the binary log complexity index is 100, the detection accuracy percentage is 0, and the processing time is 500 ms. When the binary log complexity index is 50, the detection accuracy percentage is 50%, and the processing time is 250 ms. Figure 1As shown in the figure, the figure contains five main characteristic curves. Among them, the end_log_pos recognition accuracy curve is a black thick solid line, the starting point is (0, 53%), and as the complexity index increases, it shows a slow first and then slow upward trend, reaches a key inflection point at a complexity index of 30 (30, 62%), and then the rising speed slows down, and finally reaches a saturation value at a complexity index of 100 (100, 73%). The CRC32 check pass rate curve is a black thick dashed line, the starting point is (0, 48%), the overall trend is similar to the end_log_pos curve but the value is slightly lower, and enters the stable region at a complexity index of 70 (70, 67%), indicating that the CRC32 check algorithm has good stability in high complexity logs. The co-occurrence row count accuracy curve is a black dotted line, the starting point is (0, 45%), and the curve reflects the recognition ability of the co-occurrence detector in the present application to the rows containing end_log_pos and CRC32 checksum. The processing time curve is a black thin solid line, which shows a clear nonlinear characteristic, and the processing time basically remains in the interval of 450-350 ms in the complexity index range of 0-20, but a significant performance inflection point appears after the complexity index is 60, and the processing time sharply decreases to below 70 ms, which shows that the algorithm has better time efficiency when processing high complexity logs. The acceptable accuracy threshold line marked in the figure is 40%, which divides the entire complexity range into three performance intervals: simple log area (0-20), medium complexity log area (20-50), and high complexity log area (50-100), providing an important reference basis for parameter tuning in practical applications.

[0099] Figure 3 The figure is a bidirectional neighborhood search algorithm convergence and window parameter optimization characteristic diagram, the horizontal axis represents the search window size, the unit is row, the value range is 5-75 rows, the left side of the vertical axis represents the matching success rate percentage, the value range is 0-100%, and the right side of the vertical axis represents the average iteration number, the value range is 0-50 times. As shown in the figure, the figure contains five main characteristic curves. Among them, the end_log_pos recognition accuracy curve is a black thick solid line, the starting point is (0, 53%), and as the complexity index increases, it shows a slow first and then slow upward trend, reaches a key inflection point at a complexity index of 30 (30, 62%), and then the rising speed slows down, and finally reaches a saturation value at a complexity index of 100 (100, 73%). The CRC32 check pass rate curve is a black thick dashed line, the starting point is (0, 48%), the overall trend is similar to the end_log_pos curve but the value is slightly lower, and enters the stable region at a complexity index of 70 (70, 67%), indicating that the CRC32 check algorithm has good stability in high complexity logs. The co-occurrence row count accuracy curve is a black dotted line, the starting point is (0, 45%), and the curve reflects the recognition ability of the co-occurrence detector in the present application to the rows containing end_log_pos and CRC32 checksum. The processing time curve is a black thin solid line, which shows a clear nonlinear characteristic, and the processing time basically remains in the interval of 450-350 ms in the complexity index range of 0-20, but a significant performance inflection point appears after the complexity index is 60, and the processing time sharply decreases to below 70 ms, which shows that the algorithm has better time efficiency when processing high complexity logs. The acceptable accuracy threshold line marked in the figure is 40%, which divides the entire complexity range into three performance intervals: simple log area (0-20), medium complexity log area (20-50), and high complexity log area (50-100), providing an important reference basis for parameter tuning in practical applications. Figure 3As shown, the one-way search matching rate curve is a black thick solid line, the starting point is (5, 15%), and it presents a monotonically increasing trend with the increase of window size, and reaches the highest value (75, 74%) when the window size is 75 rows. The two-way search matching rate curve is a black thick dashed line, the starting point is (5, 30%), which is obviously better than one-way search, and reaches the key performance point (35, 78%) at the window size of 35 rows, and then the growth slows down, and finally reaches (75, 83%) at 75 rows. The comparison fully proves the superiority of the two-way neighborhood search strategy in the application. The three anchor point verification success rate curve is a black dotted line, which reflects the comprehensive verification effect of the first anchor point (end_log_pos value exists), the second anchor point (CRC32 checksum exists) and the third anchor point (event header timestamp matching) in the core technology of the application. The starting point of the curve is (5, 22%), and there is a significant improvement at the window size of 40 rows (40, 68%), indicating that a proper search window has a key influence on the success rate of three anchor point verification. The average iteration number curve is a black thin solid line, which shows the convergence characteristics of the algorithm. The iteration number is high (20, 35 times) before the window size of 20 rows, and then it quickly decreases and is basically stable below 10 times at the window size of 50 rows, which reflects the efficiency of the algorithm. The window expansion failure rate curve is a black thin dashed line, which remains at a low level before the window size of 45 rows, and then sharply rises, indicating that a too large search window will cause the performance of the algorithm to deteriorate. The optimal window region marked in the figure is 30-50 rows, and the critical failure rate line is 30%, which provides a clear guidance for parameter setting in actual deployment.

[0100] The above-described and above-mentioned embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for quickly locating the failure position of MySQL binary log incremental recovery import, characterized in that, The method comprises: Step one: executing a flush-logs command on a target database server to generate a target-side latest binary log file, obtaining the target-side latest binary log file and an imported source-side binary log file, and converting the two binary log files into two line-by-line readable text files respectively to obtain a target-side binary log text file and a source-side binary log text file, wherein the conversion process keeps the relative order of event line boundaries, end_log_pos and CRC32 checksum unchanged; Step two: using a co-occurrence detector to perform line-by-line scanning on the target-side binary log text file, identifying co-occurrence lines containing end_log_pos and CRC32 checksum, and counting the identified co-occurrence lines to obtain a target-side co-occurrence count value; using the co-occurrence detector to locate a candidate co-occurrence line corresponding to the target-side co-occurrence count value in the source-side binary log text file; using a finite state machine to parse the end_log_pos value in the candidate co-occurrence line and confirming that the CRC32 checksum exists in the line; performing three-anchor point verification, if the verification passes, determining the parsed end_log_pos value as the recovery breakpoint position; if the verification fails, performing bidirectional neighborhood search with the candidate co-occurrence line as the center, enumerating the co-occurrence line pairs of the source side and the target side in a preset window and performing three-anchor point verification until a stable pairing is obtained, and determining the end_log_pos value parsed in the source-side co-occurrence line in the stable pairing as the final recovery breakpoint position; Step three: a controller generates an incremental recovery parameter according to the final recovery breakpoint position, starts from the next event corresponding to the recovery breakpoint position, and imports subsequent binary logs in ascending order of file number; if the import fails, backtracking and retrying based on the latest stable pairing; if the import is successful, recording the recovery breakpoint position and the line number mapping of the source-side and target-side co-occurrence lines; The three-anchor point verification in step two comprises: defining the first anchor point as the existence of end_log_pos in the co-occurrence line; defining the second anchor point as the existence of CRC32 checksum in the co-occurrence line; and defining the third anchor point as the timestamp in the event header associated with the co-occurrence line; wherein the event header is identified by a preset rule, which requires that the event header line contains serverid and end_log_pos, and the timestamp exists in the preset adjacent range of the line; for a group of source-side co-occurrence lines and target-side co-occurrence lines, check whether they meet the conditions of the first anchor point and the second anchor point, and find the event header in their adjacent range to extract the timestamp; when the two co-occurrence lines of the source side and the target side both meet the conditions of the first anchor point and the second anchor point, and the extracted timestamps are the same, it is determined that the three-anchor point verification passes.

2. The method for quick positioning of failed position of MySQL binary log incremental restore import according to claim 1, wherein, The step one specifically comprises: executing a flush-logs command on the target database server to lock the current binary log file and create a new binary log file, and taking the locked binary log file as the latest binary log file of the target end; selecting a source end binary log file which has been imported completely; converting the latest binary log file of the target end and the source end binary log file into a target end binary log text file and a source end binary log text file respectively; during the conversion, keeping the line boundary of events unchanged, unifying the character coding to UTF-8, unifying the line break to a single line break, and meanwhile preserving the original spelling and relative order of end_log_pos and CRC32 checksum.

3. The method for quick positioning of failed position of MySQL binary log incremental restore import according to claim 2, characterized in that, The step one further comprises text preprocessing of the target end binary log text file and the source end binary log text file, and the preprocessing comprises: decoding any binary log text file into a string from the first line, replacing the preset replacement character and continuing to process the subsequent line if the single line decoding fails; deleting the blank characters at the beginning and end of each line; uniformly converting the English letters in each line to the lower case form; and preserving an original line copy of each preprocessed line.

4. The method of Claim 3, wherein, The process of scanning the target end binary log text file line by line by using the co-occurrence detector in the step two comprises: using the two keywords of end_log_pos and CRC32 checksum to perform intraline matching on each line of the target end binary log text file; if the end_log_pos and the CRC32 checksum are both matched in the current line, the current line is identified as a co-occurrence line, and a target end co-occurrence counter is incremented by one; after completing the scanning of all lines of the target end binary log text file, the final value of the target end co-occurrence counter is recorded as the target end co-occurrence count value.

5. The method of claim 4, wherein the method further comprises: determining whether the binary log file is a MySQL binary log file; and if the binary log file is a MySQL binary log file, then determining whether the binary log file is a MySQL binary log file that is in a format that is compatible with the MySQL server. The process of locating the candidate co-occurrence line in the source end binary log text file in the step two comprises: scanning the source end binary log text file line by line and identifying the co-occurrence line by using the co-occurrence detector; each time a co-occurrence line is identified, a source end co-occurrence counter is incremented by one; when the value of the source end co-occurrence counter is equal to the target end co-occurrence count value for the first time, the scanning is stopped, and the currently identified co-occurrence line is determined as the candidate co-occurrence line.

6. The method of Claim 5, wherein, The process of parsing the end_log_pos value from the candidate co-occurrence line in step two includes: initializing the finite state machine to an initial state; scanning the candidate co-occurrence line from the beginning of the line character by character, and when the end_log_pos keyword is detected, the finite state machine is converted to a leading state; starting from the leading state, skipping non-digit characters, and when the first digit character is detected, the finite state machine is converted to a collection state; in the collection state, the subsequent digit characters are continuously read and combined into a decimal integer until a non-digit character is encountered, and the decimal integer is taken as the end_log_pos value; after the collection of the end_log_pos value is completed, the finite state machine is converted to a verification state, and in the verification state, the existence of the CRC32 checksum keyword is confirmed in the remaining part of the candidate co-occurrence line.

7. The method of quickly locating the failed position of MySQL binary log incremental restore import according to claim 6, wherein, The bidirectional neighborhood search in step two includes: defining a search window with the corresponding behavior center of the candidate co-occurrence line in the source end binary log text file and the target end binary log text file; in the search window, the co-occurrence lines of the source end and the target end are combined according to a preset enumeration strategy to form a pair of co-occurrence lines to be verified; for each pair of co-occurrence lines to be verified, three anchor point verification is performed; when the first pair of co-occurrence lines passes the three anchor point verification, the pair of co-occurrence lines is determined as a stable pair, and the bidirectional neighborhood search is terminated; if all pairs of co-occurrence lines in the search window do not pass the three anchor point verification, the range of the search window is expanded and the above enumeration and verification steps are repeated; if a stable pair is not found before the file boundary of any binary log text file is reached, the next co-occurrence line in the source end binary log text file is selected as a new candidate co-occurrence line, and the bidirectional neighborhood search is re-executed.

8. The method for quickly locating the location of MySQL binary log incremental recovery import failure as described in claim 7, characterized in that, Step three specifically includes: the controller generates an incremental recovery parameter according to the final recovery breakpoint position, which specifies that the incremental import starts from the event after the recovery breakpoint position; if the incremental import fails, the controller performs a rollback retry based on the position of the latest found stable pair; if the incremental import succeeds, the controller persists the final recovery breakpoint position, the line number of the source end and target end co-occurrence line corresponding thereto, and the result of the three anchor point verification.

Images