Method for quickly positioning MySQL binary log incremental recovery import failure position

Abstract

The invention relates to the technical field of databases, in particular to a MySQL binary log incremental recovery import failure position quick positioning method, which comprises the following steps of: 1, executing a flush-logs command in a target database server to generate a target end latest binary log file, in the conversion process, the line boundary of the event and the relative sequence of endlogpos and CRC32 checksum are kept unchanged; step 2, scanning the binary log text file of the target end line by line by adopting a co-occurrence detector, and determining an endlogpos value analyzed in the co-occurrence of the source end in the stable pairing as a final recovery breakpoint position; and step 3, the controller generates an increment recovery parameter according to the final recovery breakpoint position, and if the import is successful, the recovery breakpoint position and the common line number mapping of the source end and the target end are recorded. According to the method, the automation degree and accuracy of incremental recovery are remarkably improved, and data loss or repeated import caused by inaccurate breakpoint positioning is reduced.

Description

Technical Field

[0001] This invention belongs to the field of database technology, specifically relating to a method for quickly locating the location of MySQL binary log incremental recovery import failure. Background Technology

[0002] In the field of database operation and disaster recovery, the binary log is a core mechanism provided by MySQL databases, capable of recording all events that modify database content, such as INSERT, UPDATE, and DELETE. When using the binary log for recovery, a common technical approach is "full backup + incremental log recovery." This involves first generating a full database snapshot physically or logically, and then replaying the binary log in chronological order after the snapshot point to restore to a specified point in time. This method is widely used in industries with extremely high data consistency requirements, such as finance, e-commerce, and telecommunications. However, in practical applications, the recovery process based on the binary log often faces complex technical challenges.

[0003] The common practice in existing technologies is to rely on binlog tools (such as mysqlbinlog) to parse log files, manually or through scripts specifying the start and end offsets `end_log_pos` to determine the start and end points of the import. While this approach is generally feasible, it has significant limitations: First, the continuity of binary log files is not always reliable. For example, during log switching (flush-logs), master-slave synchronization delays, or transactions writing across files, log events may be split across multiple files. In such cases, manually locating the offset of the recovery starting point is extremely prone to error; even an error within tens of bytes can lead to subsequent event parsing failures or duplicate imports, ultimately compromising data consistency. Second, existing methods often use timestamps as the primary reference point to locate import breakpoints, but timestamps may have time zone offsets or microsecond-level precision loss at different nodes, making a strict one-to-one correspondence impossible. Furthermore, existing incremental recovery processes mostly rely on static parameter configurations, such as continuing import at a fixed `end_log_pos` offset. This approach lacks a dynamic verification mechanism; if the import fails midway, it is difficult to quickly backtrack and re-determine the breakpoint position. In practice, operations and maintenance personnel often need to repeatedly try and fail: repeatedly adjusting the starting position parameters, re-importing logs, and then observing whether errors occur. This is not only inefficient, but may also place an additional burden on the target database during multiple failures. Especially in environments with huge log volumes (tens of GB) or intensive transactions, this trial-and-error process is almost unacceptable. Summary of the Invention

[0004] The main objective of this invention is to provide a method for quickly locating the failure point of incremental recovery import in MySQL binary logs. By establishing a co-occurrence detection mechanism between the binary log text files on the target and source ends, combining a finite state machine to accurately parse the `end_log_pos` value, and introducing a three-anchor checksum consisting of CRC32 checksum and event header timestamp, high-precision confirmation of the recovery breakpoint location is achieved. When the initial check fails, the method can automatically find stable pairs using bidirectional neighborhood search, thereby solving the problem of difficulty in locating the breakpoint caused by cross-file transactions or log differences. Furthermore, the controller supports automatic resumption and rollback retry based on stable pairs during the recovery process, making the recovery process highly robust and adaptive. Compared with existing methods that rely on manual experience and fixed parameters, this invention significantly improves the automation and accuracy of incremental recovery, reduces data loss or duplicate imports caused by inaccurate breakpoint location, effectively shortens recovery time in large-scale database environments, improves system consistency and stability, and has strong engineering application value.

[0005] To solve the above problems, the technical solution of the present invention is implemented as follows: A method for quickly locating the failure point of incremental recovery import of MySQL binary logs, including: Step 1: Execute the flush-logs command on the target database server to generate a latest binary log file for the target side. Obtain the latest binary log file for the target side and an imported binary log file for the source side. Convert the two binary log files into two line-by-line readable text files respectively, resulting in the target side binary log text file and the source side binary log text file. During the conversion process, the relative order of the event line boundaries, end_log_pos, and CRC32 checksum remains unchanged. Step 2: A co-occurrence detector is used to scan the target binary log text file line by line to identify co-occurrences that simultaneously contain both end_log_pos and CRC32 checksums. The identified co-occurrences are counted to obtain a target co-occurrence count value. The co-occurrence detector is used to locate candidate co-occurrences in the source binary log text file that correspond to the target co-occurrence count value. A finite state machine is used to parse the end_log_pos value from the candidate co-occurrences and confirm the presence of a CRC32 checksum in that line. Three-anchor verification is performed. If the verification passes, the parsed end_log_pos value is determined as the recovery breakpoint location. If the verification fails, a bidirectional neighborhood search is performed centered on the candidate co-occurrence line. Within a preset window, co-occurrence pairs between the source and target ends are enumerated and three-anchor verification is performed until a stable pair is obtained. The end_log_pos value parsed from the source co-occurrence in the stable pair is determined as the final recovery breakpoint location. Step 3: A controller generates incremental recovery parameters based on the final recovery breakpoint position. Starting from the next event corresponding to the recovery breakpoint position, it imports subsequent binary logs in ascending order of file number. If the import fails, it retryes based on the most recent stable pair. If the import succeeds, it records the recovery breakpoint position and the line number mapping that is currently present on both the source and target ends.

[0006] Further, step one specifically includes: executing the flush-logs command on the target database server to lock the current binary log file and create a new binary log file, using the locked binary log file as the latest binary log file on the target side; selecting a source binary log file that has already been imported; converting the latest binary log file on the target side and the source binary log file into target binary log text files and source binary log text files respectively; during the conversion, keeping the line boundaries of events unchanged, unifying the character encoding to UTF-8, unifying the newline character to a single newline character, and preserving the original spelling and relative order of end_log_pos and CRC32 checksum.

[0007] Furthermore, step one also includes text preprocessing of the target binary log text file and the source binary log text file. The preprocessing includes: decoding each line of any binary log text file into a string starting from the first line; if decoding a single line fails, it is replaced with a preset replacement character and the subsequent lines are processed; deleting whitespace characters at the beginning and end of each line; converting all English letters in each line to lowercase; and retaining a copy of the original line that has not been preprocessed for each preprocessed line.

[0008] Furthermore, the process of scanning the target binary log text file line by line using a co-occurrence detector in step two includes: for each line of the target binary log text file, performing line-by-line matching using the preset keywords end_log_pos and CRC32 checksum; if both end_log_pos and CRC32 checksum are matched in the current line, the current line is identified as a co-occurrence line, and a target co-occurrence counter is incremented by one; after completing the scanning of all lines of the target binary log text file, the final value of the target co-occurrence counter is recorded as the target co-occurrence count value.

[0009] Furthermore, the process of locating candidate co-occurrences in the source binary log text file in step two includes: scanning the source binary log text file line by line and using a co-occurrence detector to identify co-occurrences; incrementing a source co-occurrence counter by one for each identified co-occurrence; stopping the scanning when the value of the source co-occurrence counter is equal to the target co-occurrence count value for the first time, and determining the currently identified co-occurrence as a candidate co-occurrence.

[0010] Furthermore, step two involves using a finite state machine to parse the end_log_pos value within the candidate coherent line, which includes: initializing the finite state machine to an initial state; scanning character by character starting from the beginning of the candidate coherent line, and when the end_log_pos keyword is detected, transitioning the finite state machine to a bootstrapping state; starting from the bootstrapping state, skipping non-numeric characters, and when the first numeric character is detected, transitioning the finite state machine to a data acquisition state; in the data acquisition state, continuously reading subsequent numeric characters and combining them into a decimal integer until a non-numeric character is encountered, and using this decimal integer as the end_log_pos value; after completing the acquisition of the end_log_pos value, transitioning the finite state machine to a verification state, in which the presence of the CRC32 checksum keyword is confirmed in the remaining part of the candidate coherent line.

[0011] Furthermore, the three-anchor verification in step two includes: defining the first anchor as the existence of the end_log_pos value in the co-current; defining the second anchor as the existence of the CRC32 checksum in the co-current; and defining the third anchor as the timestamp in the event header associated with the co-current. The event header is identified using a preset rule, which requires that the line containing the event header simultaneously contains both serverid and end_log_pos, and that a timestamp exists within a preset neighborhood of that line. For a set of source and target co-currents, each is checked to see if it meets the conditions of the first and second anchors, and the event header is searched within its neighborhood to extract the timestamp. When both source and target co-currents meet the conditions of the first and second anchors, and the extracted timestamps are identical, the three-anchor verification is considered successful.

[0012] Furthermore, the bidirectional neighborhood search in step two includes: defining a search window centered on the corresponding line in the source and target binary log text files of the candidate co-occurrences; within the search window, combining co-occurrences from the source and target ends according to a preset enumeration strategy to form co-occurrence pairs to be verified; performing a three-anchor verification for each co-occurrence pair to be verified; when a co-occurrence pair passes the three-anchor verification for the first time, the co-occurrence pair is determined as a stable pair, and the bidirectional neighborhood search is terminated; if all co-occurrence pairs in the search window fail the three-anchor verification, the scope of the search window is expanded and the aforementioned enumeration and verification steps are repeated; if no stable pair is found before reaching the file boundary in any binary log text file, the next co-occurrence in the source binary log text file is selected as a new candidate co-occurrence, and the bidirectional neighborhood search is re-executed.

[0013] Furthermore, step three specifically includes: the controller generates incremental recovery parameters based on the final recovery breakpoint position, which specifies that incremental import should begin from the event after the recovery breakpoint position; if incremental import fails, the controller performs a rollback retry based on the most recently found stable pairing position; if incremental import succeeds, the controller persistently stores the final recovery breakpoint position, the corresponding current row number of the source and target ends, and the result of the three anchor point verification.

[0014] The method for quickly locating the failed import position of incremental recovery of MySQL binary logs in this invention has the following beneficial effects: it effectively solves the problems of difficulty in automatically determining recovery breakpoints, import failures caused by cross-file transactions, and low efficiency of repeated manual trial and error in existing technologies. This invention introduces a co-occurrence detection mechanism between the target and source binary log text files, so that the process of locating recovery breakpoints no longer relies on manual judgment of timestamps or fixed offsets, thereby improving the automation level of the location. Furthermore, this invention uses a finite state machine to parse the end_log_pos value and combines it with a CRC32 checksum and an event header timestamp to form a three-anchor verification mechanism, achieving multi-dimensional consistency confirmation and maintaining high robustness even in scenarios with slight differences in log format or cross-file transactions. When the verification fails, this invention also designs a bidirectional neighborhood search strategy, which automatically enumerates co-occurrence pairs between the source and target ends within a preset range, gradually expanding the search window until a stable pair satisfying the three-anchor condition is found, greatly improving the reliability of breakpoint location. Meanwhile, the controller of this invention features automatic resume and rollback retry functions during the import process. If import fails, it can quickly roll back and re-import based on the most recent stable pair, avoiding complex manual intervention and reducing the risk of data loss or duplicate imports. In summary, this invention can significantly shorten recovery time in large-scale database environments, improve the consistency and stability of incremental recovery, and meet the high availability and data consistency requirements of industries such as finance and e-commerce, demonstrating significant engineering application value and promotional significance. Attached Figure Description

[0015] Figure 1 A flowchart illustrating the method for quickly locating the location of MySQL binary log incremental recovery import failure provided in this embodiment of the invention; Figure 2 This is a schematic diagram illustrating the performance characteristics of the co-occurrence detector as a function of binary log file complexity, provided in an embodiment of the present invention. Figure 3 This diagram illustrates the convergence and window parameter optimization characteristics of the bidirectional neighborhood search algorithm provided in this embodiment of the invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0017] Suppose that after executing `flush-logs`, the target end obtains the latest binary log file `mysql-bin.000895`, and the last imported binary log file on the source end is `mysql-bin.000894`. Both binary log files have been converted into target-end binary log text files and source-end binary log text files according to step one. The conversion preserves the relative order of event line boundaries, `end_log_pos`, and CRC32 checksums. Character encoding is uniformly set to UTF-8, and newline characters are uniformly set to a single newline character. This embodiment demonstrates the entire process from target-end co-occurrence detection, source-end candidate co-occurrence location, finite state machine parsing, three-anchor point verification, bidirectional neighborhood search to final recovery breakpoint location determination, and import and rollback retries from the next event.

[0018] Obtain the target binary log text file co-occurrence count: For clarity, several lines from the target binary log text file relevant to this alignment are extracted (only key lines are shown; the actual file contains more context lines), with line numbers generated during the transformation: T#690:#at521600 T#691:24091815:42:08serverid100end_log_pos521760CRC320x9a7b2c11 T#692:SETTIMESTAMP=1695051728; T#693:#at521760 T#694:24091815:42:09serverid100end_log_pos521972CRC320xa2b3cc90 T#695:SETTIMESTAMP=1695051729; T#696:#at521972 T#697:24091815:42:10serverid100end_log_pos522112CRC320x5f3a1cde T#698:SETTIMESTAMP=1695051730; T#699:#at522112 T#700:24091815:42:11serverid100end_log_pos522368CRC320x8e71a2b3 T#701:SETTIMESTAMP=1695051731; T#702:#at522368 T#703:24091815:42:12serverid100end_log_pos522640CRC320x1c2d3e4f T#704:SETTIMESTAMP=1695051732; The co-occurrence detector identifies lines that simultaneously contain both end_log_pos and a CRC32 checksum (e.g., T#691, T#694, T#697, T#700, T#703). Note: The first binary log text file on the target end Line text. : The total number of lines in the target binary log text file. :like If it contains both end_log_pos and CRC32 checksum, then ,otherwise . Target co-occurrence count value. Scanning and counting by row yields: In the shown segment, T#691, T#694, T#697, T#700, and T#703 all satisfy the co-occurrence condition. If two other co-occurrences previously appeared within the entire scope of this document, the specific values ​​obtained at the end of this statistical analysis are: ;here This means that the target binary log text file contains a total of 7 occurrences of the corresponding common occurrence from the first line to the current line. Locating candidate common occurrences and parsing the end_log_pos value in the source binary log text file: The relevant segment in the source binary log text file (corresponding to mysql-bin.000894) is illustrated below: S#730:#at521560 S#731:24091815:42:08serverid101end_log_pos521720CRC320x9a7b2c11 S#732:SETTIMESTAMP=1695051728; S#733:#at521720 S#734:24091815:42:09serverid101end_log_pos521932CRC320xa2b3cc90 S#735:SETTIMESTAMP=1695051729; S#736:#at521932 S#737:24091815:42:10serverid101end_log_pos522112CRC320x5f3a1cde S#738:SETTIMESTAMP=1695051730; S#739:#at522112 S#740:24091815:42:11serverid101end_log_pos522240CRC320xff11aa22 S#741:SETTIMESTAMP=1695051731; S#742:#at522240 S#743:24091815:42:11serverid101end_log_pos522368CRC320x8e71a2b3 S#744:SETTIMESTAMP=1695051731; S#745:#at522368 S#746:24091815:42:12serverid101end_log_pos522640CRC320x1c2d3e4f S#747:SETTIMESTAMP=1695051732; Same definition: The first binary log text file from the source end Line text. : The total number of lines in the source binary log text file. :like If it contains both end_log_pos and CRC32 checksum, then ,otherwise . :satisfy The smallest line number is used to locate candidate co-occurrences. The source file is scanned line by line, and the source co-occurrence counter is incremented by one for each co-occurrence found; when the source co-occurrence counter first equals... When the time stops, the currently identified row is the candidate co-occurrence. Based on the above fragment, assuming the 7th co-occurrence first appears at S#737 (illustrated), then: candidate co-occurrence Within the candidate co-occurrence, the finite state machine resolves `end_log_pos`: starting from the keyword `end_log_pos`, skipping whitespace and optional separators, it collects consecutive digits to obtain a decimal integer. Note: The end_log_pos value (candidate value) obtained from parsing within the candidate co-current context. For line S#737: S#737:24091815:42:10serverid101end_log_pos522112CRC320x5f3a1cde; the finite state machine acquisition results are: .

[0019] Definition of three-anchor point verification: The candidate currently contains a resolvable end_log_pos value. The candidate currently contains a CRC32 checksum keyword, and a segment of hexadecimal characters (at least 8 characters in length) can be read after it. : The timestamp text of the current associated event header of the candidate source (extracted from this line or no more than 3 lines thereafter). : The timestamp text of the corresponding current associated event header on the target end (extracted from this line or no more than 3 subsequent lines). Near candidate line S#737: the candidate line meets the following conditions. and The source timestamp text is extracted from S#738: Near the coexisting T#700 in the target region and candidate region: the target region's timestamp text is extracted from T#701. ;Compare: Therefore, the three-anchor-point verification failed, triggering a bidirectional neighborhood search.

[0020] Perform a bidirectional neighborhood search using candidate co-occurrence behavior centers and obtain stable pairings: Define the bidirectional neighborhood search as follows: : Search window radius, representing the maximum number of rows included both upwards and downwards from the center row. This example initially retrieves... . : In absolute row difference The set of co-occurrence pairs that need to be checked within the layer is enumerated in order of proximity to the center. Stable pairing: A co-occurrence pair between the source and target ends that simultaneously passes the three anchor point checks. Search process overview: Centered on the candidate co-occurrence S#737 at the source end and the candidate region T#700 at the target end, first check... The combination within, check again. The combinations within, increasing sequentially. Until Stop when a stable pairing is found.

[0021] examine Source S#740 and target T#700: S#740 line: 24091815:42:11serverid101end_log_pos522240CRC320xff11aa22; parsed to obtain Its timestamp text is taken from S#741: The timestamp text in line T#700 on the target end is taken from T#701: Three-anchor point determination: If the end_log_pos and CRC32 checksum are true in the same row, and... The condition is met. The three-anchor point check is now satisfied. To improve robustness, another pair of combinations closer to the center on the same layer, S#743 and T#700, is also checked (still belonging to...). Another pair under the "nearest center priority" order): S#743:24091815:42:11serverid101end_log_pos522368CRC320x8e71a2b3; parsed to obtain Its timestamp text is taken from S#744: The target timestamp text is consistent with that of T#700 / T#701, and the CRC32 checksum is the same as that in the target T#700 line, which is 0x8e71a2b3. This pair shows stronger consistency in terms of field values ​​in addition to the three anchor points, so the latter is given priority as a stable pair.

[0022] Therefore, the stable pairing is determined as follows: Source end common current: S#743 (end_log_pos522368, CRC32 checksum 0x8e71a2b3, timestamp text "24091815:42:11"). Target end common current: T#700 (end_log_pos522368, CRC32 checksum 0x8e71a2b3, timestamp text "24091815:42:11"). The end_log_pos value parsed from the source end common current in the stable pairing is determined as the final recovery breakpoint position. Note: : The final breakpoint location (stable value). Explanation: S#743 was chosen over S#740 within the layer because the CRC32 checksums on both sides are completely consistent and the timestamp text is consistent, making the row-level correspondence more certain; based on the strategy of prioritizing closer to the center, S#743 has the best alignment with S#700, reducing the need to continue searching.

[0023] Will Applied to incremental recovery parameter generation and import: Starting file and starting offset: The starting file is located at the source mysql-bin.000894. The starting offset is set to... Since `end_log_pos` marks the end boundary of the current event, we continue parsing forward from this position to the next event header, thus obtaining the import starting point. Let the starting position of the parsed next event header be: : The starting position within the file for the next event header. (In this example, the text displays "#at522368", indicating that the next event indeed begins from here.) This is from mysql-bin.000894. Begin importing subsequent events and the subsequent file mysql-bin.000895 in ascending file sequence. During import, commit events grouped together when transaction start and end markers are encountered to ensure consistent transaction boundaries. During the import process, for each event commit, record the start position of the event header and the timestamp text; for example, if three events are committed sequentially in mysql-bin.000894, their event header start positions will be 522368, 522640, and 523040 respectively, with corresponding timestamp texts of "24091815:42:12", "24091815:42:13", and "24091815:42:15". Note: : No. The starting position of the event header for a successfully submitted event. : No. A timestamp text indicating a successfully submitted event. Example record: Suppose that during the import of mysql-bin.000895, after parsing event header position 120144, the target database server returns a constraint conflict error. The controller immediately pauses the import and retryes based on the most recent stable pair. The rollback point selection: the most recent stable pair remains S#743 and T#700, corresponding to... Rewind to the next event header at that position, and start again from... The import process begins. The reason for rolling back to this position is that it has passed three-anchor point verification and field consistency checks, minimizing the probability of mismatches and avoiding duplicate imports caused by returning to uncertain historical positions. Re-import after verification: After rolling back, the controller performs consistency checks on fields near the rollback point: whether end_log_pos is 522368, whether the CRC32 checksum is 0x8e71a2b3, and whether the timestamp text is "24091815:42:11". If all three are consistent, replay continues. If the re-import fails again, the rollback strategy remains unchanged, and the same stable pair is used as the baseline rollback point for retrying until the import succeeds or structural conflicts are manually addressed.

[0024] After successful import, the controller records the following data for subsequent batches to be directly reset and audited: Restore breakpoint position: The current line number mapping between the source and target is: S#743↔T#700. Import range: [522368, end of file] in mysql-bin.000894 and [beginning of file, end of file] in mysql-bin.000895. Key snapshot: A list for quick resumption of transmission after an external interruption. Obtain the co-occurrence count value at the target end. Locate and analyze candidate coexisting entities at the source end. After the three-anchor-point verification failed, a stable pair S#743↔T#700 was obtained through bidirectional neighborhood search, and the breakpoint location was finally determined. It imports subsequent binary logs in ascending order of file number, starting from the next event corresponding to the recovery breakpoint. In case of failure, it retryes based on the most recent stable pair. After success, it records the recovery breakpoint position and line number mapping for use in subsequent batch resets and audits.

[0025] refer to Figure 1 A method for quickly locating the location of MySQL binary log incremental recovery import failure, which includes: Step 1: Execute the flush-logs command on the target database server to generate a latest binary log file for the target side. Obtain the latest binary log file for the target side and an imported binary log file for the source side. Convert the two binary log files into two line-by-line readable text files respectively, resulting in the target side binary log text file and the source side binary log text file. During the conversion process, the relative order of the event line boundaries, end_log_pos, and CRC32 checksum remains unchanged.

[0026] In one implementation, a session with permissions to execute the `flush-logs` command and read binary log files is established on the target database server. The `flush-logs` command is triggered, immediately closing the currently being written binary log file and generating a new file in the same directory, which serves as the latest binary log file on the target side. This fixes the log segmentation point on the target side at time boundaries, preventing subsequent conversion stages from being affected by concurrent writes, thus ensuring the stability of text content during subsequent matching and counting. The precise filename and path of the latest binary log file on the target side are obtained by executing a command to query the current binary log status, avoiding positioning errors caused by filename increment rules or time zone differences. Based on the progress record of the recovery process, an imported source binary log file is located. This file was already imported in the previous round of incremental import; selecting this file as a reference allows the subsequently generated target binary log text file and the source binary log text file to form comparable segments on the event timeline, thereby making the line count of the co-occurrence detector monotonic and reproducible.

[0027] The goal of the transformation is to losslessly convert the binary log content into line-by-line readable text content, resulting in target and source binary log text files. The transformation follows the event line boundary preservation rule: the header information of each event is output line by line, maintaining the natural separation between events and avoiding merging adjacent event content into the same line. This approach allows the co-occurrence detector to perform counting at the event granularity with a single linear scan, eliminating the need for cross-event backtracking and maintaining linear complexity and stable memory usage even with large files. The transformation also follows the identifier relative order preservation rule: ensuring that `end_log_pos` and the CRC32 checksum are presented in the original order of appearance within the same event header line. Maintaining relative order reduces the complexity of subsequent finite state machine branch decisions, decreases the possibility of ambiguous matching, and improves the consistency of breakpoint resolution. When generating the target and source binary log text files, a fixed-size buffer is used for block writing. Line integrity checks are performed at each block boundary. If a half-line of an event header is detected at the end of a block, this half-line is retained in the header of the next block and continued until a complete line is formed before writing it out. This method can reduce peak memory usage in high-concurrency and large-file scenarios, and ensure that the event header line is not corrupted by block splitting.

[0028] The official binary log decoding tool is used to stream the binary log file into a text line stream. To ensure that the event header contains end_log_pos and the CRC32 checksum, the option to output key fields in the event header and line-level time information is enabled during the call. This ensures that each event header in the text appears in a parsable, fixed token form, facilitating subsequent parsing and verification. A line-by-line standardization process is established, processing the decoded text in the following order: uniform character encoding to UTF-8, uniform newline character to a single newline character, and removal of byte order markers to avoid the impact of line break differences from different operating systems on line counting. Line breaks in the event header are standardized: if the tool output is broken into multiple lines due to terminal line breaks or line width limitations, adjacent lines of the event header are combined into a single line in the original left-to-right reading order, without crossing events, ensuring that end_log_pos and the CRC32 checksum are on the same line. This eliminates line break noise at the display level without changing the line boundaries between events. Non-event header content is output line by line as is, without changing its relative position to adjacent event header lines. This allows for the restoration of more context when needed without interfering with the co-occurrence detector's judgment, as the detector only counts lines containing `end_log_pos` and a CRC32 checksum. A relative order check is performed before writing each line: if both `end_log_pos` and a CRC32 checksum appear in the same line, the word order in that line is verified to match the order in the input text; if line breaks and merging introduce a risk of misordering, the input segments are concatenated in their original order to ensure the relative order remains unchanged. The normalized text stream is written to stable storage, generating target-side and source-side binary log text files. Each text file undergoes a line count and keyword coverage check after writing to confirm the presence of `end_log_pos` at least in the event header line and, in scenarios where checksum output is enabled, a CRC32 checksum. This check helps detect tool option configuration errors or log corruption early, allowing for rapid failure and correction before entering the co-occurrence detection phase.

[0029] Executing the `flush-logs` command before retrieving the latest binary log file from the target end creates a clear boundary at the moment of log switching. This ensures that the last complete event in the target binary log file forms a stable, comparable range with the imported range in the source binary log file, reducing the probability of event truncation due to concurrent writes. Consolidating the event header into a single line while maintaining the relative order of `end_log_pos` and the CRC32 checksum allows the co-occurrence detector to perform co-occurrence judgment within a single line, and the finite state machine to extract `end_log_pos` and confirm the existence of the CRC32 checksum within a single line, avoiding cross-line state propagation and improving the determinism and speed of parsing. Using unified character encoding and newline characters helps maintain consistent line numbers and matching results across different platforms and file systems, preventing inconsistent co-occurrence counts for the same text in different environments.

[0030] Without storing intermediate text, the binary log file is directly piped into the decoding tool. The decoded output enters the line-by-line normalization process in real time and is directly written to both the target and source binary log text files. This method is suitable for scenarios with large file sizes or limited storage space, reducing disk usage and input / output latency. In scenarios with read-only slave databases, the `flush-logs` command is executed on the read-only slave database to obtain the latest binary log file from the target instance for conversion, avoiding interference with the log write path of the master instance. The target binary log text file obtained from the read-only slave database still meets the requirements for maintaining event line boundaries and relative order, and can be directly used for subsequent co-occurrence detection and breakpoint parsing. 5.4 Strict Order Preservation Method: When the event header lines output by the original tool have inconsistent field order or missing optional fields, the fields are not rearranged or supplemented; only line wrapping and encoding line break normalization are performed to maintain the relative order in the input. This ensures consistency with the assumptions of the subsequent co-occurrence detector and finite state machine, avoiding matching ambiguities caused by arbitrarily rewriting the field order.

[0031] No write operations are performed on the latest binary log file on the target end between executing the `flush-logs` command and starting the conversion. This prevents tail writes due to buffer delays from occurring during the conversion, thus affecting the stability of event line boundaries. After generating the target and source binary log text files, the file generation time, total number of lines, and number of lines containing `end_log_pos` and CRC32 checksums are recorded in the recovery process's audit log to quickly assess input quality before proceeding to subsequent steps. When version differences in the decoding tool cause inconsistencies in whether or not to output CRC32 checksums, ensure the presence of a CRC32 checksum field in the text by enabling the option to print checksums or performing the conversion in an environment that supports checksums. If the target environment does not support outputting CRC32 checksums, environmental corrections should be performed before proceeding to subsequent steps to prevent the co-occurrence detector from failing to generate valid counts.

[0032] Step 2: A co-occurrence detector is used to scan the target binary log text file line by line to identify co-occurrences that simultaneously contain both end_log_pos and CRC32 checksums. The identified co-occurrences are counted to obtain a target co-occurrence count value. The co-occurrence detector is used to locate candidate co-occurrences in the source binary log text file that correspond to the target co-occurrence count value. A finite state machine is used to parse the end_log_pos value from the candidate co-occurrences and confirm the presence of a CRC32 checksum in that line. A three-anchor check is performed. If the check passes, the parsed end_log_pos value is determined as the recovery breakpoint location. If the check fails, a bidirectional neighborhood search is performed centered on the candidate co-occurrence line. Within a preset window, co-occurrence pairs between the source and target ends are enumerated and three-anchor checks are performed until a stable pair is obtained. The end_log_pos value parsed from the source co-occurrence in the stable pair is determined as the final recovery breakpoint location.

[0033] In one implementation, the input consists of a target binary log text file and a source binary log text file. Both have been generated according to step one, ensuring that the relative order of event line boundaries, end_log_pos, and CRC32 checksums remains unchanged. The co-occurrence detector processes the text files line by line, detecting whether end_log_pos and CRC32 checksums appear simultaneously within a single line. This line-by-line detection avoids misjudgments caused by cross-line state propagation, ensuring that the identification result depends only on the explicit character content of the current line, thus maintaining stability even when the log contains line breaks, comments, or changes to optional fields. The target binary log text file is inspected line by line, and the identified co-occurrences are counted to obtain the target co-occurrence count. Subsequently, the same detection strategy is used to inspect and count the source binary log text file line by line. When the source count first equals the target co-occurrence count, the current line is identified as a candidate co-occurrence. By aligning with line counts in this way, a stable mapping can be achieved without relying on line numbers or timestamp order. This is because the common occurrence only appears in the event header containing key fields, and its distribution in the logs at both ends has comparable consistency.

[0034] The co-occurrence detector's line-by-line scanning and co-occurrence identification process includes: reading a line of text, preserving the entire line as an immutable string, and obtaining a view solely for matching. This allows for direct referencing of the original line when specific segments need to be traced back later, avoiding repeated parsing. Two keyword matches are performed within this line, targeting `end_log_pos` and the CRC32 checksum, respectively. The matching strategy combines character-by-character prefix progression with partial backtracking: starting from the beginning of the line, each keyword is progressed character-by-character, backtracking to the longest known prefix position of the keyword when an inconsistent character is encountered, continuing until the keyword appears completely or the line ends. The same progression process is then performed for the other keyword. When both keywords appear on the same line, the line is considered a co-occurrence. If a line contains multiple `end_log_pos` occurrences or multiple CRC32 checksums, it is still counted as only one co-occurrence, because the goal of co-occurrence detection is to confirm the existence of the same line, not the number of occurrences. The above process is repeated for each line of the target binary log text file, incrementing the target co-occurrence counter by one for each co-occurrence identified. At the end of the file scan, the final value of the co-occurrence counter at the target end is recorded as the target end co-occurrence count value. This provides a robust line-by-line metric without parsing numerical details, which can be used for subsequent sequential location at the source end. The same line-by-line scan and co-occurrence identification are performed on the source end binary log text file. For each line of co-occurrence identified, the source end co-occurrence counter is incremented by one. When the source end co-occurrence counter value first equals the target end co-occurrence count value, the scan stops immediately, and the current line is identified as a candidate co-occurrence. Using the first-equal-stop criterion ensures that the candidate co-occurrence and the target end co-occurrence are aligned in statistical order, reducing offsets caused by insert events or differences in optional fields.

[0035] The finite state machine parses the `end_log_pos` value within a candidate common line and confirms the existence of the CRC32 checksum. Specifically, the finite state machine is designed to reliably extract the decimal integer immediately following `end_log_pos` within a single line and confirm the existence of a CRC32 checksum within the same line. Single-line parsing avoids ambiguity caused by reading across lines and allows the implementation to work as a character stream, thus adapting to different whitespace and punctuation styles. For the initial state: Scan character by character starting from the beginning of the line, searching for the complete `end_log_pos` keyword. Upon detecting the keyword, enter the bootstrapping state. If the keyword is not detected until the end of the line, the line is determined not to meet the parsing conditions, and a new candidate common line needs to be located. For the bootstrapping state: Continue scanning after the `end_log_pos` keyword, skipping spaces, tabs, colons, or equal signs in sequence until the first numeric character is encountered. This accommodates formatting characters inserted between keywords and values ​​by different tools, while ensuring that the final captured character sequence begins with a number. For the acquisition state: Read numeric characters continuously starting from the first numeric character, treating the read numeric sequence as a decimal integer. The data acquisition ends upon encountering the first non-numeric character, and the decimal integer is latched as the `end_log_pos` value. If no numeric characters are read, the line is marked as a failed parsing line, requiring re-locating candidate common lines. For the verification status: continue scanning within the same line to confirm the existence of a complete CRC32 checksum keyword. If present, a hexadecimal character segment with a length of at least 8 characters can be read for consistency checks in subsequent steps; if the hexadecimal segment is missing or invalid, the existence of the CRC32 checksum keyword is the only passing condition. The existence of the verification status prevents misjudgment due to values ​​unrelated to `end_log_pos` within the line, and provides usable information for subsequent three-anchor verification. For submission: after the verification status passes, the latched `end_log_pos` value is output as the parsing result, and the current line number, the start and end positions of the `end_log_pos` value within the line, and the position of the CRC32 checksum keyword within the line are recorded together as context for subsequent steps. Recording these positions facilitates tracking and backtracking. When subsequent validation fails or neighborhood searches are needed, the specific character range can be located directly without rescanning the entire line. The reason for using a finite state machine for character-by-character parsing instead of direct line-by-line matching is that character-by-character parsing is more tolerant of variations in input formatting. For example, some tools may insert different numbers of spaces between keywords and numbers, or change the separator from a colon to an equals sign. Character-by-character parsing can absorb these variations without sacrificing determinism, reducing false rejections and improving portability.

[0036] When no co-occurrence is detected in the target binary log text file, the exception is recorded and the current process is terminated, prompting a check of whether the conversion option in step one includes output containing end_log_pos and CRC32 checksum. Continuing in this situation will not yield a valid target co-occurrence count. If the source binary log text file does not show a co-occurrence count equal to the target count before the scan is complete, it indicates a structural difference in the selected comparison file between the two logs that is not aligned. At this point, the largest identified source co-occurrence count and its corresponding line number are recorded for neighbor search or backtracking strategies in subsequent steps. When a candidate co-occurrence fails to parse the end_log_pos value or confirm the existence of the CRC32 checksum during the finite state machine stage, the line is marked as a parsing failure line, and the next co-occurrence is scanned from the source binary log text file as a new candidate co-occurrence, repeating the finite state machine parsing. This single-line replacement retry approach avoids full file backtracking while maintaining line alignment. To reduce the impact of occasional character corruption on the results, the maximum number of characters to skip between the bootstrap state and the acquisition state of the finite state machine is limited, for example, no more than 32 non-numeric characters. Limiting the skipping range can prevent erroneous acquisition caused by skipping to the next field, while also accommodating reasonable variations in separator length in the output of different tools.

[0037] Using common-current counts to establish line alignment between the two ends avoids directly relying on file line numbers or timestamp sorting. When logs contain deleted comments, added optional fields, or blank lines inserted by different export tools, file line numbers are unstable. However, the relative distribution of event header lines containing `end_log_pos` and CRC32 checksums is more stable at both ends, making common-current counts a more reliable alignment basis. Restricting the parsing of `end_log_pos` values ​​to candidate common-current counts ensures that the extracted values ​​indeed come from event headers containing key fields, avoiding the extraction of similar numbers from unstructured description lines, thus improving the accuracy of subsequent breakpoint calculations. Retaining the hexadecimal fragment following the CRC32 checksum keyword provides information for subsequent consistency checks. When content offsets occur near the same event in the logs at both ends, the presence or absence of the CRC32 checksum and its readability help determine whether the candidate common-current count truly corresponds, thereby reducing false matches.

[0038] In the co-occurrence detector, complete matching templates are pre-set for end_log_pos and CRC32 checksums respectively. First, it's determined whether both templates in the same line match simultaneously. If a match is found, the line is identified as a co-occurrence. In the finite state machine stage, the end-position template is used only once to extract decimal integers from candidate co-occurrences, while the checksum template is used to confirm the existence of the CRC32 checksum keyword. This method is simple to implement and easy to migrate across languages, but backtracking control is needed for extremely long lines. Within the same line, two pointers advance simultaneously: the left pointer focuses on finding end_log_pos, and the right pointer focuses on finding the CRC32 checksum. A match is recorded when either pointer reaches the matching endpoint; if both match, it's considered a co-occurrence. In the finite state machine stage, guidance and data collection begin from the left-hand match position, and the keyword is confirmed near the right-hand match position. This method allows parallel processing of two keywords in the same line in a multi-core environment, improving the parsing speed of long lines. For extremely large text files, a fixed-size buffer is read and scanned line by line within the buffer. When a half-line appears at the end of the buffer, it is concatenated with the beginning of the next buffer before matching, ensuring the integrity of a single line. This approach can run stably in memory-constrained environments while maintaining the correctness of co-occurrence detection and finite state machine resolution.

[0039] The process involves retrieving a single co-occurrence from the target binary log text file, a candidate co-occurrence from the source binary log text file, the `end_log_pos` value parsed from the candidate co-occurrence, and the CRC32 checksum confirmed within the candidate co-occurrence. The three anchor points consist of the following: the first anchor point is the `end_log_pos` value within the candidate co-occurrence; the second anchor point is the CRC32 checksum within the candidate co-occurrence; and the third anchor point is the timestamp text of the event header associated with the respective main lines on both sides. By simultaneously checking these three items, a consistent cross-file match can be established using the structural identifier and timestamp of the same event, without relying on line numbers or external historical information. The first anchor point limits the parsing of the target's originating location field, the second anchor point limits the line to the event header context containing checksum information, and the third anchor point is used to verify the consistency of the event occurrence order across sources, thereby reducing the risk of mismatches caused by relying solely on text similarity.

[0040] At the common occurrences in the target binary log text file, the event header is searched both upwards and downwards. The event header is identified according to preset rules: the line containing the event header must contain both `serverid` and `end_log_pos`, and a timestamp must exist within a preset neighboring range of that line. A preferred setting for the preset neighboring range is the current line and no more than three lines following it. The reason for choosing no more than three lines is that mainstream export tools output the timestamp immediately after the event header; timestamps appearing beyond this range usually belong to the next event or contextual comments and should not be included in the time identifier of the same event. After identifying the event header, the timestamp text is extracted in the following order: first, obvious timestamp identifier fragments, such as `settimestamp` or `ts`, are searched within the event header line; then, line by line, a complete time expression consisting of numbers and separators is searched within the preset neighboring range. If multiple time expressions appear simultaneously, the most complete expression containing year, month, day, hour, minute, and second is selected first. The above identification and extraction process is repeated at the candidate common occurrences in the source binary log text file to obtain the source timestamp text. To avoid misjudgments caused by mere format differences, the timestamp text on both sides is standardized before comparison: extra spaces are removed, and the text is unified into a fixed order of year, month, day, hour, minute, and second, while retaining numbers and necessary separators. Maintaining necessary separators helps distinguish different fields and avoids ambiguity caused by stringing different time fields together as a single number.

[0041] Verify the first anchor point: Confirm that the `end_log_pos` value has been parsed from the candidate coherent line and that this value consists of consecutive digits. This confirmation ensures that the recovered breakpoint location comes from the end position field, rather than other erroneously collected numbers. Verify the second anchor point: Confirm that a CRC32 checksum keyword exists within the candidate coherent line, and then read a hexadecimal fragment with a length of at least 8 bytes. This confirmation proves that the line contains checksum information, indicating it belongs to the event header range. Even if the hexadecimal fragments differ in length or case, the presence of the keyword is sufficient to satisfy the second anchor point, ensuring compatibility with the output styles of different tools. Verify the third anchor point: Perform a complete consistency comparison between the timestamp text of the target end and the timestamp text of the source end. The advantage of a complete consistency comparison is that it avoids mistaking similar times for the same event, thus maintaining the certainty of event-level alignment. When the timestamp texts on both sides are completely consistent, and both the first and second anchor points are satisfied, the three-anchor point verification is considered successful. When the three anchor points pass the verification, the end_log_pos value parsed from the candidate common line is directly determined as the breakpoint location for recovery, and the line numbers and timestamp texts on both sides used for comparison are recorded for subsequent auditing and rollback verification. If any anchor point fails to meet the requirements, the verification is deemed to have failed, providing a trigger condition for entering the bidirectional neighborhood search.

[0042] Centered on the candidate co-occurrence lines from the source end, a symmetrical row window is created in both the source and target binary log text files. A preferred setting is 16 lines on each side, forming a pre-defined window with 16 lines above and below the central line. The choice of 16 lines is based on the fact that event headers typically have high density, and 16 lines can cover adjacent events within the same time period, while keeping the combined enumeration within an engineeringly acceptable range. Only co-occurrence lines are included in the window. If a line does not meet the co-occurrence criteria, that line is not included in the enumeration, thereby reducing invalid comparisons and ensuring that each verification falls within the event header range containing the key fields.

[0043] Enumerate co-occurrence pairs between the source and target ends in ascending order of absolute row difference from the center. For any absolute row difference d, first attempt to compare co-occurrence pairs with the same offset, i.e., combinations of d rows upward or downward from the source center with d rows upward or downward from the target center. This strategy prioritizes exploring combinations most likely to align when the event rhythms on both sides are similar, enabling stable pairings with fewer attempts. When combinations with the same offset fail the three-anchor check, expand to cross combinations within the same ring layer, i.e., d rows upward from the source center with d rows downward from the target center, and vice versa. Cross combinations are used to cover situations where relative displacement is inconsistent due to insertion or loss events. When multiple co-occurrence pairs exist within the same absolute row difference, enumerate in order of proximity to the center, first enumerating co-occurrence pairs closer to the center, then enumerating those farther away. Proximity to the center shortens the search path and reduces the risk of mismatch, as candidate co-occurrence pairs are usually close to the actual breakpoint region. For each enumerated co-occurrence pair, perform the three-anchor check as described in Example 1. The first co-occurrence pair to pass the three-anchor check is defined as a stable pair, and enumeration is immediately terminated. The definition of a stable pair requires that three constraints be met simultaneously, ensuring consistent alignment at the event level even in the presence of text format differences.

[0044] If no stable pairing is formed within the preset window, the window is proportionally expanded to twice its original size in a symmetrical manner on both sides, while maintaining the constraint of only including common pairs. The enumeration and verification process is then repeated. Proportional expansion balances the search range and cost without changing the enumeration order rules, gradually covering a larger neighborhood. When either side reaches the file boundary and still no stable pairing is formed, the next common pairing is scanned backward in the source binary log text file as a new candidate common pairing. The window and center are reset, and a bidirectional neighborhood search is performed again. This approach can overcome local misalignments caused by transactions crossing file or event boundaries, continuing forward until an aligned event is found. When a stable pairing occurs, the end_log_pos value parsed from the source common pairing is used to determine the final recovery breakpoint position. The source and target line numbers, timestamp text, and window size of this pairing are recorded for reference in subsequent rollback strategies. If a stable pairing is not formed within the acceptable maximum window range and acceptable scan span, the currently resolved maximum end_log_pos value and its corresponding source end row number are saved as the initial reference point for subsequent rollback retries. This avoids unbounded search caused by infinite expansion while retaining the most reliable end position clues to date.

[0045] The three-anchor check uses three constraints—endpoint presence, checksum presence, and timestamp consistency—to simultaneously limit candidate events. Endpoint presence eliminates the risk of erroneously extracting numbers from non-critical fields; checksum presence limits the judgment range to the event header containing the CRC32 checksum; and timestamp consistency provides verifiable evidence for cross-source alignment. The combination of these three constraints maintains stability under conditions of format changes, additions or removals of optional fields, and blank line insertions. Bidirectional neighborhood search expands centered on candidate co-occurrence lines, first attempting with the same offset, then expanding by cross-combinations. This quickly covers the most likely aligned neighborhoods while avoiding the exponential explosion caused by full combinations. In most scenarios, the actual corresponding event and candidate co-occurrence lines are close; prioritizing distance reduces search costs and shortens recovery time. Proportionally expanding the window helps gradually increase coverage when local alignment fails, while maintaining a priority order closer to the center, ensuring rapid convergence once a reliable pair is found.

[0046] Before performing a fully consistent comparison, the timestamp text on both sides is uniformly formatted. In scenarios containing time zone identifiers, the time is first converted to the same time zone before performing a fully consistent comparison. This method is suitable for database systems deployed across time zones, eliminating the impact of time differences in recording the same event on different nodes. Within the same absolute row difference, co-occurrence pairs containing both longer hexadecimal segments are compared first, followed by co-occurrence pairs containing shorter segments. Longer hexadecimal segments usually come from complete CRC32 checksum outputs; prioritizing the comparison of these row pairs can increase the probability of a single hit. The window growth is changed from proportional expansion to fixed-step expansion, for example, adding 8 rows on each side each time. Fixed-step expansion is suitable for scenarios where the log event density is known and fine-grained control of search costs is desired. When no timestamp is found within the preset proximity range, but the next data row of the same event contains a field that can be deduced as a timestamp, it is allowed to extend the search by 2 rows for supplementary searching. This method is used to be compatible with some export tools that place the timestamp multiple lines after the event header, without disrupting the main decision logic of the preset proximity range.

[0047] If the timestamp texts on both sides are still inconsistent after formatting, but the first and second anchors of the three anchors are satisfied and the end_log_pos value shows a monotonically increasing relationship between adjacent line pairs, the condition for the third anchor should not be relaxed. A bidirectional neighborhood search should continue to find line pairs that satisfy the three anchor checks to avoid mismatching adjacent but asynchronous events as stable pairs. If a stable pair still does not appear after multiple window expansions, the conversion settings in step one should be reviewed to ensure that the line boundaries of the events and the relative order of end_log_pos and CRC32 checksums remain unchanged. If necessary, the target binary log text file and the source binary log text file should be regenerated before executing the process of this embodiment.

[0048] Step 3: A controller generates incremental recovery parameters based on the final recovery breakpoint position. Starting from the next event corresponding to the recovery breakpoint position, it imports subsequent binary logs in ascending order of file number. If the import fails, it retryes based on the most recent stable pair. If the import succeeds, it records the recovery breakpoint position and the line number mapping that is currently present on both the source and target ends.

[0049] In one implementation, the controller reads the final recovery breakpoint position, the corresponding source-end co-occurrence and target-end co-occurrence, and the stable paired timestamp text. The starting file and starting offset are determined: the source binary log file containing the final recovery breakpoint position is located in the source binary log file set, and this final recovery breakpoint position is used as the starting offset. Importing begins from the next event after the final recovery breakpoint position because `end_log_pos` identifies the end boundary of the current event. Setting the starting offset to this boundary avoids replaying successfully imported events while ensuring that subsequent imports are continuously replayed using complete events as the smallest unit. Subsequent binary log files are sorted according to their filename sequence numbers in the directory, forming an ordered list from the starting file to the last available file. Importing in ascending file number order maintains the consistency of the event time sequence and prevents transaction replay failures caused by out-of-order cross-file imports. Incremental recovery parameters are constructed, including the starting file path, starting offset, ordered file list, event header parsing switch, CRC32 checksum verification switch, and a skip strategy for encountering unrecognizable events. Enable the event header parsing switch and the CRC32 checksum verification switch to perform format verification and checksum verification on each event header during the import process, thereby detecting problems as early as possible and triggering rollback when input is abnormal.

[0050] The controller opens the starting file, positions the read pointer at the starting offset, and parses the event header at the current position to confirm that the position is exactly at the start boundary of an event. If the read pointer is not at a valid start of the event header, it proceeds forward until a complete event header is detected before starting the import. Boundary confirmation avoids starting from the middle of a half-event, preventing the generation of incomplete statement sequences. Events in the starting file are traversed sequentially, each event is converted into an executable statement on the target end and immediately committed to the target database server. When a transaction start marker is encountered, the commit is not made until a transaction end marker is encountered, thus maintaining consistency between the source and target ends at transaction boundaries. After the current file is imported, the controller switches to the next binary log file in an ordered list, continuing the import from the first event header of that file until the end of the list. List switching follows a strict sequential process, without skipping any files to avoid data gaps due to omissions. During the import process, the controller records the current filename, the starting position of the current event header in the file, the timestamp text of the current event header, and the end_log_pos of the most recently successfully committed event as real-time progress. Real-time progress is used to quickly pinpoint rollback points in case of failure and to write audit logs in case of success.

[0051] When a statement execution failure, event parsing failure, or an error returned by the target database server occurs during the import process, the controller immediately pauses the import and locates the nearest stable pair. The nearest stable pair refers to the pair among the stable pairs recorded before the current failure position where the source-end common row number is the largest and does not exceed the source-end row number corresponding to the failure position. The reason for selecting the nearest stable pair is that this pair has passed three-anchor verification, providing a verified end-position marker. Rolling back to this marker minimizes the scope of duplicate imports and avoids crossing into unverified areas. The controller resets the starting file to the source-end binary log file corresponding to the nearest stable pair and resets the starting offset to the end_log_pos value parsed from the source-end common row of this stable pair. Importing still starts from the next event corresponding to this end_log_pos value to maintain idempotent protection for successful events. Incremental recovery parameters are regenerated, and the import process is executed again. If it fails again, the controller repeatedly selects an earlier stable pair as the new rollback point. Earlier stable pairs roll back sequentially from nearest to farthest row number until the import can proceed continuously or all stable pairs are exhausted. Before each rollback retry, the controller performs an independent consistency check on the event header at the rollback point: it reads the event header lines near the rollback point and verifies whether the end_log_pos, CRC32 checksum, and timestamp text are consistent with previous records. This check can detect cases where the input file has been replaced or corrupted before entering replay, preventing the use of an incorrect starting point for import.

[0052] Once the entire ordered list of files has been imported successfully without any unhandled errors, the controller writes the final recovery breakpoint location, starting file, last imported file, and time range to the audit log. The controller also persists the common line number mapping between the source and target ends to persistent media. This line number mapping includes: candidate common line numbers used to determine the final recovery breakpoint location and their corresponding common line numbers on the target end; stable line number pairs used as rollback criteria during the import process; and the starting and ending line numbers for each rollback. This persistence of the line number mapping allows subsequent batches to directly locate the last successful position and provides traceable evidence for fault analysis. The controller also saves a real-time progress snapshot recorded during the import process, including the current filename at each file switch and the end_log_pos of the most recent successful commit event. This snapshot is used for rapid resumption of the import after an abnormal power outage or external interruption, without requiring re-execution of the three-anchor check and bidirectional neighborhood search.

[0053] When the target database server's statement execution strategy is sensitive to duplicate events, import must begin from the next event after the final recovery breakpoint, and replaying events falling before the recovery breakpoint is not allowed. This constraint avoids unique constraint conflicts caused by duplicate commits. When the physical naming and time order of the source binary log files are not completely consistent, the controller should use a naming rule containing an incrementing sequence number as the sorting basis, not the file modification time, to prevent order errors caused by file system timestamp drift. When there are empty files with no events for a long time, the file should still be included in the ordered list and traversed empty to ensure the continuity of file sequence numbers is verified and to prevent skipping from causing an offset in the order judgment of subsequent files. When multiple rollbacks fail to advance, the controller should stop expanding the rollback range, retain the rollback point information that can stably parse the event header, and prompt for verification of whether the inputs of steps one and two meet the constraints of row boundaries and relative order preservation, preventing invalid retries on incorrect inputs.

[0054] Figure 2 This is a graph showing the performance of the co-occurrence detector as a function of binary log file complexity. The horizontal axis represents the binary log complexity exponent, ranging from 0 to 100. The left side of the vertical axis represents the detection accuracy percentage, ranging from 0 to 100%. The right side of the vertical axis represents the processing time in milliseconds, ranging from 0 to 500 ms. Figure 1As shown, the figure contains five main characteristic curves. The end_log_pos recognition accuracy curve is represented by a thick black solid line, starting at (0, 53%). It exhibits a rapid initial increase followed by a slower increase as the complexity exponent increases, reaching a critical inflection point (30, 62%) at a complexity exponent of 30. The rate of increase then slows, eventually reaching a saturation value (100, 73%) at a complexity exponent of 100. The CRC32 checksum pass rate curve is represented by a thick black dashed line, starting at (0, 48%). Its overall trend is similar to the end_log_pos curve, but the value is slightly lower. It enters a stable region (70, 67%) at a complexity exponent of 70, indicating that the CRC32 checksum algorithm has good stability in high-complexity logs. The co-occurrence count accuracy curve is represented by a black dotted line, starting at (0, 45%). This curve reflects the co-occurrence detector's ability to recognize rows containing both end_log_pos and CRC32 checksums. The processing time curve, represented by a thin black solid line, exhibits a clear non-linear characteristic. Within a complexity index of 0-20, the processing time remains relatively stable between 450-350ms. However, a significant performance inflection point occurs after a complexity index of 60, with the processing time dropping sharply to below 70ms. This indicates that the algorithm has better time efficiency when processing high-complexity logs. The acceptable accuracy threshold line marked in the figure is 40%, dividing the entire complexity range into three performance intervals: simple log (0-20), medium-complexity log (20-50), and high-complexity log (50-100). This provides an important reference for parameter tuning in practical applications.

[0055] Figure 3 This graph shows the convergence and window parameter optimization characteristics of the bidirectional neighborhood search algorithm. The horizontal axis represents the search window size in rows, ranging from 5 to 75 rows. The left side of the vertical axis represents the matching success rate percentage, ranging from 0 to 100%, and the right side represents the average number of iterations, ranging from 0 to 50. Figure 3As shown, the one-way search matching rate curve is represented by a thick black solid line, starting at (5, 15%). It exhibits a monotonically increasing trend with increasing window size, reaching its highest value (75, 74%) when the window size is 75 rows. The two-way search matching rate curve is represented by a thick black dashed line, starting at (5, 30%). It is significantly better than the one-way search, reaching a critical performance point (35, 78%) at a window size of 35 rows, after which the growth rate slows down, eventually reaching (75, 83%) at 75 rows. This comparison fully demonstrates the superiority of the two-way neighborhood search strategy in this invention. The three-anchor verification success rate curve is represented by a black dotted line, reflecting the comprehensive verification effect of the first anchor (existence of end_log_pos value), the second anchor (existence of CRC32 checksum), and the third anchor (event header timestamp matching) in the core technology of this invention. This curve starts at (5, 22%) and shows a significant improvement at a window size of 40 rows (40, 68%), indicating that an appropriate search window has a crucial impact on the success rate of the three-anchor verification. The average number of iterations curve, represented by a thin solid black line, illustrates the algorithm's convergence characteristics. The number of iterations is high (20-35) before the window size reaches 20 rows, then drops rapidly, stabilizing below 10 iterations at a window size of 50 rows, demonstrating the algorithm's efficiency. The window expansion failure rate curve, represented by a thin dashed black line, remains low before the window size reaches 45 rows, then rises sharply, indicating that an excessively large search window degrades algorithm performance. The optimal window region, marked in the figure, is 30-50 rows, with a critical failure rate of 30%, providing clear guidance for parameter settings in practical deployments.

[0056] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for quickly locating the location of MySQL binary log incremental recovery import failure, characterized in that... The method includes: Step 1: Execute the flush-logs command on the target database server to generate a latest binary log file for the target side. Obtain the latest binary log file for the target side and an imported binary log file for the source side. Convert the two binary log files into two line-by-line readable text files respectively, resulting in the target side binary log text file and the source side binary log text file. During the conversion process, the relative order of the event line boundaries, end_log_pos, and CRC32 checksum remains unchanged. Step 2: A co-occurrence detector is used to scan the target binary log text file line by line to identify co-occurrences that simultaneously contain both end_log_pos and CRC32 checksums. The identified co-occurrences are counted to obtain a target co-occurrence count value. The co-occurrence detector is used to locate candidate co-occurrences in the source binary log text file that correspond to the target co-occurrence count value. A finite state machine is used to parse the end_log_pos value from the candidate co-occurrences and confirm the presence of a CRC32 checksum in that line. Three-anchor verification is performed. If the verification passes, the parsed end_log_pos value is determined as the recovery breakpoint location. If the verification fails, a bidirectional neighborhood search is performed centered on the candidate co-occurrence line. Within a preset window, co-occurrence pairs between the source and target ends are enumerated and three-anchor verification is performed until a stable pair is obtained. The end_log_pos value parsed from the source co-occurrence in the stable pair is determined as the final recovery breakpoint location. Step 3: A controller generates incremental recovery parameters based on the final recovery breakpoint position. Starting from the next event corresponding to the recovery breakpoint position, it imports subsequent binary logs in ascending order of file number. If the import fails, it retryes based on the most recent stable pair. If the import succeeds, it records the recovery breakpoint position and the line number mapping that is currently present on both the source and target ends.

2. The method for quickly locating the location of MySQL binary log incremental recovery import failure as described in claim 1, characterized in that, Step one specifically includes: executing the flush-logs command on the target database server to lock the current binary log file and create a new binary log file, using the locked binary log file as the latest binary log file on the target side; selecting a source binary log file that has already been imported; converting the latest binary log file on the target side and the source binary log file into target binary log text files and source binary log text files respectively; during the conversion, keeping the line boundaries of events unchanged, unifying the character encoding to UTF-8, unifying the newline character to a single newline character, and preserving the original spelling and relative order of end_log_pos and CRC32 checksum.

3. The method for quickly locating the location of MySQL binary log incremental recovery import failure as described in claim 2, characterized in that, Step one also includes text preprocessing of the target binary log text file and the source binary log text file. The preprocessing includes: decoding each line of any binary log text file into a string starting from the first line; if decoding a single line fails, replacing it with a preset replacement character and continuing to process subsequent lines; deleting whitespace characters at the beginning and end of each line; converting all English letters in each line to lowercase; and retaining a copy of the original line that has not been preprocessed for each preprocessed line.

4. The method for quickly locating the location of MySQL binary log incremental recovery import failure as described in claim 3, characterized in that, Step two involves scanning the target binary log text file line by line using a co-occurrence detector. This includes: performing line-by-line matching for each line of the target binary log text file using the preset keywords end_log_pos and CRC32 checksum; if both end_log_pos and CRC32 checksum are matched in the current line, the current line is identified as a co-occurrence line, and a target co-occurrence counter is incremented; after completing the scanning of all lines of the target binary log text file, the final value of the target co-occurrence counter is recorded as the target co-occurrence count value.

5. The method for quickly locating the location of MySQL binary log incremental recovery import failure as described in claim 4, characterized in that, Step two involves locating candidate co-occurrences in the source binary log text file, which includes: scanning the source binary log text file line by line and using a co-occurrence detector to identify co-occurrences; incrementing a source co-occurrence counter by one for each identified co-occurrence; stopping the scan when the value of the source co-occurrence counter is equal to the target co-occurrence counter value for the first time, and determining the currently identified co-occurrence as a candidate co-occurrence.

6. The method for quickly locating the location of MySQL binary log incremental recovery import failure as described in claim 5, characterized in that, Step two involves using a finite state machine to parse the end_log_pos value from the candidate coherent lines. This includes: initializing the finite state machine to an initial state; scanning character by character starting from the beginning of the candidate coherent lines, and transitioning the finite state machine to a bootstrapping state when the end_log_pos keyword is detected; skipping non-numeric characters from the bootstrapping state, and transitioning the finite state machine to a data acquisition state when the first numeric character is detected; continuously reading subsequent numeric characters and combining them into a decimal integer until a non-numeric character is encountered, using this decimal integer as the end_log_pos value; and after acquiring the end_log_pos value, transitioning the finite state machine to a verification state, where the presence of the CRC32 checksum keyword is confirmed in the remaining portion of the candidate coherent lines.

7. The method for quickly locating the location of MySQL binary log incremental recovery import failure as described in claim 6, characterized in that, The three-anchor verification in step two includes: defining the first anchor as the existence of the end_log_pos value in the co-current; defining the second anchor as the existence of the CRC32 checksum in the co-current; and defining the third anchor as the timestamp in the event header associated with the co-current. The event header is identified using a preset rule, which requires that the line containing the event header simultaneously contains both serverid and end_log_pos, and that a timestamp exists within a preset neighborhood of that line. For a set of source and target co-currents, each is checked to see if it meets the conditions of the first and second anchors, and the event header is searched within its neighborhood to extract the timestamp. When both source and target co-currents meet the conditions of the first and second anchors, and the extracted timestamps are identical, the three-anchor verification is considered successful.

8. The method for quickly locating the location of MySQL binary log incremental recovery import failure as described in claim 7, characterized in that, Step two, the bidirectional neighborhood search, includes: defining a search window centered on the corresponding lines of candidate co-occurrences in the source and target binary log text files; within the search window, combining co-occurrences from the source and target ends according to a preset enumeration strategy to form co-occurrence pairs to be verified; performing a three-anchor verification for each co-occurrence pair to be verified; when a co-occurrence pair passes the three-anchor verification for the first time, the co-occurrence pair is determined as a stable pair, and the bidirectional neighborhood search is terminated; if all co-occurrence pairs in the search window fail the three-anchor verification, the scope of the search window is expanded and the aforementioned enumeration and verification steps are repeated; if no stable pair is found before reaching the file boundary in any binary log text file, the next co-occurrence in the source binary log text file is selected as a new candidate co-occurrence, and the bidirectional neighborhood search is re-executed.

9. The method for quickly locating the location of MySQL binary log incremental recovery import failure as described in claim 8, characterized in that, Step 3 specifically includes: the controller generates incremental recovery parameters based on the final recovery breakpoint position, which specifies that incremental import should begin from the events after the recovery breakpoint position; if incremental import fails, the controller performs a rollback retry based on the most recently found stable pairing position; if incremental import succeeds, the controller persists the final recovery breakpoint position, the corresponding current row number of the source and target ends, and the result of the three anchor point verification.

Images