Binary log file conversion method and device, computer equipment and storage medium
Patent Information
- Application Number
- CN202511770891.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-03-06
Smart Images
Figure CN121614441A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of database technology, and in particular to a binary log file conversion method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology
[0002] Large transactions refer to database transactions involving a large amount of data operations or lasting a long time. Such transactions may have a significant impact on the performance, availability, and data consistency of the database system. At the same time, the binary log files generated by large transactions are also very large. Therefore, how to better handle the binary log files generated during large transactions has become an urgent problem to be solved in this field.
[0003] Currently, the binary log files generated by large transactions are usually stored in a temporary storage area and then retrieved from the cached logs during the commit. However, since the binary log temporary files are unstructured and need to be completely copied during the commit, large transaction commits are prone to increased latency and decreased database concurrency processing capabilities. Therefore, the current conversion efficiency of binary log files is low. Summary of the Invention
[0004] Therefore, it is necessary to provide a binary log file conversion method, apparatus, computer device, computer-readable storage medium, and computer program product that improves the conversion efficiency of binary log files in order to address the above-mentioned technical problems.
[0005] Firstly, this application provides a binary log file conversion method, the method comprising:
[0006] When it is detected that a database transaction writes binary log data to the temporary storage area corresponding to the binary log cache, a binary temporary log file corresponding to the binary log data is generated in the temporary storage area. The binary temporary log file includes header reserved space and log event storage space. The log events in the log event storage space carry corresponding check codes.
[0007] If the binary temporary log file is found to meet the preset file conversion conditions, header event content conforming to the binary log format specification is written into the reserved space in the header.
[0008] Based on the configuration information of the log event storage space, the binary log data and corresponding location information are written into the log event storage space to obtain a binary candidate log file;
[0009] The binary candidate log file is converted into a binary official log file.
[0010] In one embodiment, writing header event content conforming to the binary log format specification into the reserved space in the header includes:
[0011] Obtain the header event set required by the binary log format specification;
[0012] Determine the writing order of each header event in the header event set, and determine the length of bytes occupied by each header event in the reserved space in the header;
[0013] Based on the first starting address of the reserved space in the header and the length of the occupied bytes, each header event is sequentially written to the first offset address of the reserved space in the header until all header events are written, and the header event content is obtained in the reserved space in the header.
[0014] In one embodiment, the header event set includes at least one of four events: format description event, preceding GTID event, padding event, and current large transaction GTID event.
[0015] In one embodiment, the step of writing the binary log data and corresponding location information into the log event storage space according to the configuration information of the log event storage space to obtain a binary candidate log file includes:
[0016] Based on the configuration information of the log event storage space, determine the second starting address of the log event storage space;
[0017] Based on the binary log data, determine the corresponding binary temporary log event set;
[0018] Based on the writing order of each binary temporary log event in the binary temporary log event set, generate the location information of each binary temporary log event;
[0019] Based on the location information and data content of each binary temporary log event, each binary temporary log event is sequentially written to the second offset address of the log event storage space until all binary temporary log events are written, thus obtaining the binary candidate log file.
[0020] In one embodiment, the binary temporary log event set includes at least one of four events: query events, table mapping events, row events, and transaction ID events.
[0021] In one embodiment, converting the binary candidate log file into a binary official log file includes:
[0022] Based on the official filename that conforms to the binary log format specification, the binary candidate log file is renamed to obtain the binary renamed log file, wherein the binary renamed file can fully run the validity verification algorithm of other binary official files;
[0023] Perform a validity check on the binary rename log file, and if the check passes, use the binary rename log file as the official binary log file.
[0024] In one embodiment, before writing header event content conforming to the binary log format specification into the reserved space in the header when the binary temporary log file is detected to meet preset file conversion conditions, the method further includes:
[0025] Obtain the current status of the file conversion trigger parameters of the database transaction;
[0026] If the current state is detected to be in the open state, determine the preset storage threshold configured by the log volume threshold configuration parameter corresponding to the database transaction;
[0027] If the file storage threshold configuration parameter of the binary temporary log file is detected to be greater than the preset storage threshold, it is determined that the binary temporary log file meets the preset file conversion condition.
[0028] Secondly, this application also provides a binary log file conversion apparatus, the apparatus comprising:
[0029] The generation module is used to generate a binary temporary log file corresponding to the binary log data in the temporary storage area when it is detected that a database transaction writes binary log data to the temporary storage area corresponding to the binary log cache. The binary temporary log file includes a header reserved space and a log event storage space. The log events in the log event storage space carry corresponding check codes.
[0030] The first writing module is used to write header event content conforming to the binary log format specification into the reserved space in the header when the binary temporary log file is detected to meet the preset file conversion conditions.
[0031] The second writing module is used to write the binary log data and corresponding position information into the log event storage space according to the configuration information of the reserved space in the header, so as to obtain a binary candidate log file;
[0032] The conversion module is used to convert the binary candidate log file into a binary official log file.
[0033] In one embodiment, the first writing module is further configured to:
[0034] Obtain the header event set required by the binary log format specification;
[0035] Determine the writing order of each header event in the header event set, and determine the length of bytes occupied by each header event in the reserved space in the header;
[0036] Based on the first starting address of the reserved space in the header and the length of the occupied bytes, each header event is sequentially written to the first offset address of the reserved space in the header until all header events are written, and the header event content is obtained in the reserved space in the header.
[0037] In one embodiment, the header event set includes at least one of four events: format description event, preceding GTID event, padding event, and current large transaction GTID event.
[0038] In one embodiment, the second writing module is further configured to:
[0039] Based on the configuration information of the log event storage space, determine the second starting address of the log event storage space;
[0040] Based on the binary log data, determine the corresponding binary temporary log event set;
[0041] Based on the writing order of each binary temporary log event in the binary temporary log event set, generate the location information of each binary temporary log event;
[0042] Based on the location information and data content of each binary temporary log event, each binary temporary log event is sequentially written to the second offset address of the log event storage space until all binary temporary log events are written, thus obtaining the binary candidate log file.
[0043] In one embodiment, the binary temporary log event set includes at least one of four events: query events, table mapping events, row events, and transaction ID events.
[0044] In one embodiment, the conversion module is further configured to:
[0045] Based on the official filename that conforms to the binary log format specification, the binary candidate log file is renamed to obtain the binary renamed log file, wherein the binary renamed file can fully run the validity verification algorithm of other binary official files;
[0046] Perform a validity check on the binary rename log file, and if the check passes, use the binary rename log file as the official binary log file.
[0047] In one embodiment, the device is further used to:
[0048] Obtain the current status of the file conversion trigger parameters of the database transaction;
[0049] If the current state is detected to be in the open state, determine the preset storage threshold configured by the log volume threshold configuration parameter corresponding to the database transaction;
[0050] If the file storage threshold configuration parameter of the binary temporary log file is detected to be greater than the preset storage threshold, it is determined that the binary temporary log file meets the preset file conversion condition.
[0051] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0052] When it is detected that a database transaction writes binary log data to the temporary storage area corresponding to the binary log cache, a binary temporary log file corresponding to the binary log data is generated in the temporary storage area. The binary temporary log file includes a header reserved space and a log event storage space, and the log events in the log event storage space carry corresponding checksums. When it is detected that the binary temporary log file meets the preset file conversion conditions, header event content conforming to the binary log format specification is written to the header reserved space. According to the configuration information of the log event storage space, the binary log data and corresponding position information are written to the log event storage space to obtain a binary candidate log file. The binary candidate log file is then converted into a binary official log file.
[0053] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:
[0054] When it is detected that a database transaction writes binary log data to the temporary storage area corresponding to the binary log cache, a binary temporary log file corresponding to the binary log data is generated in the temporary storage area. The binary temporary log file includes a header reserved space and a log event storage space, and the log events in the log event storage space carry corresponding checksums. When it is detected that the binary temporary log file meets the preset file conversion conditions, header event content conforming to the binary log format specification is written to the header reserved space. According to the configuration information of the log event storage space, the binary log data and corresponding position information are written to the log event storage space to obtain a binary candidate log file. The binary candidate log file is then converted into a binary official log file.
[0055] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:
[0056] When it is detected that a database transaction writes binary log data to the temporary storage area corresponding to the binary log cache, a binary temporary log file corresponding to the binary log data is generated in the temporary storage area. The binary temporary log file includes a header reserved space and a log event storage space, and the log events in the log event storage space carry corresponding checksums. When it is detected that the binary temporary log file meets the preset file conversion conditions, header event content conforming to the binary log format specification is written to the header reserved space. According to the configuration information of the log event storage space, the binary log data and corresponding position information are written to the log event storage space to obtain a binary candidate log file. The binary candidate log file is then converted into a binary official log file.
[0057] The aforementioned binary log file conversion method, apparatus, computer equipment, computer-readable storage medium, and computer program product first, upon detecting that a database transaction is writing binary log data to the temporary storage area corresponding to the binary log cache, generate a binary temporary log file corresponding to the binary log data in the temporary storage area. This binary temporary log file includes header reserved space and log event storage space, with each log event in the log event storage space carrying a corresponding checksum. Then, upon detecting that the binary temporary log file meets preset file conversion conditions, header event content conforming to the binary log format specification is written to the header reserved space. Next, based on the configuration information of the log event storage space, binary log data and corresponding location information are written to the log event storage space to obtain a binary candidate log file. Finally, the binary log file is converted into a binary log file. The binary candidate log file is converted into a binary official log file. During the process of writing binary log data to the temporary storage area corresponding to the binary log cache in a database transaction, the structured temporary log file can directly complete the data storage, location information, and calculations. Then, during the conversion, a renaming operation is used to perform a full data copy and file reconstruction. This allows for the direct submission of the converted binary official log file from the temporary storage area, rather than relying on redundant data migration and repeated calculations between the temporary and official files. This overcomes the technical shortcomings of the unstructured design of binary log temporary files, which requires a complete copy of the binary log temporary file during submission, leading to increased latency and decreased database concurrency during large transaction submissions. Therefore, the conversion efficiency of binary log files is improved. Attached Figure Description
[0058] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0059] Figure 1 This is a flowchart illustrating a binary log file conversion method in one embodiment;
[0060] Figure 2 This is a schematic diagram illustrating the writing of a binary temporary log file in a binary log file conversion method according to one embodiment.
[0061] Figure 3 This is a schematic diagram illustrating the conversion of a binary temporary log file into a binary candidate log file in one embodiment of the binary log file conversion method.
[0062] Figure 4This is a flowchart illustrating a binary log file conversion method in another embodiment;
[0063] Figure 5 This is a schematic diagram of the modified binary candidate log file group submission for the binary log file conversion method in another embodiment;
[0064] Figure 6 A block diagram of a binary log file conversion device;
[0065] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0066] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0067] First, to clearly understand this solution, the following terms are explained: 1) A binary log file refers to a log file that records transaction operation details, data change traces, and related metadata in chronological order when the database system performs data operations. Specifically, it can be a Binlog file. The Binlog file is an important log recording format maintained by the MySQL server layer, mainly used to record various database change operations. It is a key mechanism for MySQL to achieve data recovery, data replication (master-slave replication), and data auditing; 2) A temporary storage area refers to a memory or disk-level cache space in the database system used to temporarily cache binary log data generated by transactions, avoiding frequent I / O overhead caused by directly writing to the disk. Specifically, it can be a Binlog Cache. The Binlog Cache is a temporary storage area allocated by the MySQL server for each client session (Connection). It is used to temporarily store events that are about to be written to the Binlog before the transaction is committed. When the transaction is committed, the Binlog... Data in the cache will be written to the Binlog file all at once; 3) Large transactions refer to database transactions involving a large amount of data operations or lasting a long time. Such transactions may have a significant impact on the performance, availability and data consistency of the database system. It is understandable that the log files generated by large transactions may reach hundreds of MB or even hundreds of GB.
[0068] In business scenarios with large transactions, the commit of large transactions often results in excessively long Binlog file processing times, instances becoming unwritable for extended periods, or instances becoming stuck. The main reason for this is that the method of writing the content in the Binlog Cache to the Binlog file all at once during the commit process involves a copying process: reading cache content, modifying the header and checksum of each event, and then writing it back to the Binlog file. In other words, the database process needs to load and retrieve all log data generated during the execution of large transactions from the memory area corresponding to the Binlog Cache. During this process, because the temporary binary log file is unstructured and must be completely copied during commit, it easily leads to increased latency and decreased database concurrency during large transaction commits. Therefore, there is an urgent need for a binary log file conversion method that can improve the efficiency of binary log file conversion.
[0069] In one embodiment, such as Figure 2As shown, a binary log file conversion method is provided. This embodiment uses the method applied to a terminal as an example. The terminal includes, but is not limited to, personal computers, laptops, smartphones, and tablets. The terminal includes a generation module, a first writing module, a second writing module, and a conversion module. The generation module is used to generate a binary temporary log file corresponding to the binary log data in the temporary storage area when it detects that a database transaction is writing binary log data to the temporary storage area corresponding to the binary log cache. The binary temporary log file includes a header reserved space and a log event storage space. The log events in the log event storage space carry corresponding check codes. The first writing module is used to convert binary temporary log files when it detects that the binary temporary log file meets preset file conversion conditions. The first writing module writes header event content conforming to the binary log format specification into the reserved space in the header; the second writing module writes binary log data and corresponding position information into the log event storage space according to the configuration information of the reserved space in the header, to obtain a binary candidate log file; the conversion module converts the binary candidate log file into a binary official log file. Through the information interaction between the generation module, the first writing module, the second writing module, and the conversion module, the structured temporary log file can be directly stored and its position information and calculations can be completed. Then, during the conversion, a full data copy and file reconstruction can be performed by renaming operations, so as to achieve the purpose of directly submitting the converted binary official log file in the temporary storage area. Therefore, the conversion efficiency of binary log files can be improved. This method can also be applied to servers, and can also be applied to systems including terminals and servers, and can be implemented through the interaction between terminals and servers. In this embodiment, the method includes the following steps 202 to 208. Wherein:
[0070] Step 202: When it is detected that a database transaction writes binary log data to the temporary storage area corresponding to the binary log cache, a binary temporary log file corresponding to the binary log data is generated in the temporary storage area. The binary temporary log file includes header reserved space and log event storage space. The log events in the log event storage space carry corresponding check codes.
[0071] It should be noted that, in order to solve the problem of service bottleneck caused by large transactions, the following approach can be adopted: rename the cache directly to a binlog file to avoid copying issues. However, the following issues need to be addressed during this conversion: 1) The insertion of the binlog file header and the preceding event needs to be considered; 2) Binlog log rotation issues; 3) Modification of the MySQL group commit process; 4) Event checksum (generation and consistency guarantee of events and checks).
[0072] It should be noted that a database transaction represents a series of indivisible, continuous data operations in the database; binary log data refers to the log information recorded in binary format during the execution of a database transaction, including details of data changes, operation instructions, and transaction metadata; a binary temporary log file refers to an intermediate file generated in a temporary storage area and used for structured storage of binary log data; header reserved space represents a fixed-size storage area pre-allocated in the header of the binary temporary log file; and log time storage space represents a storage area in the binary temporary log file, excluding the header reserved space, specifically used to store specific log event data and their corresponding location information. The log events in the log event storage space carry corresponding checksums.
[0073] Understandably, to achieve binary log file conversion, three global parameters need to be set in advance: `loose_binlog_cache_free_flush` (an optimization switch to control whether file conversion is triggered), `checksum_in_binlog_cache` (a compatibility switch to verify binary log files), and `loose_binlog_cache_free_flush_limit_size` (a parameter defining the large transaction threshold, meaning that conversion from cache files to binary log files is only triggered when the amount of Binlog logs generated by a transaction exceeds the threshold). `loose_binlog_cache_free_flush` is a global system variable; modifying this parameter takes effect immediately without restarting the instance, and its value can be either `on` or `off`. When `checksum_in_binlog_cache` is enabled, a checksum will also be written when writing binlog to the cache to ensure compatibility with Binlog Cache Free Flush enabled; the binlog file will also be converted based on `opt_binlog_checksum`. The option determines whether there is a checksum, and its value range can be on or off; loose_binlog_cache_free_flush_limit_size indicates that when the transaction's Binlog exceeds this parameter value, the Binlog Cache temporary file will be converted into a Binlog official file upon commit. Specifically, its default value can be 256MB, and its value range can be 20971520~18446744073709551615.
[0074] As an example, step 202 includes: when a database transaction is triggered to write binary log data to the temporary storage area corresponding to the binary log cache, a binary temporary log file containing header reserved space and log storage space corresponding to the binary log data is generated in the temporary storage area.
[0075] In one feasible approach, refer to Figure 2 The diagram illustrates the writing process of a binary temporary log file. It's understandable that, to ensure the temporary files in the Binlog Cache (temporary storage area) can be subsequently converted into official binary log files, the usage of the Binlog Cache has been modified as follows: 1) When writing to the Binlog Cache's temporary files, a portion of space is reserved at the beginning of the file. This space is used to store the events in the Binlog header during the conversion of the temporary file to the official Binlog file; 2) When writing to the Binlog Cache, each event calculates its own end_pos based on the size of the reserved space. Figure 2 In this context, the reserved space refers to the space reserved for the header, the actual writing area of the Binlog Cache refers to the storage space for log events, and the temporary file of the Binlog Cache is the binary temporary log file.
[0076] Step 204: If the binary temporary log file is found to meet the preset file conversion conditions, write the header event content that conforms to the binary log format specification into the reserved space at the beginning.
[0077] It should be noted that the preset file conversion conditions represent the criteria for triggering the conversion of the binary temporary log file to the binary candidate log file. Specifically, this can be composed of the state of the function enable control switch mentioned above and the transaction log volume threshold. For example, in one feasible approach, the conversion is triggered when the function switch is enabled and the file storage volume of the binary temporary log file exceeds the preset threshold, thereby avoiding unnecessary process overhead under non-large transactions. The binary log format specification represents the file format standard and event structure requirements of the database system's native binary log that must be followed, which may include the file header identifier format, segment length encoding, and event type definition, etc. The header event content represents the set of various log events that constitute the core information of the binary log file header.
[0078] As an example, step 204 includes: when the function switch is turned on and the file storage size of the binary temporary log file exceeds a preset threshold, determining that the binary temporary log file meets the preset file conversion conditions, and writing header event content that conforms to the binary log format into the reserved space in the header.
[0079] In one embodiment, header event content conforming to the binary log format specification is written into the reserved space in the header, including:
[0080] Obtain the header event set required by the binary log format specification; determine the writing order of each header event in the header event set, and determine the byte length occupied by each header event in the header reserved space; according to the first starting address and the byte length occupied by the header reserved space, write each header event to the first offset address of the header reserved space in sequence until all header events are written, and obtain the header event content in the header reserved space.
[0081] It should be noted that, during the process of writing header event content into the reserved space in the header, the total number of bytes in the reserved space in the header must first be determined in advance according to the binary log format specification and the maximum possible byte length occupied by the header event set, so as to ensure that the header event set can be written completely and does not exceed the reserved space range; the first starting address represents the starting position of the reserved space in the header, which can be a file offset address; when writing different header events, they must be written in the order of writing each header event to ensure the structural integrity of the reserved space in the header.
[0082] It should be noted that the header event set includes at least one of the following: format description event, preceding GTID event, padding event, and current large transaction GTID event. The format description event can be represented as a File Header, specifically located in the first 4 bytes of each Binlog file, each containing an identifier "[0xFE 'bin']" to identify the file as a Binlog file. The preceding GTID event can be represented as Header Events, specifically including the Format description event and the Previous GTID event. The padding event can be represented as a Roam event, used to fill the remaining reserved space in the header, inheriting from the ignoreable event type already present in the current Binlog. The current large transaction GTID event can be represented as a GTID event; the GTID of the large transaction is generated during the commit phase, and the GTID event is also written to the reserved space. It can be understood that after the Rename is completed, the Binlog file header and the several events in the header (File Header and Header Events) are... Events can fill the header reserved space, which can include format description events, preceding GTID events, fill events, and the GTID events of the current large transaction.
[0083] As an example, after renaming the binary temporary log file, obtain the format description event, preceding GTID event, padding event, and current large transaction GTID event required by the binary log format specification; determine the writing order of the format description event, preceding GTID event, padding event, and current large transaction GTID event according to the binary log format specification, and determine the byte length occupied by the format description event, preceding GTID event, padding event, and current large transaction GTID event in the header reserved space according to the data element content corresponding to each of the format description event, preceding GTID event, padding event, and current large transaction GTID event; using the first starting address of the header reserved space as a reference, write the format description event, preceding GTID event, current large transaction GTID event, and padding event sequentially according to the occupied byte length; after the padding event is written, obtain the header event content in the header reserved space.
[0084] In one feasible approach, refer to Figure 3 , Figure 3 To illustrate the conversion of a binary temporary log file into a binary candidate log file, combined with... Figure 3 It is known that when committing large transactions, if the temporary Binlog Cache files need to be converted into official Binlog files, MySQL needs to disable the use of O_TMPFILE for Cache file creation to prevent the Cache files from being anonymous and thus unable to be renamed. If the operating system does not support O_TMPFILE, MySQL will create the file and then immediately delete it, retaining the handle in memory to automatically release the Cache file. This also prevents the files from being renamed, so the deletion operation needs to be removed. The deletion operation is performed in two additional stages: one when the session is closed, and the other when the cache files in the tmp directory are cleared at startup. In addition, the tmp path and binlog path must be on the same partition; otherwise, the renaming operation will become a copy operation, failing to achieve the optimization effect.
[0085] Step 206: Based on the configuration information of the log event storage space, write binary log data and corresponding location information into the log event storage space to obtain binary candidate log files.
[0086] It should be noted that the configuration information represents the storage rules and parameters of the pre-defined log event storage space; the location information represents the set of address parameters that correspond one-to-one with each binary log event and identify its storage range in the final official log file; the binary candidate log file refers to the intermediate file that is structurally complete and conforms to the binary log format specification after the binary temporary log file has been written with the header event content in the reserved space in the header and the log data and location information in the log event storage space.
[0087] As an example, step 206 includes: writing binary log data and corresponding location information into the log event storage space according to the configuration information of the log event storage space to obtain a binary candidate log file.
[0088] In one embodiment, based on the configuration information of the log event storage space, binary log data and corresponding location information are written into the log event storage space to obtain binary candidate log files, including:
[0089] Based on the configuration information of the log event storage space, determine the second starting address of the log event storage space; based on the binary log data, determine the corresponding binary temporary log event set; based on the writing order of each binary temporary log event in the binary temporary log event set, generate the location information of each binary temporary log event; based on the location information and data content of each binary temporary log event, write each binary temporary log event sequentially to the second offset address of the log event storage space until all binary temporary log events are written, thus obtaining the binary candidate log file.
[0090] It should be noted that during the process of writing binary log data and corresponding location information into the log event storage space, the second starting address must first be clearly defined as the physical starting offset address of the log event storage space. The value of the second starting address can be equal to the total number of bytes in the header reserved space. This avoids overlap between the address ranges of the header reserved space and the log event storage space, thus completely avoiding data write conflicts. Furthermore, the binary temporary log event set is a structured set of events obtained by splitting binary log data according to the transaction execution order. The splitting of each log event must follow the binary log format specification to ensure the integrity of the event structure. Simultaneously, the location information can be generated using the formula: current log event start_pos = header reserved space size + total bytes occupied by all preceding events, end position = current event start position + current log event bytes occupied. This ensures that the location information completely matches the physical address of the final official log file. Finally, the second offset address is based on the second starting address, and the actual write address is determined according to the start_pos of each event. During the writing process, the writing order of events must be consistent with the generation order, and the location information is stored in association with the event data content to ensure subsequent validity verification and rapid event location.
[0091] It should be noted that the binary temporary log event set includes at least one of four events: query event, table mapping event, row event, and transaction ID event. Among them, the query event can be represented as a Query event, the table mapping event as a Tablemap event, the row event as a Row event, and the transaction ID event as a Xid event. For example, in one possible implementation, the binary temporary log event set includes the four events: query event, table mapping event, row event, and transaction ID event.
[0092] As an example, based on the configuration information of the log event storage space, the second starting position of the log event storage space is determined; by parsing the binary log data, the corresponding binary temporary log events are obtained; based on the writing order of each binary temporary log event in the binary temporary log event set, the length of each event in bytes, and the size of the reserved space at the beginning of the binary temporary log file, the position information corresponding to each binary temporary log event is generated; based on the position information and data content of each binary temporary log event, each binary temporary log event is written to the second offset address of the log event storage space in sequence until all binary temporary log events are written, thus obtaining a binary candidate log file.
[0093] Step 208: Convert the binary candidate log file into a binary official log file.
[0094] As an example, step 208 includes converting the binary candidate log file into a binary official log file.
[0095] The aforementioned binary log file conversion method first generates a binary temporary log file corresponding to the binary log data in the temporary storage area corresponding to the binary log cache when it is detected that a database transaction is writing binary log data to the temporary storage area. This binary temporary log file includes header reserved space and log event storage space, with each log event in the log event storage space carrying a corresponding checksum. Next, if the binary temporary log file meets preset file conversion conditions, header event content conforming to the binary log format specification is written to the header reserved space. Then, according to the configuration information of the log event storage space, binary log data and corresponding location information are written to the log event storage space to obtain a binary candidate log file. Finally, the binary candidate log file is converted into a binary log file. The system creates a formal log file. During the process of writing binary log data to the temporary storage area corresponding to the binary log cache in a database transaction, the structured temporary log file can directly complete the data storage, location information, and calculations. Then, during the conversion, a renaming operation is used to perform a full data copy and file reconstruction. This allows for the direct submission of the converted binary log file obtained from the temporary storage area, rather than relying on redundant data migration and repeated calculations between the temporary and formal files. This overcomes the technical shortcomings of the unstructured design of the binary log temporary file, which requires a complete copy of the binary log temporary file during submission, leading to increased latency and decreased database concurrency during large transaction submissions. Therefore, the conversion efficiency of binary log files is improved.
[0096] In one embodiment, such as Figure 4 As shown, converting a binary candidate log file into a binary official log file includes:
[0097] Step 302: Rename the binary candidate log file according to the official file name that conforms to the binary log format specification to obtain the binary renamed log file. The binary renamed file can fully run the validity verification algorithm of other binary official files.
[0098] It should be noted that the official filenames must follow MySQL's Binlog file naming rules. Renamed log files under these rules can include server identifiers, timestamps, or sequence numbers to ensure filenames are unique and recognizable by the system. Simultaneously, renaming is necessary to replace the full data copying solution. Combined with the group commit process modification, Binlog rotation preprocessing must be completed before renaming, specifically including writing Rotate Events and obtaining the next filename, to avoid conflicts with Binlog files from non-large transactions and ensure consistency in the group commit order. Understandably, the renamed binary file can fully run the validity verification algorithms of other binary official files, facilitating subsequent validity verification of the renamed binary file.
[0099] As an example, step 302 includes: renaming the binary candidate log file according to the official file name that conforms to the binary log format specification to obtain a binary renamed log file, which can fully run the validity verification algorithm of other binary official files.
[0100] Step 304: Perform a validity check on the binary rename log file, and if the check passes, use the binary rename log file as the official binary log file.
[0101] It should be noted that the validity verification process needs to be combined with the structural design, group commit modification and checksum processing mechanism of the binary temporary log file in the above embodiments to ensure the compliance of the file format; for example, in one feasible approach, the dimensions for performing validity verification on the binary renamed log file may include position consistency dimension, structural integrity dimension and checksum consistency verification dimension, etc.
[0102] As an example, step 304 includes: performing a validity check on the binary rename log file, and, if the check passes, using the binary rename log file as the official binary log file.
[0103] Understandably, referring to Figure 5 , Figure 5This is a schematic diagram of the modified binary candidate log file group commit. Since the optimization in this embodiment modifies the flush behavior, in the group commit scenario, non-large transactions and large transactions in the flush queue are no longer flushed to the same binlog file. Therefore, the group commit process needs to be modified, as follows: 1) Large transactions are processed separately instead of being executed with the leader. Transactions following the large transaction then become the leader and execute subsequent tasks. This process preserves the order in which transactions entered the queue, so large transactions do not participate in the group commit and the transaction commit order remains unchanged; 2) The commit process for large transactions is the same as the commit process for different groups. No modification to the flush phase is needed. Adding Binlog rotation logic during the renaming process in the flush phase has a minimal impact on the overall ordered_commit process. Specifically, firstly, the large transaction itself is extracted separately from the queue and executes the flush phase as its own leader; secondly, the flush phase first performs a part of the binlog rotation operation, namely writing a Rotate Event to the current Binlog file, obtaining the next Binlog filename, and calculating the Previous GTID to be written to the next file. Event and GTID Event; Third, write the content from point two into the reserved space of the Binlog Cache file; Fourth, consider the crash recovery correctness of the renaming and rotation process. The renaming process is as follows: Open the BinlogIndex file and the Binlog index purge file, check the contents of the purge file and purge the contents of the corresponding index, and then write the renamed Binlog file name into the purge file; rename the Cache file to the Binlog file; append the Binlog file name to the Binlog Index file, and then delete the Binlog purge file; In point four above, if the system crashes before deleting the Binlog purge file, the contents corresponding to the Binlog purge file will be deleted during system recovery, thus achieving the requirement of ensuring crash recovery correctness.
[0104] It is important to emphasize that, to meet the needs of scenarios where checksums still need to be retained after optimization of some large transactions, the `checksum_in_binlog_cache` parameter can be used for control. When the `checksum_in_binlog_cache` parameter is enabled, the checksum will be written when the cache file is written to an event. Therefore, when the cache file is renamed to the binlog file, the new binlog will also have the checksum. However, enabling this parameter will reduce the performance of other non-large transactions, so it can be enabled according to user needs.
[0105] In one embodiment, before writing header event content conforming to the binary log format specification into the reserved space at the beginning of the binary temporary log file, the method further includes:
[0106] Obtain the current status of the file conversion trigger parameters for the database transaction; if the current status is detected as enabled, determine the preset storage threshold configured by the log volume threshold configuration parameter corresponding to the database transaction; if the file storage volume threshold configuration parameter of the binary temporary log file is detected as greater than the preset storage threshold, determine that the binary temporary log file meets the preset file conversion conditions.
[0107] It should be noted that the binary temporary log file is triggered to meet the preset file conversion conditions through a system global variable. Specifically, the file conversion trigger parameter can also be understood as the aforementioned function control switch loose_binlog_cache_free_flush. The binary temporary log file is determined to meet the preset file conversion conditions if and only if the file conversion trigger parameter is enabled and the file storage size threshold configuration parameter of the binary temporary log file is greater than the preset storage threshold. The preset storage threshold is loose_binlog_cache_free_flush_limit_size.
[0108] As an example, the current status of the file conversion trigger parameter of the database transaction is obtained; if the current status is detected to be enabled, the preset storage threshold configured by the log volume threshold configuration parameter corresponding to the database transaction is determined; if the file storage volume threshold configuration parameter of the binary temporary log file is detected to be greater than the preset storage threshold, the binary temporary log file is determined to meet the preset file conversion conditions.
[0109] In one feasible approach, firstly, upon detecting that a database transaction is writing binary log data to the temporary storage area corresponding to the binary log cache, a binary temporary log file corresponding to the binary log data is generated in the temporary storage area. This binary temporary log file includes header reserved space and log event storage space, with log events in the log event storage space carrying corresponding checksums. Next, the current status of the database transaction's file conversion trigger parameters is obtained. If the current status is detected as "on," the preset storage threshold configured by the log volume threshold configuration parameter corresponding to the database transaction is determined. If the file storage volume threshold configuration parameter of the binary temporary log file is detected as greater than the preset storage threshold, it is determined that the binary temporary log file meets the preset file conversion conditions. Then, the header event set required by the binary log format specification is obtained, including format description events, preceding GTID events, padding events, and the GTID event of the current large transaction. The writing order of each header event in the header event set is determined, as well as the byte length occupied by each header event in the header reserved space. Based on the first starting address and byte length of the header reserved space, each... Each header event is written to the first offset address of the header reserved space until all header events are written, and the header event content is obtained in the header reserved space. Then, according to the configuration information of the log event storage space, the second starting address of the log event storage space is determined. According to the binary log data, the corresponding binary temporary log event set is determined, where the binary temporary log event set includes query events, table mapping events, row events, and transaction ID events. According to the writing order of each binary temporary log event in the binary temporary log event set, the position information of each binary temporary log event is generated. According to the position information and data content of each binary temporary log event, each binary temporary log event is written to the second offset address of the log event storage space in sequence until all binary temporary log events are written, resulting in a binary candidate log file. Finally, the binary candidate log file is renamed according to the official file name that conforms to the binary log format specification, resulting in a binary renamed log file. The log events in the log event storage space carry corresponding check codes. The binary renamed log file is validated, and after the validation passes, the binary renamed log file is used as the binary official log file.
[0110] The above implementation achieves the following technical effects: 1) It describes an optimization method for large transaction commits, the main method of which is to use file renaming to cancel the process of copying the contents of the Cache file to the Binlog file during the large transaction commit process; 2) The process of renaming the Binlog file requires first identifying the large transaction and extracting it separately from the waiting queue of the group commit to execute the commit process, while preserving the order in which the transaction entered the queue; 3) When the Cache file is created, sufficient space is reserved in advance to fill in the necessary header event content when it is converted into a Binlog file; 4) The renaming process of the Cache file correctly handles the details of Binlog rotation and has process crash recovery capability; 5) checksum_in_binlog_cache can control whether the Cache file contains a checksum to meet the scenario where the Binlog file needs to have a complete checksum.
[0111] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0112] Based on the same inventive concept, this application also provides a binary log file conversion apparatus for implementing the binary log file conversion method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the binary log file conversion apparatus provided below can be found in the limitations of the binary log file conversion method described above, and will not be repeated here.
[0113] In one exemplary embodiment, such as Figure 6 As shown, a binary log file conversion device is provided, comprising: a generation module 401, a first writing module 402, a second writing module 403, and a conversion module 404, wherein:
[0114] The generation module 401 is used to generate a binary temporary log file corresponding to the binary log data in the temporary storage area when it is detected that the database transaction writes binary log data to the temporary storage area corresponding to the binary log cache. The binary temporary log file includes a header reserved space and a log event storage space. The log events in the log event storage space carry corresponding check codes.
[0115] The first writing module 402 is used to write header event content that conforms to the binary log format specification into the reserved space in the header when the binary temporary log file is detected to meet the preset file conversion conditions.
[0116] The second writing module 403 is used to write binary log data and corresponding position information into the log event storage space according to the configuration information of the reserved space in the header, so as to obtain a binary candidate log file;
[0117] Transformation module 404 is used to convert binary candidate log files into binary official log files.
[0118] In one embodiment, the first writing module 402 is further configured to:
[0119] Obtain the header event set required by the binary log format specification; determine the writing order of each header event in the header event set, and determine the byte length occupied by each header event in the header reserved space; according to the first starting address and the byte length occupied by the header reserved space, write each header event to the first offset address of the header reserved space in sequence until all header events are written, and obtain the header event content in the header reserved space.
[0120] In one embodiment, the header event set includes at least one of four events: a format description event, a preceding GTID event, a padding event, and a GTID event for the current large transaction.
[0121] In one embodiment, the second writing module 403 is further configured to:
[0122] Based on the configuration information of the log event storage space, determine the second starting address of the log event storage space; based on the binary log data, determine the corresponding binary temporary log event set; based on the writing order of each binary temporary log event in the binary temporary log event set, generate the location information of each binary temporary log event; based on the location information and data content of each binary temporary log event, write each binary temporary log event sequentially to the second offset address of the log event storage space until all binary temporary log events are written, thus obtaining the binary candidate log file.
[0123] In one embodiment, the binary temporary log event set includes at least one of four events: query events, table mapping events, row events, and transaction ID events.
[0124] In one embodiment, the conversion module 404 is further configured to:
[0125] Based on the official filename conforming to the binary log format specification, the binary candidate log file is renamed to obtain the binary renamed log file. The binary renamed file can fully run the validity verification algorithm of other binary official files. The validity verification is performed on the binary renamed log file, and after the verification passes, the binary renamed log file is used as the binary official log file.
[0126] In one embodiment, the binary log file conversion apparatus is further used for:
[0127] Obtain the current status of the file conversion trigger parameters for the database transaction; if the current status is detected as enabled, determine the preset storage threshold configured by the log volume threshold configuration parameter corresponding to the database transaction; if the file storage volume threshold configuration parameter of the binary temporary log file is detected as greater than the preset storage threshold, determine that the binary temporary log file meets the preset file conversion conditions.
[0128] The modules in the aforementioned binary log file conversion device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0129] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a binary log file conversion method. Those skilled in the art will understand that... Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0130] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.
[0131] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0132] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0133] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0134] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0135] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A binary log file conversion method, characterized by, The method comprises: In the case of detecting that the database transaction writes binary log data to the corresponding temporary storage area of the binary log cache, a binary temporary log file corresponding to the binary log data is generated in the temporary storage area, wherein the binary temporary log file comprises a header reserved space and a log event storage space, and a log event of the log event storage space carries corresponding check code; In the case of detecting that the binary temporary log file meets a preset file conversion condition, a header event content conforming to a binary log format specification is written in the header reserved space; According to configuration information of the log event storage space, the binary log data and corresponding position information are written in the log event storage space to obtain a binary candidate log file; The binary candidate log file is converted into a binary formal log file.
2. The method of claim 1, wherein, The writing of the header event content conforming to the binary log format specification in the header reserved space comprises: Obtaining a header event set required by a binary log format specification; Determining a writing order of each header event in the header event set, and determining an occupied byte length of each header event in the header reserved space; According to a first starting address of the header reserved space and the occupied byte length, each header event is sequentially written into a first offset address of the header reserved space until the writing of all header events is completed, and the header event content is obtained in the header reserved space.
3. The method of claim 2, wherein, The header event set comprises at least one of a format description event, a prequel GTID event, a padding event and a GTID event of a current large transaction.
4. The method of claim 1, wherein, According to the configuration information of the log event storage space, the binary log data and corresponding position information are written in the log event storage space to obtain a binary candidate log file, comprising: According to the configuration information of the log event storage space, a second starting address of the log event storage space is determined; According to the binary log data, a corresponding binary temporary log event set is determined; According to a writing order of each binary temporary log event in the binary temporary log event set, position information of each binary temporary log event is generated; According to the position information and data content of each binary temporary log event, each binary temporary log event is sequentially written into a second offset address of the log event storage space until the writing of all binary temporary log events is completed, and the binary candidate log file is obtained.
5. The method of claim 4, wherein, The binary temporary log event set comprises at least one of a query event, a table mapping event, a row event and a transaction ID event.
6. The method of claim 1, wherein, The conversion of the binary candidate log file into a binary formal log file comprises: According to a formal file name conforming to a binary log format specification, the binary candidate log file is renamed to obtain a binary renamed log file, wherein the binary renamed file can completely run an effectiveness check algorithm of other binary formal files. The validity check is performed on the binary renaming log file, and the binary renaming log file is taken as the binary formal log file after the check passes.
7. The method of claim 1, wherein, In the case where it is detected that the binary temporary log file meets the preset file conversion condition, before the header reserved space writes the header event content conforming to the binary log format specification, the method further includes: Obtaining the current state of the file conversion trigger parameter of the database transaction; In the case where it is detected that the current state is an open state, determining a preset storage threshold configured by a log amount threshold configuration parameter corresponding to the database transaction; In the case where it is detected that the file storage amount threshold configuration parameter of the binary temporary log file is greater than the preset storage threshold, it is determined that the binary temporary log file meets the preset file conversion condition.
8. A binary log file conversion apparatus, characterized by, The device includes: The generating module is configured to, in the case where it is detected that the database transaction writes binary log data to a temporary storage area corresponding to a binary log cache, generate a binary temporary log file corresponding to the binary log data in the temporary storage area, wherein the binary temporary log file includes a header reserved space and a log event storage space, and a log event of the log event storage space carries a corresponding check code. The first writing module is configured to, in the case where it is detected that the binary temporary log file meets a preset file conversion condition, write header event content conforming to a binary log format specification in the header reserved space. The second writing module is configured to, according to configuration information of the header reserved space, write the binary log data and corresponding position information in the log event storage space to obtain a binary candidate log file. The conversion module is configured to convert the binary candidate log file into a binary formal log file. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.