Log file processing method, system, electronic device and readable storage medium

Abstract

This application discloses a log file processing method, system, electronic device, and readable storage medium. The method includes: generating a log, wherein the log is used to record operations performed on a database, and saving the log in a temporary file; and converting the temporary file containing the log into a log file according to a log file format. This application solves the problem of decreased database stability caused by the long time required to generate database log files, thereby improving database stability.

Description

Technical Field

[0001] The present application relates to the field of databases, and in particular to a log file processing method, system, electronic device, and readable storage medium. Background Art

[0002] Stability is an important indicator for measuring database products and one of the indicators that users value more when choosing database products.

[0003] When executing operations, the database can use events. Events are procedural database objects that the database calls at specific moments. Events are tasks that run according to a schedule. These events are called "scheduled events." When creating an event, it is created as a named database object containing SQL statements (or stored procedures) that are executed at a specific time or recur periodically. The execution of an event involves the generation of binary log files.

[0004] The binary log (Binlog) is a binary-formatted file used to record SQL statements used by users to update the database. For example, SQL statements that modify database tables and content will be recorded in the Binlog, but queries on database tables and other content will not be recorded. By default, the Binlog log is in binary format and cannot be viewed using commands in text viewing tools. Instead, it must be parsed and viewed using corresponding tools. Binlog can be used for database recovery. For example, when data is written to the database, the updated SQL statements are also written to the corresponding Binlog file. When backing up the database, you can only back up all the data for a period of time. However, if a database server failure is discovered after the backup, you can restore the database based on the update statements recorded in the Binlog.

[0005] If a transaction includes a large number of database operations, it is called a large transaction. During a large transaction, the large log volume can cause a long time to generate the Binlog file. During this time, other transactions cannot be committed, resulting in decreased database stability and even a possible database hang or crash. Summary of the Invention

[0006] The embodiments of the present application provide a log file processing method, system, electronic device and readable storage medium to at least solve the problem of decreased database stability caused by the long time spent in generating database log files.

[0007] According to one aspect of the present application, a log file processing method is provided, including: generating a log, wherein the log is used to record operations performed on a database, and saving the log in a temporary file; converting the temporary file containing the log into a log file according to the format of the log file.

[0008] According to another aspect of the present application, a log file processing system is also provided for processing log files using the above method, wherein the logs stored in the log files are used to record structured query language SQL statement information corresponding to operations performed on the database.

[0009] According to another aspect of the present application, an electronic device is also provided, comprising a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the above-mentioned method steps.

[0010] According to another aspect of the present application, a readable storage medium is provided, on which computer instructions are stored, wherein the computer instructions implement the above method steps when executed by a processor.

[0011] In an embodiment of the present application, a log is generated, wherein the log is used to record operations performed on the database, the log is saved in a temporary file, and the temporary file containing the log is converted into a log file according to the log file format. This application solves the problem of reduced database stability caused by the long time it takes to generate database log files, thereby improving database stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0013] Figure 1 is a flow chart of a log file processing method according to an embodiment of the present application;

[0014] Figure 2 is a schematic diagram of a temporary file reserved space according to an embodiment of the present application; and

[0015] Figure 3 Schematic diagram of log file processing according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0017] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0018] The following embodiments involve processing in a database, and the technical terms involved in the following embodiments are first explained below.

[0019] Database instance

[0020] A database service process running on a server to provide external database services is called a database instance. If you want to run multiple database service processes on a server, you can open different ports and run multiple database service processes through different port numbers. The multiple database service processes running are called multiple instances.

[0021] Database cluster

[0022] In many scenarios, a single database cannot meet the service needs. In this case, a database cluster is used. In a database cluster, there is usually a master database (Master) and at least one slave database (Slave). The slave database can be regarded as a backup of the master database. The database cluster can provide higher concurrent access.

[0023] SQL

[0024] SQL stands for Structured Query Language. SQL is a computer language used to store, retrieve, and modify data in relational databases. For example, SQL allows for database operations, including creating and deleting databases, querying and modifying records, and adding fields. SQL is the standard language for relational database services (RDS), and all relational database management systems use SQL as their standard processing language.

[0025] Data Manipulation Language (DML) is a programming language used for database operations, performing access tasks on database objects and data. It is typically a subset of the database-specific programming language SQL. For example, in the standard SQL language used in the information software industry, the core instructions are INSERT, UPDATE, and DELETE, representing insert, update, and delete, respectively.

[0026] Database events

[0027] An event is a procedural database object that the database calls at a specific moment. It's a task that runs according to a schedule. These events are called "scheduled events." When you create an event, you create it as a named database object containing a SQL statement (or stored procedure) that executes at a specific time or recurs periodically.

[0028] Database transactions

[0029] A transaction is a logical execution unit consisting of one or more SQL statements. A transaction can be likened to a container that contains a bunch of SQL statements. Either all of these statements are executed successfully, or none of them are executed successfully (atomicity).

[0030] GTID

[0031] The Global Transaction ID (GTID) is a unique identifier generated on the master database and bound to the transaction. This identifier is unique not only on the master database but also within the database cluster.

[0032] GTID=server_uuid:transaction_id

[0033] Example: 3E11FA47-71CA-11E1-9E33-C80AA9429562:1

[0034] A GTID is actually composed of an instance identifier (server_uuid, abbreviated as UUID) and a transaction identifier (transaction_id, abbreviated as TID). The UUID uniquely identifies a database instance. The TID represents the number of transactions committed on that instance and increases monotonically with each transaction submission. The UUID identifies the source database instance where the transaction was executed. The TID is a transaction sequence number generated on that primary database, starting at 1. 1-2 represents the second transaction, and 1-n represents n transactions. In this example, 3E11FA47-71CA-11E1-9E33-C80AA9429562 is the server_uuid of this database instance, and 1 is the transaction number of the first transaction submitted on this node. If 10 transactions were submitted, the GTID would be: 3E11FA47-71CA-11E1-9E33-C80AA9429562:1-10.

[0035] Binlog

[0036] The binary log (Binlog) is a binary-formatted file that records SQL statements used to update the database. For example, SQL statements that modify database tables and content are recorded in the Binlog, but queries of database tables and content are not recorded. By default, the Binlog is in binary format and cannot be viewed using commands in text-based tools. Instead, it can be parsed using appropriate tools.

[0037] Binlog can be used for database recovery. For example, when data is written to the database, the updated SQL statements are also written to the corresponding Binlog file. When backing up a database, it's possible to back up all data for a specific period of time. However, if a database server failure is discovered after the backup, the database can be restored based on the update statements recorded in the Binlog. Binlog is also used for data backup in master-slave databases.

[0038] Binlog cache

[0039] A cache in the database used to temporarily store event-level logical logs. When a transaction is committed, the contents stored in this cache are copied to the Binlog file.

[0040] Binlog flush

[0041] A binlog flush is used to refresh the binlog file. The flush logs command creates a new binlog file. Some databases also invoke the flush logs operation when the database is restarted. When the binlog file size reaches a preset value or the flush logs command is executed, the binlog rotates. A rotation event (Rotate_Event) is added to the end of the currently used binlog file, recording the name and location of the next binlog file.

[0042] During the database transaction commit process, Binlog entries are written to the Binlog file. Flushing the Binlog in the database requires a large number of I / O operations, which is slow and can pose risks to the database instance. For example, instances with strong I / O capabilities can withstand this I / O pressure, but the process is still lengthy and, during this time, the instance cannot submit other transactions, impacting availability. Alternatively, instances with poor I / O capabilities can face significant I / O pressure, leading to hangs or even crashes due to I / O bottlenecks. To explain the causes of I / O pressure, the following describes the Binlog file writing process.

[0043] The Binlog file writing process involves the Binlog cache and temporary Binlog files. Both the Binlog cache and temporary Binlog files are written during a transaction. Once the transaction commits, both the Binlog cache and temporary Binlog files are released. Furthermore, if a transaction contains multiple DML statements, they share the same Binlog cache and temporary Binlog files. The Binlog file writing process includes the following steps:

[0044] Step 1: Start the database transaction.

[0045] Step 2: Execute the DML statement in the database transaction. When the DML statement is executed for the first time, memory space is allocated for use as the Binlog cache.

[0046] Step 3: Logs generated during the execution of DML statements are continuously written to the Binlog cache.

[0047] Step 4: If the Binlog cache is full, the data in the Binlog cache is written to a temporary Binlog file and the Binlog cache is cleared. If the size of the temporary Binlog file exceeds the preset threshold (max_binlog_cache_size), an error is thrown.

[0048] Step 5: Transaction commit. All data in the Binlog cache and temporary Binlog files are written to the Binlog file, and the Binlog cache and temporary Binlog files are released.

[0049] In step 5 above, the temporary Binlog file data needs to be written to the Binlog file. Reading data from the temporary Binlog file and writing data to the Binlog file both require I / O operations, which can put I / O pressure on the database instance. An alternative implementation can limit the maximum I / O bandwidth to prevent Binlog flushing from causing excessive I / O pressure. However, this approach has significant drawbacks: it can slow down Binlog flushing, resulting in extended periods of time when the instance cannot commit transactions, and can also affect database stability.

[0050] In order to solve the above problems, a log file processing method is provided in the following embodiment: Figure 1 is a flow chart of a log file processing method according to an embodiment of the present application, such as Figure 1 As shown below, Figure 1 The steps involved in the method shown are described.

[0051] Step S102: Generate a log, wherein the log is used to record operations performed on the database.

[0052] For example, when performing operations on a database, the log file can record the operations performed on the database. Generally speaking, database operations can be performed using the SQL language. Therefore, the log file can record the SQL statement information corresponding to the operations performed on the database. In addition, other information can also be recorded as needed. For example, when performing operations on the database using a database client, the log file can also record information related to the database client, such as the type of database client, the name of the user performing the database operation, and information such as the IP address and port number.

[0053] When generating logs, you can generate a corresponding log entry for each database operation. After generating a log entry, the log's end position is recorded and the next log entry is generated. Of course, you can also generate logs for database transactions. Database transactions are explained below and will not be discussed here.

[0054] In this step, the log formats generated by different databases may vary, but all logs record SQL statement information. These logs may be called binary logs (Binlogs) in some databases. The following embodiments use Binlogs as an example. It should be noted that the following embodiments are also applicable to logs and log file generation in other formats. The following example illustrates how the logs in this step record SQL statement information.

[0055] Take the statement insert into tb values(1) as an example. This statement is used to insert data with the value 1 into the table named tb. After the statement is executed, the log records the statement. The log records are shown in Table 1 below:

[0056] Table 1

[0057] Event_type Server_id Info Query 10 BEGIN Table_map 10 table_id:389(db1.tb) Write_rows 10 table_id:389flags:STMT_END_F

[0058] The SQL statement information recorded in the log is expressed in Table 1 above. Event_type identifies the event type, Server_id identifies the database instance where the SQL statement was executed, and Info provides specific information. In the table, the first record has an event type of Query, with the specific information "BEGIN," which indicates the start of transaction execution. The second record has an event type of Table_map, which describes table metadata, such as the number of fields, field types, whether fields are null by default, and whether fields are resized. The specific information "table_id:389" identifies the table, and "db1.tb" indicates that this table is a tb table in database db1. The third record has an event type of Write_rows, which records insert data. The specific information "table_id:389flags:STMT_END_F" indicates the table to which the write occurred, with STMT_END_F indicating completion of execution. From this example, we can see that when recording logs, the execution of each statement will be split into multiple events for recording.

[0059] It should be noted that when recording the log, only DML statement information can be recorded, such as statements such as insert, delete, and update data. These statements will cause the data in the database to change, so after recording, the recorded log can be used to perform database synchronization or recovery operations. Statement information for reading database data can be omitted, which is conducive to controlling the number of logs and improving the efficiency of recording the log. In an optional embodiment, a configuration interface can also be provided to the user for which SQL statement information needs to be recorded, and the user can configure which SQL statements need to be recorded in the log when executing.

[0060] Step S104: Save the log in a temporary file.

[0061] Daily record generally can not directly be written in the log file, because can cause the repeated reading and writing to log file like this, speed is slower, so can at first log file be kept in temporary file, this temporary file can be the temporary file that is positioned at the internal memory, the temporary file that daily record is written in the internal memory is temporarily preserved, can not cause the repeated reading and writing log file like this, can improve processing speed.In addition, daily record is kept in temporary file and also is to allow the daily record that the Binlogevent that same database affairs is corresponding can be continuously present in the log file, therefore, daily record is at first kept in temporary file, in step S106, changes again.

[0062] In some cases, can also open up cache space in advance, use this part cache space to come cache journal, in this case, when needs generate log file, can earlier log book be taken out from cache space and then the log book in the cache space is written in temporary file, then perform following step S106.In another optional embodiment, can also cache space and temporary file be used together, at first cache space (for example, Binlog cache) is set, the log book that database operation produces is first kept in the cache space, after cache space is full, the log book in the cache space can be transferred to in temporary file, can combine cache space and temporary file like this.

[0063] Step S106: converting the temporary file storing the log into a log file according to the format of the log file.

[0064] In this step, the log files are no longer read from the temporary file and then written to the log file. This reading and then writing action would cause a large number of IO operations, which would result in a long time to write the log file, affecting the stability of the database. By directly converting the temporary file into a log file, a large number of IO operations can be avoided and the log file can be directly obtained. Therefore, the above steps solve the problem of database stability degradation caused by the long time spent on generating database log files, thereby improving the stability of the database.

[0065] It should be noted that if the method of reading logs from temporary files and then writing them to log files is adopted, then the database or the operating system where the database is located does not need to manage the temporary files. Figure 1 The steps shown in the figure convert a temporary file into a log file. In an optional embodiment, the temporary file can be registered as a formal file in the operating system or database. The formal file can be managed by the file management system of the operating system or database, and its life cycle is the same as that of other files managed by the operating system or database. By performing file management on the temporary file, it is more convenient to convert the temporary file into a log file.

[0066] Considering that there are some differences between temporary files and log files, when saving temporary files as log files, the temporary files need to be converted. For example, log files have their own naming rules. When converting temporary files into log files, the temporary files need to be renamed according to the naming rules of log files.

[0067] For example, in a MySQL database, log files are named mysql-bin.000016, where mysql represents the name of the database software, bin represents the binary system, and 000016 is an ascending number. When generating log files, you can obtain the name of the last generated log file. If the name of the last generated log file is mysql-bin.000016, you can rename the temporary file to mysql-bin.000017, which is the name of the new log file.

[0068] In another optional embodiment, in some database software, the log file may also include some information that is not in the temporary file. In this case, when converting the temporary file into a log file, it is also necessary to fill in the missing information in the temporary file. That is, in this optional embodiment, converting the temporary file containing the log into the log file according to the format of the log file may include the following steps: generating the information that is missing from the temporary file compared to the log file according to the format of the log file; saving the missing information in the corresponding position of the temporary file; and converting the temporary file containing the missing information into the log file. Normally, the missing information is basically the information used to connect other log files. In this case, the missing information can be filled in according to the format of the log file in which the missing information is stored.

[0069] When supplementing the missing information in the temporary file, the missing information can be inserted into the temporary file according to its corresponding position in the log file. Or in order to supplement the missing information more conveniently, the position where the missing information is located can be reserved in advance in the process of saving the log in the temporary file, and the reserved position will not be used to save the log, so that the missing information can be directly saved in the corresponding reserved position in the future. Saving the log in the temporary file may include the following steps: reserving a space of the size of the missing information at the corresponding position in the temporary file according to the size of the missing information and the position in the log file; and saving the log in the part of the temporary file except the reserved space. This optional implementation method can achieve space reservation and improve the efficiency of supplementing missing information to the temporary file.

[0070] Through the above optional implementation, a space of a predetermined size is reserved at the corresponding position of the temporary file, wherein the space of the predetermined size is used to store the information missing from converting the temporary file to the log file; the log is stored in the portion of the temporary file excluding the reserved space. Then, the missing information in the space of the predetermined size is stored in the corresponding position of the temporary file; the temporary file storing the missing information is converted to the log file. This improves the efficiency of the conversion from temporary files to log files. This optional implementation can be used to store the missing parts of various temporary files compared to log files.

[0071] For example, in some database systems, the log file will record some information in the header, that is, in this example, the information missing from the temporary file compared to the log file is the header information. The header information can be various information, such as the header information includes at least one of the following: information used to describe the log file format, identification information of the database transaction where the log recorded at the end of the previous log file is located, and identification information of the database transaction to which the current operation on the database belongs. In this example, the header information corresponding to the temporary file is generated according to the format of the log file; the header information is configured at the beginning of the temporary file; and the temporary file configured with the header information is converted into the log file. In order to be able to add the header information to the historical file more conveniently, a space of the size specified can be reserved at the beginning of the temporary file according to the size of the header information; and the log is saved in the part of the temporary file other than the reserved space.

[0072] Figure 2 Schematic diagram of temporary file reserved space according to an embodiment of the present application, such as Figure 2 As shown, along the timeline (Timeline), a transaction (Trx is short for transaction) is first executed (Execute) and then committed (Commit). During transaction execution, space is reserved (Reserve Space) in a temporary file (Binlog Cache File). Logs are written line by line (Write Rows) starting after the reserved space. Logs are written in events (Events). The user generates log files (Binlog Files) for each log file. Log files can be numbered sequentially, i.e., Binlog001, Binlog002, Binlog003, Binlog004, and so on. After rotating the log files, a Write Rotate event is added to the end of the currently used Binlog file, recording the name and location of the next Binlog file. Figure 2The Binlog cache shown in the figure is primarily used to store Binlogs generated during transaction execution. When the size exceeds a certain limit, it is transferred to a temporary file managed by the operating system. When writing temporary files, the Binlog cache reserves 4KB of space in the file header to supplement the header with the Format Description (FD), the Previous GTID (Pre GTID), and the GTID when the file is converted to a Binlog file. To read the temporary file, you need to skip the reserved file header. The following describes FD, Pre GTID, and GTID.

[0073] A global transaction ID (GTID) is a unique identifier generated by the master database and bound to a transaction. Not all database systems support this feature. Database systems that support this feature also record the GTID when recording Binlogs.

[0074] Let's take the "insert into tb values(1)" statement as an example. This statement is used to insert data with a value of 1 into the table named tb. After the statement is executed, the log records the statement. If GTID is supported, the log records are as shown in Table 2 below:

[0075] Table 2

[0076]

[0077] In Table 2, format description is used to indicate various formats. For example, the binlog version is 4, and the service version is 5.7.19-17-debug-log. Previous_GTID is used to indicate the GTID sequence number at the end of the previous binlog file. GTID identifies the sequence number of the current GTID, that is, "b0d850c2-dbd0-11e9-90c3-080027b8bded:1".

[0078] In the above example, the log can be first saved in a pre-opened cache (for example, Binlog cache), and after the cache is full, the log is saved in the temporary file (the temporary file can also be configured in Binlogcache); considering that there may be logs in the cache, it is necessary to read the logs saved in the cache from the cache, save the read logs in the temporary file, and convert the temporary file into the log file according to the format of the log file. It should be noted that, as an optional implementation method, the temporary file can be registered as a formal file in the operating system where the database is located or in the database, and the formal file can be managed by the file management system in the operating system or the database, and its life cycle is the same as the life cycle of other files managed by the operating system or the database. In this way, a temporary file can be directly generated as a log file.

[0079] In the above-mentioned embodiment, the log file is generated according to the database transaction. For example, one database transaction may correspond to one log file. The database transaction may include multiple SQL statements for operating the database. Different database transactions may include different numbers of SQL statements for operating the database. Therefore, transactions may be large or small. In an optional embodiment, the method of converting a temporary file into a log file may be more efficient when the transaction is relatively large. That is, in this optional embodiment, the log is saved in a temporary file, and the temporary file is converted into a log file according to the format of the log file, including: when the database is operated through a database transaction, determining whether the size of the log generated by the database transaction exceeds a threshold, and if it exceeds the threshold, saving the log in the temporary file; and converting the temporary file into the log file when the database transaction is committed.

[0080] In this alternative embodiment, temporary files can be managed by the database, eliminating the need for interaction with the database's operating system. In this case, the database generates the temporary files, which can be managed by the database kernel. That is, the temporary files of the binlog cache no longer use temporary files provided by the operating system, but are instead managed by the database kernel and can be deleted or converted to formal log files as needed.

[0081] In another optional embodiment, if the size of the log generated by the database transaction does not exceed a threshold, the log is saved in a temporary file. When the database transaction is committed, the log is read from the temporary file and written to a pre-generated log file. This method requires I / O reading and writing, but since the log is relatively small, the efficiency of this method is acceptable. In this optional embodiment, a temporary file provided by the operating system where the database resides can be used as the temporary file for storing the log.

[0082] When reading logs from a temporary file and then writing them to a log file, the process of writing to the log file can be divided into two parts during actual execution. These two parts can be called writing (write) and syncing (sync). These two steps are explained below.

[0083] write: Write the log to a temporary file in memory.

[0084] sync: Logs are read from temporary files and written to the log file for persistence to disk only when the operating system executes sync. The sync_binlog parameter can be used to control the timing of writes and synchronization. For example, sync_binlog = 0 means that each transaction is only written, without sync, and the binlog is not persisted. sync_binlog = 1 means that a sync is performed with each transaction. sync_binlog = N means that each transaction is written, but sync is performed after N transaction commits for persistence.

[0085] If temporary files are converted directly into log files, the sync phase in the Binlog commit process is no longer necessary for large transaction commits because the Binlog cache is automatically synced (i.e., converted into log files) before the commit phase. This process avoids the large amount of read I / O generated by reading temporary binlog cache files and the large amount of write I / O generated by writing binlog files. After this optimization, no binlog I / O operations are required during the large transaction commit phase. Figure 3 Schematic diagram of log file processing according to an embodiment of the present application. Figure 3 As shown, after the database transaction is committed (Ordered_

[0086] commit), the temporary file needs to be converted into a log file. When executing this function, the size of the temporary file can be checked first. If the temporary file is larger than a threshold (such as 100M bits), the function of converting the temporary file into a log file is executed. If the temporary file is less than or equal to the threshold, the original logic is executed (i.e., the log is read from the temporary file and then written to the log file). This check can be called the first check. After checking that the size of the temporary file is larger than the threshold, a log lock is acquired. Only after the log lock is acquired can the log file be operated. After the log lock is acquired, a second check can be performed. For example, it can be checked whether the space reserved in the temporary file is sufficient to write the header information. If not, the log lock is released and the original logic is executed. If the space reserved in the temporary file is sufficient to write the header information, the header information is written to the temporary file. The header information can include: FD events, PreGTID events, GTID events, etc. After the header information is written to the temporary file, the temporary file is converted to persistence (Rotate with Persist). The persistent conversion process is as follows: first, the previously opened log file is closed. Then, the temporary file is renamed and persisted as the log file. The log file is then opened and indexed (seeked). Finally, the log file is added to the Binlog index. After the log file is generated, the post-flush process, the post-sync process, the commit process, and the forced rotation process are identical to those in the prior art and will not be further described here.

[0087] Figure 3 In the MySQL Binlog commit process shown in the figure, the flush stage generates GTID events, reads the contents of the Binlog cache, and writes them to the Binlog file. For large transaction commits, the flush stage only generates GTID events and writes them to the Binlog cache temporary file. A Binlog rotate operation is then performed to switch the Binlog from the current file to the temporary file in the Binlog cache. This process includes adding the contents of the Binlog file header and renaming the temporary file. The temporary file in the Binlog cache is then converted into a log file. Figure 3In the process shown, the Binlog cache is modified to enable direct conversion to Binlog. The Binlog commit process is also modified so that during the large transaction commit process, the Binlog cache is directly converted to Binlog files without having to write to Binlog files. This thoroughly optimizes Binlog I / O during the large transaction commit phase, making the large transaction commit process fast and stable.

[0088] Through the above optional implementation, the method of directly converting binlog cache temporary files into binlog files is used to avoid I / O operations during the large transaction submission phase, optimize the stability of large transaction submission, and thus improve the stability of the database. In an optional implementation, a system is also provided, which can be referred to as a log file processing system, used for processing log files using the above method. The logs stored in the log files are used to record Structured Query Language (SQL) statement information corresponding to operations performed on the database. This log file processing system can be located in various database systems.

[0089] In this embodiment, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the method in the above embodiment.

[0090] The above program can be run in the processor, or it can be stored in the memory (or computer-readable medium), which includes permanent and non-permanent, removable and non-removable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more blocks can be implemented by different modules corresponding to different steps.

[0091] This embodiment provides such a device or system. The device is referred to as a log file processing device and includes: a generation module for generating a log, wherein the log is used to record operations performed on a database; a storage module for storing the log in a temporary file; and a conversion module for converting the temporary file containing the log into a log file according to a log file format.

[0092] The system or device is used to implement the functions of the method in the above-mentioned embodiment. Each module in the system or device corresponds to each step in the method, which has been explained in the method and will not be repeated here.

[0093] Optionally, the saving module is used to reserve a space of a predetermined size at a corresponding position of the temporary file, wherein the space of the predetermined size is used to save information missing from converting the temporary file into the log file; and save the log in the part of the temporary file other than the reserved space.

[0094] Optionally, the conversion module is configured to save the missing information in the space of the predetermined size in a location corresponding to the temporary file; and convert the temporary file storing the missing information into the log file.

[0095] Optionally, the information missing from the temporary file compared to the log file is header information, and / or the header information includes at least one of the following: information used to describe the log file format, identification information of the database transaction in which the log recorded at the end of the previous log file is located, and identification information of the database transaction to which the current operation performed on the database belongs.

[0096] Optionally, the generating module is configured to generate a corresponding log for each operation performed on the database; after generating a log, the generating module records the end position of the log and continues to generate the next log.

[0097] Optionally, the saving module is used to determine whether the log size generated by the database transaction exceeds a threshold when the database is operated through the database transaction, and if it exceeds the threshold, save the log in the temporary file; the conversion module is used to convert the temporary file into the log file when the database transaction is committed.

[0098] Optionally, a generation module is also included, which is used to generate the temporary file through the database before saving the log in the temporary file, wherein the temporary file is registered as a formal file in the operating system where the database is located or in the database. The formal file can be managed by the file management system in the operating system or the database, and its life cycle is the same as the life cycle of other files managed by the operating system or the database.

[0099] Optionally, the saving module is also used to save the log in a temporary file when it is determined that the log size generated by the database transaction does not exceed a threshold; the device also includes: a reading and writing module, used to read the log from the temporary file when the database transaction is committed, and write the read log into a pre-generated log file.

[0100] Optionally, the saving module uses a temporary file provided by the operating system where the database is located as a temporary file for saving the log.

[0101] Optionally, the saving module is used to save the log in a pre-opened cache, and after the cache is full, save the log in the temporary file;

[0102] The conversion module is used to read the log stored in the cache from the cache, save the read log in the temporary file, and convert the temporary file into the log file according to the format of the log file.

[0103] Optionally, the log is a binary log.

[0104] The above optional implementation solves the problem of decreased database stability caused by the long time spent on generating database log files, and implements a method for quickly and stably submitting large transactions. This method is of great significance for improving the stability of database instances. The impact of submitting large transactions is greatly reduced compared to existing technologies, thereby improving the stability of the database.

[0105] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A log file processing method, comprising: Generate a log, wherein the log is used to record operations performed on the database, Determine whether the log size exceeds the set threshold; If the log size exceeds the set threshold, the log is saved in a temporary file, including: Check the size of the temporary file; when the size of the temporary file exceeds a threshold, check whether the reserved space of the temporary file is sufficient to write the header information; if the reserved space of the temporary file is sufficient to write the header information, write the header information to the temporary file; the header information includes: FD event, PreGTID event, GTID event; Converting the temporary file storing the log into a log file according to the format of the log file includes: converting the temporary file into the log file; Saving the log in the temporary file includes: Reserving a space of a predetermined size at a corresponding position of the temporary file, wherein the space of the predetermined size is used to store information missing from converting the temporary file into the log file; and storing the log in a portion of the temporary file excluding the reserved space; Converting the temporary file storing the log into the log file according to the format of the log file includes: Saving the missing information in the space of the predetermined size in a location corresponding to the temporary file; and converting the temporary file storing the missing information into the log file; Saving the log in a temporary file includes: saving the log in a pre-opened cache, and after the cache is full, saving the log in the temporary file; Converting the temporary file storing the log into the log file according to the log file format includes: reading the log stored in the cache from the cache, saving the read log in the temporary file, and converting the temporary file into the log file according to the log file format.

2. The method according to claim 1, wherein The information missing from the temporary file compared to the log file is header information, and / or the header information includes at least one of the following: information used to describe the log file format, identification information of the database transaction in which the log recorded at the end of the previous log file is located, and identification information of the database transaction to which the current operation performed on the database belongs.

3. The method according to claim 1, wherein Generating the log includes: Each operation on the database generates a corresponding log; After generating a log, the end position of the log is recorded and the next log is generated.

4. The method according to claim 1, wherein The log is saved in a temporary file, and the temporary file is converted into a log file according to the format of the log file, including: In the case of operating the database through a database transaction, determining whether a log size generated by the database transaction exceeds a threshold, and if so, saving the log in the temporary file; When the database transaction is committed, the temporary file is converted into the log file.

5. The method according to any one of claims 1 to 4, wherein Before saving the log in the temporary file, the method further includes: The temporary file is generated by the database, wherein the temporary file is registered as a formal file in the operating system where the database is located or in the database, and the formal file can be managed by a file management system in the operating system or in the database.

6. The method according to claim 4, wherein: Also includes: If it is determined that the size of the log generated by the database transaction does not exceed a threshold, saving the log in a temporary file; When the database transaction is committed, the log is read from the temporary file and written into a pre-generated log file.

7. The method according to claim 6, wherein: Before writing the read log into the pre-generated log file, the method further includes: A temporary file provided by the operating system where the database is located is used as a temporary file for storing the log.

8. A log file processing system, configured to process a log file using the method according to any one of claims 1 to 7, wherein the log stored in the log file is used to record structured query language (SQL) statement information corresponding to operations performed on a database.

9. An electronic device comprising a memory and a processor; wherein: The memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps according to any one of claims 1 to 7.

10. A readable storage medium having computer instructions stored thereon, wherein: When the computer instructions are executed by a processor, the method steps according to any one of claims 1 to 7 are implemented.

Images