Log processing method, computing device, storage medium and program product

Abstract

The embodiment of the invention provides a log processing method, computing equipment, a storage medium and a program product. The method comprises the following steps: under the condition that a target transaction of a database is detected to meet a target condition, storing a temporary file generated by the target transaction to a storage node; the temporary file stores a binary log for the database; and in response to a transaction submission operation of the target transaction, processing the temporary file in the storage node to generate a binary log file. According to the technical scheme provided by the embodiment of the invention, the transaction submission efficiency is improved.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of computer technologies, and in particular, to a log processing method, a computing device, a storage medium, and a program product. Background Art

[0002] Binary logs (binlogs) are used to record operation information such as insertions, deletions, and modifications performed on a database, and are an important type of log in the database. Data recovery and data replication can be achieved based on binary logs. A logical unit composed of a series of operations performed on various data in the database is called a transaction, and binary logs are generated during the execution of the transaction.

[0003] In the prior art, the binary logs generated during the execution of a transaction are first written into a cache. Since all update operations on data are permanently saved to the database only when the transaction is committed, the binary logs are written into a binary log file (binlog file) only when the transaction is committed. In a current data processing architecture, operations such as transaction processing and log generation are completed by computing nodes, and the database and the binary log file are stored in storage nodes. During the execution of a transaction, the cache may overflow. At this time, the binary logs are saved to a local temporary file of the computing node, and the binary logs in the cache or the temporary file are written into the binary log file of the storage node when the transaction is committed.

[0004] If multiple transactions submit and perform binary log file writing operations simultaneously, the binary logs in the temporary files are written into the binary log file serially by multiple transactions. After one transaction finishes writing, other transactions can start writing. For large transactions with a long running time and a large amount of data being operated on, writing the binary logs from the temporary file into the binary log file will consume a large amount of time. Therefore, the submission of other transactions will be blocked, affecting the transaction submission efficiency. Summary of the Invention

[0005] Embodiments of the present application provide a log processing method, a computing device, a storage medium, and a program product to solve the problem of low transaction submission efficiency in the prior art.

[0006] In a first aspect, embodiments of the present application provide a log processing method, including:

[0007] When it is detected that a target transaction of the database meets a target condition, storing a temporary file generated by the target transaction in a storage node; the temporary file stores binary logs for the database;

[0008] In response to a transaction submission operation of the target transaction, processing the temporary file in the storage node to generate a binary log file.

[0009] Optionally, the processing of the temporary file in the storage node to generate a binary log file in response to the transaction commit operation for the target transaction includes:

[0010] In response to the transaction commit operation for the target transaction, write a transaction identifier in the temporary file and rename it according to a target naming method to serve as the binary log file.

[0011] Optionally, the writing of a transaction identifier in the temporary file and renaming it according to a target naming method to serve as the binary log file in response to the transaction commit operation for the target transaction includes:

[0012] In response to the transaction commit operation for the target transaction, add a target space at the end of the temporary file to write the transaction identifier in the target space;

[0013] Rename the temporary file according to the target naming method to generate the binary log file.

[0014] Optionally, the writing of a transaction identifier in the temporary file and renaming it according to a target naming method to serve as the binary log file in response to the transaction commit operation for the target transaction includes:

[0015] In response to the transaction commit operation for the target transaction, write the transaction identifier in the target space reserved at the head of the temporary file;

[0016] Rename the temporary file according to the target naming method to generate the binary log file.

[0017] Optionally, the method further includes:

[0018] Receive a file acquisition request sent by a requester for the target transaction;

[0019] Acquire the binary log file corresponding to the target transaction;

[0020] According to the transaction identifier in the target space of the binary log file and the binary log entries in the binary log file, re-encapsulate and generate a first file in a target format;

[0021] Provide the first file to the requester.

[0022] Optionally, the processing of the temporary file in the storage node to generate a binary log file in response to the transaction commit operation for the target transaction includes:

[0023] In response to the transaction commit operation for the target transaction, rename the temporary file to obtain a target file;

[0024] Generate a binary log file according to the transaction identifier of the target transaction and the file index information of the target file in a target format; the binary log file is used to obtain the file index information and obtain the target file according to the file index information.

[0025] Optionally, the method further includes:

[0026] Receive a file acquisition request sent by a requester for the target transaction;

[0027] Obtain the binary log file corresponding to the target transaction;

[0028] Obtain the target file according to the file index information in the binary log file;

[0029] Repackage the transaction identifier in the binary log file and the target file to generate a second file in the target format;

[0030] Provide the second file to the requester.

[0031] Optionally, the method further includes:

[0032] In the case where the cache corresponding to the target transaction reaches a predetermined capacity, write the binary log in the cache into a temporary file, and write the binary log currently generated by the target transaction into the cache.

[0033] Optionally, the processing the temporary file in the storage node to generate a binary log file in response to the transaction commit operation of the target transaction includes:

[0034] In response to the transaction commit operation of the target transaction, write the binary log in the cache corresponding to the target transaction into the temporary file in the storage node;

[0035] Process the temporary file in the storage node to generate a binary log file.

[0036] Optionally, the processing the temporary file in the storage node to generate a binary log file in response to the transaction commit operation of the target transaction includes:

[0037] In response to the transaction commit operation of the target transaction, apply for a resource lock and, in the case of obtaining the resource lock, process the temporary file in the storage node to generate a binary log file;

[0038] Release the resource lock when the processing of the temporary file to generate a binary log file is completed.

[0039] Optionally, the method further includes:

[0040] When it is detected that the execution of the target transaction fails, delete the temporary file from the storage node.

[0041] Optionally, the database is a cloud-native database; the storage node is a network storage device.

[0042] Optionally, the target transaction of the database meeting the target conditions includes:

[0043] The data volume of the binary log in the temporary file generated by the target transaction of the database reaches a predetermined data volume;

[0044] and / or, the execution duration of the target transaction reaches a predetermined duration;

[0045] and / or, the resources occupied by the target transaction reach a predetermined resource.

[0046] In a second aspect, an embodiment of the present application provides a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the log processing method as described in the first aspect above.

[0047] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a computer, it implements the log processing method as described in the first aspect above.

[0048] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a computer, it implements the log processing method as described in the first aspect above.

[0049] In an embodiment of the present application, when the target transaction of the database meets the target conditions, the temporary file generated by the target transaction is stored in the storage node, where the temporary file stores the binary log for the database. After that, in response to the transaction commit operation of the target transaction, the temporary file in the storage node can be directly processed to generate a binary log file. Through the technical solution of the embodiment of the present application, before the transaction is committed, the binary log has been written into the storage node, and the time consumed for writing the binary log into the storage node is dispersed during the transaction execution process. Therefore, it is possible to avoid the large amount of write duration caused by writing the binary log during transaction commit and blocking the commit of other transactions, and the commit duration of the target transaction can be shortened, improving the efficiency of transaction commit.

[0050] These aspects or other aspects of the present application will be more clearly understood in the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0052] Figure 1 It shows a flowchart of an embodiment of a log processing method provided by the present application;

[0053] Figure 2 It shows a schematic structural diagram of an embodiment of a database processing system provided by the present application;

[0054] Figure 3 It shows a schematic diagram of a computing node processing transaction execution and submission in an actual application of the embodiments of the present application;

[0055] Figure 4 It shows a schematic structural diagram of an embodiment of a log processing device provided by the present application;

[0056] Figure 5 It shows a schematic structural diagram of an embodiment of a computing device provided by the present application. Detailed implementation manners

[0057] To enable those skilled in the art of the present technology to better understand the solutions of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application.

[0058] In some processes described in the specification, claims, and the above-mentioned drawings of the present application, there are multiple operations that appear in a specific order. However, it should be clearly understood that these operations can be executed not in the order in which they appear in this document or in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and these operations can be executed in sequence or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit that "first" and "second" are of different types.

[0059] The technical solution of the embodiment of the present application can be applied to the transaction processing scenario of a database. The inventor found in the process of implementing the present application that since users have high requirements for the performance stability of the database, especially in the cloud-native architecture, and transaction submission, especially large transaction submission, will cause database jitter. When a large transaction is submitted, not only its own submission will be very slow, but other transactions being submitted at this time also have to wait for the submission of this large transaction, resulting in an impact on the overall transaction submission efficiency. The inventor further studied and found that the main reason for the slow transaction submission is that a large number of binary logs are generated during the transaction execution process. These binary logs need to be sequentially written into the binary log file during transaction submission, and the writing process is very slow. And combined with the description of the background technology, it can be known that the binary logs generated during the transaction execution process will be first written into the cache. After the cache overflows, at this time, the binary logs will be saved to the local temporary file of the computing node. The inventor further studied and found that the temporary file is often relatively large, and a lot of the writing time is spent on writing the binary logs in the temporary file into the binary log file. In addition, since the temporary file needs to occupy local storage resources, and the local storage resources will limit the capacity of the binary logs, it may cause some transactions to fail to execute. In addition, the temporary files generated by transactions will also squeeze the local storage resources that other tasks can use, resulting in an impact on the device performance.

[0060] To solve the problem of low transaction submission efficiency, the inventor proposed the technical solution of the present application after a series of studies. In the embodiment of the present application, when it is detected that the target transaction of the database meets the target condition, the temporary file generated by the target transaction is stored in the storage node, where the temporary file stores the binary logs for the database. After that, in response to the transaction submission operation of the target transaction, the temporary file in the storage node can be directly processed to generate a binary log file. Through the technical solution of the embodiment of the present application, before the transaction is submitted, the binary logs have been written into the storage node, and the time consumed for writing the binary logs into the storage node is dispersed during the transaction execution process. Therefore, it is possible to avoid the large amount of writing time and blocking of other transaction submissions caused by writing binary logs during transaction submission, shorten the submission duration of the target transaction, and improve the transaction submission efficiency.

[0061] In addition, since the temporary file will be stored in the storage node during the transaction execution process, it is not necessary to occupy a large amount of local storage resources, which can avoid transaction execution failures caused by capacity limitations and will not affect other task processing, improving the device performance.

[0062] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0063] The technical solutions of the embodiments of the present application can be applied to a database system, which may include at least one computing node and a storage node. Among them, the database is deployed in the storage node.

[0064] In a practical application, the technical solutions of the embodiments of the present application can be applied to a cloud-native scenario, and the database may refer to a cloud-native database. A cloud-native database is a service built, deployed, and distributed through a cloud platform. This cloud-native attribute is its biggest feature compared to other types of databases. As a cloud platform, a cloud-native database is distributed in the form of PaaS (Platform-as-a-Service), and is often referred to as DBaaS (DataBase-as-a-Service). Users can use this platform for various purposes, such as storing, managing, and extracting data.

[0065] A cloud-native database is usually implemented by installing database software on top of cloud infrastructure, which enables the cloud-native database to have direct accessibility and runtime scalability that traditional databases do not have.

[0066] In a cloud-native scenario, the database system may be a database processing system built and deployed in a cloud platform. The above storage node may be a network storage device, such as a cloud disk, etc.; the computing node may be a cloud server, etc.

[0067] Among them, the computing node can be responsible for processing database queries and transactions, and the storage node can be responsible for storing database data. A transaction is a set of indivisible operations on a database, which may include operations such as inserting, updating, and deleting database tables. The computing node can communicate with the storage node by sending I / O (Input / Output) requests through an interface. This interface can be a set of APIs (Application Programming Interface) that define operation specifications such as data reading, writing, and updating, or a communication protocol based on network protocols and data transmission formats, providing functions such as data reading, writing, and updating.

[0068] During the execution of a transaction, the computing node coordinates and manages each operation, and at the same time saves the binary log generated by the target transaction to the local cache of the computing node. When the cache overflows, the binary log is saved to the local temporary file of the computing node. When the transaction is committed, the binary log generated by the target transaction is written into the binary log file of the storage node to ensure data consistency and reliability.

[0069] It should be noted that the embodiments of the present application may involve the use of user data. In practical applications, user-specific personal data can be used in the solutions described herein within the scope permitted by applicable laws and regulations (for example, with the user's explicit consent, and giving the user a practical notice, etc.) and in compliance with the requirements of the applicable laws and regulations of the country where it is located.

[0070] It should be noted that the technical solutions of the embodiments of the present application are applicable to a network virtual environment. The users described generally refer to "virtual users". Real users can register user accounts on the server through the registration method to obtain user identities in the network environment.

[0071] The implementation details of the technical solutions of the embodiments of the present application are elaborated in detail below.

[0072] Figure 1 It is a flowchart of an embodiment of a log processing method provided by the present application. The technical solution of this embodiment can be executed by a computing node. The method may include the following steps:

[0073] 101: When it is detected that the target transaction of the database meets the target condition, store the temporary file generated by the target transaction to the storage node.

[0074] Among them, the target transaction may refer to any transaction opened in the database.

[0075] Among them, the target condition can be used to identify whether the target transaction is a large transaction. That is, the technical solutions of the embodiments of the present application can be executed for large transactions to ensure database performance. Of course, the technical solutions of the embodiments of the present application can also be applicable to all transactions. At this time, the target condition can be empty.

[0076] Among them, the target transaction meeting the target condition may include one or more of the following implementation manners: the data volume of the binary log in the temporary file generated by the target transaction reaches a predetermined data volume; or, the execution duration of the target transaction reaches a predetermined duration; or, the resources occupied by the target transaction reach a predetermined resource, etc.

[0077] Therefore, as an optional manner, the target transaction of the database meeting the target condition can be: the data volume of the binary log in the temporary file generated by the target transaction of the database reaches a predetermined data volume.

[0078] As described in the foregoing, since binary logs are continuously generated during the execution of the target transaction, the binary logs are first written into the cache. Once the cache reaches the predetermined capacity, the binary logs in the cache are written into a local temporary file, and the cache is cleared to continue writing newly generated binary logs. At this time, the data volume of the binary logs in the temporary file can be detected. If the data volume of the binary logs in the temporary file reaches the predetermined data volume, it indicates that the target transaction is a large transaction.

[0079] As another optional method, the target condition that the target transaction of the database satisfies can be: the execution duration of the target transaction of the database reaches the predetermined duration.

[0080] As yet another optional method, the target condition that the target transaction of the database satisfies can be: the resources occupied by the target transaction of the database reach the predetermined resources.

[0081] In addition, it can also be determined whether the target transaction is a large transaction by combining multiple conditions such as the data volume of the binary logs in the temporary file reaching the predetermined data volume, the execution duration of the target transaction reaching the predetermined duration, and the resources occupied by the target transaction reaching the predetermined resources. This application does not limit this.

[0082] Among them, the temporary file stores the binary logs for the database. The binary logs record the operations performed by the target transaction on the database, such as insert, delete, update, etc. operations. The binary logs can be composed of binary log entries (BinlogEvent), and each entry records an operation on the database. At the same time, the temporary file can also record the statement execution time.

[0083] Among them, during the execution of the target transaction, the temporary file generated by the target transaction is first stored in the local disk of the physical host where the computing node is running. The temporary file generated by the target transaction can be stored in the storage node when it is detected that the target transaction of the database satisfies the target condition.

[0084] Since the temporary file will be stored in the storage node during the execution of the transaction, it does not need to occupy too much local storage resources, can avoid transaction execution failure caused by capacity limitations, and does not affect other task processing, which can improve resource utilization.

[0085] Optionally, in order to improve resource utilization, after the temporary file generated by the target transaction is stored in the storage node, the temporary file in the local disk can be cleared, and the clearing timing is not specifically limited in this application.

[0086] 102: In response to the transaction commit operation of the target transaction, process the temporary file in the storage node to generate a binary log file.

[0087] Among them, in practical applications, the binary log file can be a file type with a specific file format. It is a persistent log file that will be retained for a long time for data recovery, data backup, data replication, and auditing when data problems occur.

[0088] Among them, data recovery, for example, means that if the database loses data due to certain reasons, the data can be restored by replaying the binary log. By re-executing the operation statements in the binary log file, the data in the database can be reconstructed.

[0089] Data backup, for example, means that the binary log file can record all operations in the database, including insert, delete, and update operations. Therefore, it can be used to back up the data in the database. By backing up the binary log, the database can be restored to a specific point in time to recover accidentally deleted data or solve database corruption problems.

[0090] Data replication, for example, means that the binary log file can be used to implement master-slave replication of the database. In the master database, all operations are recorded in the binary log file, and then the logs in the binary log file are used on the slave database to synchronize these operations to the slave database to maintain data consistency between the master and slave databases.

[0091] Auditing, for example, means that the binary log file is also used for auditing purposes. By reviewing the logs, it is possible to see when the data was changed and how the changes were made.

[0092] The inventors found during the implementation of this application that in practical applications, when the binary log file usually exceeds a certain size, a new file will be created to continue writing the binary logs of different transactions. Temporary files, especially those of large transactions, are usually relatively large. Therefore, in the embodiments of this application, the temporary file can be directly processed to generate a binary log file, which can correspond only to the target transaction and store the binary log of the target transaction.

[0093] In this embodiment, when the target transaction of the database meets the target conditions, the temporary file generated by the target transaction is stored in the storage node. Among them, the temporary file stores the binary log for the database. After that, in response to the transaction commit operation of the target transaction, the temporary file in the storage node can be directly processed to generate a binary log file. Through the technical solution of the embodiments of this application, before the transaction is committed, the binary log has been written to the storage node, and the time consumed for writing the binary log to the storage node is dispersed during the transaction execution process. Therefore, it is possible to avoid the large amount of write time consumed by writing the binary log when the transaction is committed and blocking the submission of other transactions, and the submission time of the target transaction can be shortened, improving the efficiency of transaction submission.

[0094] Among them, there are multiple implementation manners for processing the temporary files in the storage node to generate binary log files:

[0095] As an alternative implementation manner, in response to the transaction commit operation of the target transaction, processing the temporary files in the storage node to generate binary log files may include: in response to the transaction commit operation of the target transaction, writing a transaction identifier in the temporary file and renaming it according to a target naming manner to be used as a binary log file.

[0096] Since the temporary file only saves the binary log, and different transactions need to be distinguished by the transaction identifier, which may be, for example, a GTID (Global Transaction Identifier), and can be used to uniquely identify a transaction for easy tracking and management of the transaction. And the transaction identifier is only generated when the transaction is committed. Therefore, in the embodiments of the present application, the format of the temporary file can be directly changed, the transaction identifier can be written in the temporary file, and it can be renamed to generate a binary log file.

[0097] Since the temporary file will be deleted in case of transaction execution failure, renaming the temporary file can effectively distinguish the temporary file and the binary log file, so that the binary log file can be persistently saved to avoid being accidentally deleted, etc.

[0098] Among them, the target naming manner can be preset and is different from the naming manner of the temporary file. Optionally, the target naming manner may be, for example, the binary log naming manner, and the present application does not limit this.

[0099] In some embodiments, in response to the transaction commit operation of the target transaction, writing a transaction identifier in the temporary file and renaming it according to a target naming manner to be used as a binary log file may include: in response to the transaction commit operation of the target transaction, adding a target space at the end of the temporary file to write the transaction identifier in the target space; renaming the temporary file according to the target naming manner to generate a binary log file.

[0100] In some embodiments, in response to the transaction commit operation of the target transaction, writing a transaction identifier in the temporary file and renaming it according to a target naming manner to be used as a binary log file may include: in response to the transaction commit operation of the target transaction, writing the transaction identifier in the target space reserved at the head of the temporary file; renaming the temporary file according to the target naming manner to generate a binary log file.

[0101] The binary log file in the traditional way is a file with a target format, which is the original file format of the binary log file and can be composed of a file header, file entries, etc. The file entries can include log entries written to the binary log. In addition, the file entries can also include transaction entries written with transaction identifiers, that is, the transaction identifier needs to be included in the binary log file. Since there is no need to recreate the binary log file, in the embodiments of the present application, target space can be reserved at the file header when generating the temporary file to write the transaction identifier when the transaction is committed; since the size of the transaction identifier is unknown, the reserved target space at the header can be set according to the maximum occupied space of the transaction identifier. In order to make reasonable use of space, avoid space waste, and reduce the file occupancy size, it can also be that when the transaction is committed, target space is added at the end of the temporary file, and the target space can be set in combination with the size of the transaction identifier generated when the transaction is committed.

[0102] In practical applications, the binary log file can be used for data backup, data recovery, data replication, and auditing, etc. Therefore, the binary log file can be provided to the requester for corresponding processing. In the above two implementation manners, the binary log file is generated by changing the format of the temporary file. In some embodiments, for the convenience of the requester to process, the method may further include: receiving a file acquisition request sent by the requester for a target transaction; acquiring the binary log file corresponding to the target transaction; re-encapsulating according to the transaction identifier in the target space of the binary log file and the binary log entries in the binary log file to generate a first file in the target format; providing the first file to the requester.

[0103] That is, in the embodiments of the present application, when receiving a file acquisition request from the requester for a target transaction, the content of the binary log file can be re-encapsulated according to the target format to generate a first file that conforms to the original file format, and the requester can directly perform corresponding processing based on the first file. For example, the requester can use the first file for data recovery, data backup, data replication, or auditing, etc.

[0104] As another optional implementation manner, processing the temporary file in the storage node to generate a binary log file in response to the transaction submission operation of the target transaction may include: in response to the transaction submission operation of the target transaction, renaming the temporary file to obtain a target file; generating a binary log file according to the transaction identifier of the target transaction and the file index information of the target file in the target format.

[0105] Among them, the binary log file can be used to obtain file index information and obtain the target file according to the file index information.

[0106] The file index information of the target file may include metadata information such as the file name, file size, and file creation time of the target file.

[0107] In the binary log file generated according to the transaction identifier of the target transaction and the file index information of the target file in the target format, the file index information is also written into the binary log file as a log entry, but the log entry records the file index information rather than the operation information for the database.

[0108] Among them, according to the transaction identifier of the target transaction and the file index information of the target file, a binary log file is generated in the target format. Specifically, after writing the transaction identifier of the target transaction and the file index information into the binary log file, a transaction start marker (BEGIN) is added between the transaction identifier and the file index information, and a transaction end marker (COMMIT) or rollback marker (ROLLBACK) is added after the file index information.

[0109] In the above implementation, the binary log is actually stored in the target file, and the binary log file only stores the file index information. For the convenience of the requester to process, in some embodiments, the method may further include: receiving a file acquisition request sent by the requester for the target transaction; obtaining the binary log file corresponding to the target transaction; obtaining the target file according to the file index information in the binary log file; repackaging the transaction identifier in the binary log file and the target file to generate a second file in the target format; providing the second file to the requester.

[0110] In some embodiments, during the execution of the target transaction, the generated binary log may be first written into the cache corresponding to the target transaction, and the cache may be the memory allocated by the computing node to the target transaction.

[0111] Therefore, the method may include: when the cache corresponding to the target transaction reaches a predetermined capacity, writing the binary log in the cache into a temporary file, and writing the binary log currently generated by the target transaction into the cache.

[0112] Among them, when the temporary file is stored locally and not stored in the storage node, once the cache reaches the predetermined capacity, all the binary logs in the cache are written into the local temporary file, and the binary logs in the cache can be cleared to write the binary logs newly generated by the target transaction; when the temporary file is stored in the storage node, once the cache reaches the predetermined capacity, all the binary logs in the cache are written into the temporary file of the storage node, and the binary logs in the cache can be cleared to write the binary logs newly generated by the target transaction.

[0113] Since the cache is located in the memory while the local temporary file is located in storage devices such as disks, by first writing the generated binary log to the cache, and when the cache overflows, writing all the binary logs in the cache to the temporary file and clearing the binary logs in the cache for the newly generated binary logs of the target transaction to be written, compared with directly writing the binary logs generated during the execution of the target transaction to the temporary file, it can reduce the number of file writes, avoid frequent device I / O (input / output) operations, reduce device I / O pressure, and improve the write efficiency.

[0114] When a transaction is committed, there may be a situation where the binary logs are still retained in the current cache because the cache has not reached the predetermined capacity. Therefore, in some embodiments, processing the temporary file in the storage node to generate a binary log file in response to the transaction commit operation of the target transaction may include: in response to the transaction commit operation of the target transaction, writing the binary logs in the cache corresponding to the target transaction to the temporary file in the storage node; processing the temporary file in the storage node to generate a binary log file.

[0115] In some embodiments, a resource lock can be used to protect the binary log file generated during the transaction commit process of the target transaction to prevent other transactions from writing to or modifying the binary log file simultaneously. Therefore, processing the temporary file in the storage node to generate a binary log file in response to the transaction commit operation of the target transaction may include:

[0116] In response to the transaction commit operation of the target transaction, applying for a resource lock and, when the resource lock is obtained, processing the temporary file in the storage node to generate a binary log file; releasing the resource lock when the generation of the binary log file from the temporary file is completed.

[0117] In some embodiments, the method may further include: deleting the temporary file from the storage node when it is detected that the execution of the target transaction fails.

[0118] In a practical application, the technical solution of the embodiment of the present application can be applied to the cloud native scenario, for example, applied to Figure 2The cloud-native database processing system shown, for example, can be PolarDB (Polar Database, a relational database management system based on cloud computing). The cloud-native database processing system can include at least one computing node 201 (only one computing node is illustrated for example), and a storage node 202. The database is a cloud-native database, which can be deployed in the storage node 202. The storage node 202 can be a network storage device, which can be provided by PolarFS (Polar File System) for example. The computing node can be a cloud server, which can be provided by PolarDB for example.

[0119] The computing node 201 can execute a target transaction and commit the target transaction when the execution is completed. The temporary files generated during the execution of the target transaction by the computing node 201 can be written to the storage node 202 when the target transaction meets the target conditions; when the target transaction is committed, if there is a binary log in the cache that has not been written to the temporary file, the binary log can be first written to the temporary file in the storage node 202, and the temporary file can be processed such as renamed to obtain a binary log file.

[0120] For ease of understanding, Figure 3 A schematic diagram showing the computing node 201 processing transaction execution and commit is shown. The computing node 201 can process the target transaction opened by the database. During the execution of the target transaction, the computing node can save the binary log generated by the target transaction to the cache corresponding to the target transaction in the computing node. When the cache corresponding to the target transaction reaches a predetermined capacity, the binary log in the cache can be written to the temporary file, and the binary log currently generated by the target transaction can be written to the cache.

[0121] When the computing node 201 detects that the target transaction of the database meets the target conditions, for example, when it detects that the data volume of the binary log in the temporary file reaches a predetermined data volume, it identifies the target transaction as a large transaction, and stores the temporary file generated by the target transaction to the storage node 202. Thus, it does not need to occupy a large amount of local storage resources, can avoid transaction execution failure caused by capacity limitations, and will not affect other task processing, which can improve the device performance.

[0122] Among them, when the cache corresponding to the target transaction reaches the predetermined capacity, all the binary logs in the cache can be written into a local temporary file, and the binary logs in the cache are cleared. After that, the binary logs newly generated by the target transaction are still written into the cache. Once the cache reaches the predetermined capacity, regardless of whether the temporary file is stored locally or in the storage node 202, the operation of writing all the binary logs in the cache into the temporary file and clearing the binary logs in the cache is executed.

[0123] Furthermore, when the computing node 201 responds to the transaction commit operation of the target transaction, there may be a situation where the binary logs are still retained because the current cache has not reached the predetermined capacity. Therefore, the binary logs in the cache corresponding to the target transaction can be written into the temporary file in the storage node 202.

[0124] In addition, when the target transaction is committed, multiple transactions may be committed simultaneously. The computing node 201 can respond to the transaction commit operation of the target transaction, apply for a resource lock, and, in the case of obtaining the resource lock, perform processing such as renaming the temporary file in the storage node 202 to generate a binary log file. Among them, the resource lock can protect the binary log file generated during the transaction commit process of the target transaction, preventing other transactions from writing to or modifying the binary log file simultaneously.

[0125] For example, while the target transaction is committing, other transactions B, C... Z are also committing, and it has reached the stage of writing their respective binary logs into the binary log file. Since the resource lock is held by the target transaction, B, C... Z need to wait for the target transaction to complete the commit to obtain the resource lock. When the processing of the temporary file of the target transaction to generate the binary log file is completed, the resource lock is released. Furthermore, B, C... Z can obtain the resource lock. Among them, since a large number of binary logs generated by the target transaction have been written into the temporary file of the storage node 202 before the transaction commit, the amount of binary log data in the cache corresponding to the target transaction during the transaction commit is small. The computing node 201 can quickly complete writing the binary log data in the cache into the temporary file in the storage node 202, and can also quickly process the temporary file to generate the binary log file and release the resource lock, so that B, C... Z can quickly obtain the resource lock and avoid blocking B, C... Z from committing transactions.

[0126] In addition, during the process of the computing node 201 writing the binary logs of the target transaction into the storage node 202, or after the writing operation is completed, the storage node 202 can return an acknowledgment message to the computing node 201 to confirm the success of the writing or return an exception message to inform the reason for the failure, etc.

[0127] In the technical solution of the embodiment of the present application, since the temporary file will be stored in the storage node during the execution of the transaction, it is not necessary to occupy a large amount of local storage resources, which can avoid the failure of transaction execution caused by capacity limitations and will not affect other task processing, and can improve resource utilization. And before the transaction is committed, the binary log of the target transaction has been written into the storage node, and the time consumed for writing the binary log into the storage node is dispersed during the execution of the transaction. Therefore, it is possible to avoid the large amount of writing time and blocking of other transaction submissions caused by writing the binary log during transaction submission, shorten the submission time of the target transaction, and improve the efficiency of transaction submission.

[0128] Figure 4 FIG. 4 is a schematic structural diagram of an embodiment of a log processing device provided by an embodiment of the present application. The method device includes:

[0129] A detection module 401, configured to detect that a target transaction of a database meets a target condition, and store a temporary file generated by the target transaction in a storage node; wherein, the temporary file stores a binary log for the database;

[0130] A generation module 402, configured to, in response to a transaction submission operation of the target transaction, process the temporary file in the storage node to generate a binary log file.

[0131] Wherein, the target transaction meeting the target condition may include one or more of the following implementation manners: the data volume of the binary log in the temporary file generated by the target transaction reaches a predetermined data volume; or, the execution duration of the target transaction reaches a predetermined duration; or, the resources occupied by the target transaction reach a predetermined resource, etc.

[0132] Therefore, as an optional manner, the target transaction of the database meeting the target condition may be: the data volume of the binary log in the temporary file generated by the target transaction of the database reaches a predetermined data volume.

[0133] Combined with the foregoing description, since binary logs will be continuously generated during the execution of the target transaction, the binary logs will be first written into the cache. When the cache reaches a predetermined capacity, the binary logs in the cache will be written into the local temporary file, and the cache will be cleared to continue writing the newly generated binary logs. At this time, the data volume of the binary logs in the temporary file can be detected. If the data volume of the binary logs in the temporary file reaches the predetermined data volume, it indicates that the target transaction is a large transaction.

[0134] As another optional manner, the target transaction of the database meeting the target condition may be: the execution duration of the target transaction of the database reaches a predetermined duration.

[0135] As yet another optional manner, the target transaction of the database meeting the target condition may be: the resources occupied by the target transaction of the database reach a predetermined resource.

[0136] In addition, it may also be determined whether the target transaction is a large transaction by combining multiple conditions such as the data volume of the binary log in the temporary file reaching a predetermined data volume, the execution duration of the target transaction reaching a predetermined duration, and the resources occupied by the target transaction reaching a predetermined resource. This application does not limit this.

[0137] Among them, there are various implementation manners for the generation module to process the temporary file in the storage node to generate a binary log file:

[0138] As an optional implementation manner, when the generation module responds to the transaction submission operation of the target transaction, processing the temporary file in the storage node to generate a binary log file may include: responding to the transaction submission operation of the target transaction, writing a transaction identifier in the temporary file, and renaming it according to the target naming manner to be used as the binary log file.

[0139] Since the temporary file only saves the binary log, and different transactions need to be distinguished by the transaction identifier, which may be, for example, a GTID, and can be used to uniquely identify a transaction for easy tracking and management of the transaction. And the transaction identifier is only generated when the transaction is committed. Therefore, in the embodiments of this application, the format of the temporary file can be directly changed, the transaction identifier can be written in the temporary file, and it can be renamed to generate a binary log file.

[0140] Since the temporary file will be deleted in case of transaction execution failure, renaming the temporary file can effectively distinguish the temporary file and the binary log file, so that the binary log file can be persistently saved to avoid being accidentally deleted, etc.

[0141] Among them, the target naming manner can be preset and is different from the naming manner of the temporary file. Optionally, the target naming manner can be, for example, the naming manner of the binary log. This application does not limit this.

[0142] In some embodiments, when responding to the transaction submission operation of the target transaction, writing a transaction identifier in the temporary file and renaming it according to the target naming manner to be used as the binary log file may include: responding to the transaction submission operation of the target transaction, adding a target space at the end of the temporary file to write the transaction identifier in the target space; renaming the temporary file according to the target naming manner to generate a binary log file.

[0143] In some embodiments, in response to a transaction commit operation of a target transaction, a transaction identifier is written in a temporary file and renamed according to a target naming method to be used as a binary log file, which may include: writing a transaction identifier in a target space reserved at the head of the temporary file in response to the transaction commit operation of the target transaction; renaming the temporary file according to the target naming method to generate a binary log file.

[0144] The binary log file in the traditional way is a file with a target format, and the target format is the original file format of the binary log file, which may be composed of a file header and file entries, etc. The file entries may include log entries written to the binary log. In addition, the file entries may also include transaction entries for writing transaction identifiers, etc., that is, the binary log file needs to include transaction identifiers. Since there is no need to recreate the binary log file, in the embodiments of the present application, a target space can be reserved at the head of the file when generating the temporary file to write the transaction identifier when the transaction is committed; since the size of the transaction identifier is unknown, the target space reserved at the head can be set according to the maximum occupied space of the transaction identifier. To make reasonable use of space and avoid space waste so as to reduce the file occupancy size, it can also be that in the case of transaction commit, a target space is added at the end of the temporary file, and the target space can be set in combination with the size of the transaction identifier generated when the transaction is committed.

[0145] In practical applications, since the binary log file can be used for data backup, data recovery, data replication, and auditing, etc., the binary log file can be provided to the requester for corresponding processing. In the above two implementation manners, the binary log file is generated by changing the format of the temporary file. In some embodiments, the method may further include: receiving a file acquisition request sent by the requester for a target transaction; acquiring the binary log file corresponding to the target transaction; repackaging and generating a first file in the target format according to the transaction identifier in the target space of the binary log file and the binary log entries in the binary log file; providing the first file to the requester.

[0146] That is, in the embodiments of the present application, when receiving a file acquisition request from the requester for a target transaction, the content of the binary log file can be repackaged according to the target format to generate a first file that conforms to the original file format, and the requester can directly perform corresponding processing based on the first file. For example, the requester can use the first file for data recovery, data backup, data replication, and auditing, etc.

[0147] As another alternative implementation, when the generation module responds to the transaction commit operation of the target transaction, processing the temporary file in the storage node to generate a binary log file may include: in response to the transaction commit operation of the target transaction, renaming the temporary file to obtain a target file; generating a binary log file in accordance with the target format according to the transaction identifier of the target transaction and the file index information of the target file.

[0148] Wherein, the binary log file can be used to obtain the file index information and obtain the target file according to the file index information.

[0149] The file index information of the target file may include metadata information such as the file name, file size, and file creation time of the target file, for example.

[0150] In the binary log file generated in accordance with the target format according to the transaction identifier of the target transaction and the file index information of the target file, the file index information is also written into the binary log file as a log entry, but the log entry records the file index information instead of the operation information for the database.

[0151] Wherein, generating a binary log file in accordance with the target format according to the transaction identifier of the target transaction and the file index information of the target file may specifically be, after writing the transaction identifier of the target transaction and the file index information into the binary log file, adding a transaction start marker (BEGIN) between the transaction identifier and the file index information, and adding a transaction end marker (COMMIT) or a rollback marker (ROLLBACK) after the file index information.

[0152] In the above implementation, the binary log is actually stored in the target file, while the binary log file only saves the file index information. For the convenience of the requester to process, in some embodiments, the method may further include: receiving a file acquisition request sent by the requester for the target transaction; obtaining the binary log file corresponding to the target transaction; obtaining the target file according to the file index information in the binary log file; repackaging the transaction identifier in the binary log file and the target file to generate a second file in the target format; providing the second file to the requester.

[0153] In some embodiments, during the execution of the target transaction, the generated binary log may be first written into the cache corresponding to the target transaction, and the cache may be the memory allocated by the computing node to the target transaction.

[0154] The apparatus may, when the cache corresponding to the target transaction reaches a predetermined capacity, write the binary log in the cache into a temporary file, and write the binary log currently generated by the target transaction into the cache.

[0155] Among them, when the temporary file is stored locally and not stored in the storage node, once the cache reaches the predetermined capacity, all the binary logs in the cache are written into the local temporary file, and the binary logs in the cache can be cleared to write the binary logs newly generated by the target transaction; when the temporary file is stored in the storage node, once the cache reaches the predetermined capacity, all the binary logs in the cache are written into the temporary file of the storage node, and the binary logs in the cache can be cleared to write the binary logs newly generated by the target transaction.

[0156] When a transaction is committed, there may be a situation where the binary logs are still retained in the current cache because the cache has not reached the predetermined capacity. Therefore, in some embodiments, the generating module processing the temporary file in the storage node to generate a binary log file in response to the transaction commit operation of the target transaction may include: in response to the transaction commit operation of the target transaction, writing the binary logs in the cache corresponding to the target transaction into the temporary file in the storage node; processing the temporary file in the storage node to generate a binary log file.

[0157] In some embodiments, the generating module processing the temporary file in the storage node to generate a binary log file in response to the transaction commit operation of the target transaction may include: in response to the transaction commit operation of the target transaction, applying for a resource lock and, when the resource lock is obtained, processing the temporary file in the storage node to generate a binary log file; releasing the resource lock when the generation of the binary log file from the temporary file is completed.

[0158] In some embodiments, the apparatus can also be used to delete the temporary file from the storage node when it is detected that the execution of the target transaction fails.

[0159] Figure 4 The described log processing apparatus can execute Figure 1 the log processing method described in the illustrated embodiment, and its implementation principle and technical effects will not be elaborated further. For the log processing apparatus in the above embodiments, the specific manners in which each module and unit perform operations have been described in detail in the embodiments related to the method, and will not be elaborated in detail here.

[0160] An embodiment of the present application also provides a computing device, as Figure 5 shown, the device may include a storage component 501 and a processing component 502;

[0161] The storage component 501 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component to implement the log processing method described in the embodiment as Figure 1 shown.

[0162] Of course, the computing device will necessarily also include other components, such as input / output interfaces, display components, communication components, etc.

[0163] The input / output interface provides an interface between the processing component and the peripheral interface module, and the above peripheral interface module can be an output device, an input device, etc. The communication component is configured to facilitate communication between the computing device and other devices in a wired or wireless manner, etc.

[0164] Among them, the processing component 502 can include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component can also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components for executing the above method.

[0165] The storage component 501 is configured to store various types of data to support the operation of the terminal. The storage component 501 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0166] It should be noted that the above computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. It can be implemented as a distributed cluster composed of multiple servers or terminal devices, or can be implemented as a single server or a single terminal device.

[0167] It should be noted that when the above computing device implements Figure 1 the log processing method described in the illustrated embodiment, it can be a physical device or an elastic computing host provided by a cloud computing platform, etc. It can be implemented as a distributed cluster composed of multiple servers or terminal devices, or can be implemented as a single server or a single terminal device.

[0168] The embodiment of the present application also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a computer, it can implement the above Figure 1 log processing method described in the illustrated embodiment. The computer-readable medium can be included in the electronic device described in the above embodiment; or can exist alone without being assembled into the electronic device.

[0169] An embodiment of the present application further provides a computer program product, which includes computer programs / instructions; when the computer programs / instructions are executed by a computer, the log processing method described in the foregoing embodiments as shown in Figure 1 can be implemented. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by a processor, various functions defined in the system of the present application are executed.

[0170] In the foregoing corresponding embodiments, the computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or combined with an instruction execution system, apparatus, or device.

[0171] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, apparatuses, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0172] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement without creative effort.

[0173] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of each embodiment of the present application.

Claims

1. A log processing method, characterized in that, Including: When a target transaction in the detection database meets the target condition, storing the temporary file generated by the target transaction to a storage node; The temporary file stores the binary log for the database; In response to the transaction commit operation of the target transaction, processing the temporary file in the storage node to generate a binary log file.

2. The method according to claim 1, wherein The processing the temporary file in the storage node to generate a binary log file in response to the transaction commit operation of the target transaction includes: In response to the transaction commit operation of the target transaction, writing a transaction identifier in the temporary file and renaming it according to a target naming method to be used as a binary log file.

3. The method according to claim 2, wherein The writing a transaction identifier in the temporary file and renaming it according to a target naming method to be used as a binary log file in response to the transaction commit operation of the target transaction includes: In response to the transaction commit operation of the target transaction, adding a target space at the end of the temporary file to write the transaction identifier in the target space; Renaming the temporary file according to the target naming method to generate a binary log file.

4. The method according to claim 2, wherein The writing a transaction identifier in the temporary file and renaming it according to a target naming method to be used as a binary log file in response to the transaction commit operation of the target transaction includes: In response to the transaction commit operation of the target transaction, writing the transaction identifier in the target space reserved at the head of the temporary file; Renaming the temporary file according to the target naming method to generate a binary log file.

5. The method according to claim 3 or 4, characterized in that, Also including: Receiving a file acquisition request sent by a requester for the target transaction; Obtaining the binary log file corresponding to the target transaction; According to the transaction identifier in the target space of the binary log file and the binary log entries in the binary log file, re-encapsulating to generate a first file in a target format; Providing the first file to the requester.

6. The method according to claim 1, wherein The processing the temporary file in the storage node to generate a binary log file in response to the transaction commit operation of the target transaction includes: In response to the transaction commit operation of the target transaction, renaming the temporary file to obtain a target file; According to the transaction identifier of the target transaction and the file index information of the target file, generating a binary log file according to a target format; the binary log file is used to obtain the file index information and obtain the target file according to the file index information.

7. The method according to claim 6, wherein Also including: Receiving a file acquisition request sent by a requester for the target transaction; Obtaining the binary log file corresponding to the target transaction; According to the file index information in the binary log file, obtaining the target file; Re-encapsulating the transaction identifier in the binary log file and the target file to generate a second file in a target format; Providing the second file to the requester.

8. The method according to claim 1, characterized in that, Also including: When the cache corresponding to the target transaction reaches a predetermined capacity, writing the binary log in the cache to a temporary file, and writing the binary log currently generated by the target transaction to the cache.

9. The method according to claim 8, wherein The transaction commit operation in response to the target transaction, and processing the temporary file in the storage node to generate a binary log file includes: In response to the transaction commit operation of the target transaction, writing the binary log in the cache corresponding to the target transaction into the temporary file in the storage node; Processing the temporary file in the storage node to generate a binary log file.

10. The method according to claim 1, characterized in that The transaction commit operation in response to the target transaction, and processing the temporary file in the storage node to generate a binary log file includes: In response to the transaction commit operation of the target transaction, applying for a resource lock and, when the resource lock is obtained, processing the temporary file in the storage node to generate a binary log file; Releasing the resource lock when the generation of the binary log file from the temporary file is completed.

11. The method according to claim 1, characterized in that, It further includes: When it is detected that the execution of the target transaction fails, deleting the temporary file from the storage node.

12. The method according to claim 1, characterized in that, The database is a cloud-native database; the storage node is a network storage device.

13. The method according to claim 1, characterized in that The target transaction of the database meeting the target conditions includes: The data volume of the binary log in the temporary file generated by the target transaction of the database reaches a predetermined data volume; and / or, the execution duration of the target transaction reaches a predetermined duration; and / or, the resources occupied by the target transaction reach a predetermined resource.

14. A computing device, characterized in that, It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the log processing method according to any one of claims 1 to 13.

15. A computer-readable storage medium, characterized in that, A computer program is stored, and when the computer program is executed by a computer, the log processing method according to any one of claims 1 to 13 is implemented.

16. A computer program product, characterized in that, It includes computer programs / instructions, and when the computer programs / instructions are executed by a computer, the log processing method according to any one of claims 1 to 13 is implemented.

Images