Log storage method, database system, server and storage medium

Abstract

The embodiment of the invention provides a log storage method, a database system, a server and a storage medium. In the log storage method, any database node in a distributed database can obtain a transaction log of a target transaction by using a storage engine component, and write the transaction log into a target online log file. The database node can utilize the storage engine to persistently store the transaction log in the target online log file into the corresponding target block device, and execute transaction-level persistent storage operation on metadata information of the target online log file. In the implementation mode, based on the online log file supporting the write-in operation at any position, the database node can directly store the transaction log in a block storage mode based on the storage engine component of the database node under the condition of not depending on a file system, and resource overhead caused by introduction of the file system and the file storage mode is reduced.

Description

Technical Field

[0001] This application relates to the field of computer technologies, and in particular, to a log storage method, a database system, a server, and a storage medium. Background Art

[0002] In MySQL (a relational database management system), the binary log (Binlog) is used to record the changes made to the database within MySQL, including operations such as insertions, updates, and deletions in the database. The binary log is mainly used for master-slave replication and incremental recovery of the database and is an important file in a distributed database system. In the current distributed database system, the file system is mainly relied on to perform operations such as creating, reading, and modifying the binary log. To support a wide range of applications in the distributed database, the file system is designed as a complex general component. The complex design of the file system makes the file system have a high operation complexity in storing the binary log, thus introducing unnecessary overhead. Therefore, there is a need to propose a new solution. Summary of the Invention

[0003] Multiple aspects of this application provide a log storage method, a database system, a server, and a storage medium, so as to reduce the operation complexity of storing the binary log in a distributed database and reduce resource overhead.

[0004] An embodiment of this application provides a log storage method, which is applicable to any database node in a distributed database. The method includes: a storage engine component obtains a transaction log of a target transaction; writes the transaction log into a target online log file, where the target online log file is an online log file that supports write operations at any position; and persistently stores the transaction log in the target online log file into a corresponding target block device.

[0005] Optionally, writing the transaction log into the target online log file includes: determining a first online log file in a startup state; determining whether the data volume of the transaction log is greater than the remaining writable data volume of the first online log file; if so, closing the first online log file and starting a new second online log file as the target online log file; and writing the transaction log into the target online log file.

[0006] Optionally, writing the transaction log into the target online log file includes: writing the transaction log into a target memory cache block corresponding to the target online log file; and persistently storing the transaction log in the target online log file into a corresponding target block device includes: synchronizing the transaction log written in the target memory cache block to a target block device associated with the target online log file according to a log synchronization parameter.

[0007] Optionally, it further includes: determining the log synchronization parameter according to at least one of the load information of the distributed database, the throughput capacity information of the target block device, and the backlog information of the online log files to be synchronized in the distributed database.

[0008] Optionally, the storage space size of the target memory cache block is the same as that of the target block device.

[0009] Optionally, it further includes: during the process of writing the transaction log to the target online log file, determining the first block device associated with the target online log file; if the data volume of the transaction log is greater than the first block device, applying to the storage management component for allocating a second block device; associating the target online log file with the second block device to expand the target online log file.

[0010] Optionally, the method further includes: during the process of persisting the transaction log, in response to the commit instruction of the target transaction, writing the metadata information of the target online log file to the redo log file of the target transaction; and writing the commit event of the target transaction to the redo log file of the target transaction; and after completing the persistent storage of the transaction log, completing the commit operation of the target transaction.

[0011] Optionally, the method further includes: after the target transaction is committed, performing a persistent operation on the redo log file of the target transaction to perform a transaction-level persistent operation on the metadata information of the target online log file; or after the transaction group to which the target transaction belongs is committed, performing a persistent operation on the redo log file of the target transaction to perform a transaction group-level persistent operation on the metadata information of the target online log file.

[0012] Optionally, the target online log file includes: an initial online log file created during the initialization of the database node or an online log file that is put into circular use after being archived.

[0013] Optionally, after persistently storing the transaction log in the target online log file to the corresponding target block device, it further includes: archiving the target online log file to a specified storage space and updating the target online log file to an archived state so that the target online log file can be put into circular use after being archived.

[0014] The embodiment of the present application also provides a distributed database system, including: a plurality of database nodes, a data block file component, and a storage component; wherein, the storage component is used to provide at least one block device; the database file component is used to provide at least one online log file; any one of the plurality of database nodes is used to obtain the transaction log of the target transaction by using the storage engine component; write the transaction log into the target online log file in the at least one online log file; the target online log file is an online log file that supports write operations at any position; and persistently store the transaction log in the target online log file onto the target block device associated with the target online log in the at least one block device.

[0015] Optionally, the system further includes: a storage management component; the storage engine component is further used for: during the process of writing the transaction log into the target online log file, determining the first block device already associated with the target online log file; if the data volume of the transaction log is greater than the first block device, sending a block device application to the storage management component, so that the storage management component allocates a second block device for the target online log from the at least one block device; and associating the target online log file with the second block device to expand the target online log file.

[0016] The embodiment of the present application also provides a server, including: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to: execute the steps in the method provided by the embodiment of the present application.

[0017] The embodiment of the present application also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, it can implement the steps in the method provided by the embodiment of the present application.

[0018] In the embodiment of the present application, any database node in the distributed database can obtain the transaction log of the target transaction by using the storage engine component and write the transaction log into the target online log file. The database node can persistently store the transaction log in the target online log file onto the corresponding target block device by using the storage engine, and perform a transaction-level persistent storage operation on the metadata information of the target online log file. In this implementation manner, based on the online log file that supports write operations at any position, the database node can directly store the transaction log in a block storage manner based on its own storage engine component without relying on the file system, reducing the resource overhead caused by introducing the file system and the file storage method. Description of the Drawings

[0019] The accompanying drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0020] Figure 1 It is a schematic diagram of the architecture of shared storage dependent on the file system;

[0021] Figure 2 It is a schematic flowchart of the log storage method provided by an exemplary embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of the architecture of shared storage independent of the file system of the present application;

[0023] Figure 4 It is a schematic diagram of the structure of a distributed database system provided by an exemplary embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of the structure of a server provided by an exemplary embodiment of the present application. Detailed implementation manners

[0025] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0026] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms of "a", "the" and "said" used in the embodiments of the present invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. "Plural" generally includes at least two, but does not exclude the case of including at least one.

[0027] It should be understood that the term " / and / " used herein is only a description of the associated relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after.

[0028] It should also be noted that the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, such that a commodity or system including a series of elements not only includes those elements but also other elements not expressly listed, or elements inherent to such commodity or system. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the commodity or system including said element.

[0029] Shared storage refers to the technology where multiple computer systems or processes can share the same physical storage device or storage system. In a shared storage system, multiple computers or processes can access the data in the same storage device or storage system and perform operations such as reading, writing, and modifying the data, thereby achieving data sharing and collaborative processing. Shared storage systems can improve data sharing and reliability, thus supporting larger-scale applications and services. For example, in large-scale database applications, shared storage can improve data access speed and reliability, thereby achieving high performance and high availability. At the same time, to ensure data consistency and integrity, shared storage systems usually implement some mechanisms such as locking, synchronization, and caching.

[0030] Figure 1 Schematically shows the master-slave architecture form of a typical shared storage-based database system. In Figure 1 the shown architecture, there are multiple database nodes, and the multiple database nodes include: one master node and multiple slave nodes. Among them, any database node is implemented as a database instance. Among them, the master node can be a write node, and the slave nodes can be read nodes. The write node is responsible for processing data write operations in the database system, and the slave nodes are responsible for reading the data written by the master node. In some embodiments, role switching can occur between the master node and the slave nodes. For example, when the original master node fails, one of the slave nodes can be switched to a new master node for writing data. The log data generated locally by the master node can be written into the binary log file in the database file component and persistently stored in the storage base from the binary log file. As Figure 1 shown, between the database file component and the storage base, data reading and writing are achieved in a file storage manner through the file system. In this architecture, based on the file system (usually a distributed file system), the existence of shared storage can be masked from multiple database nodes, enabling the database storage engine in the distributed database system to access and modify data like a single-machine database.

[0031] However, the file system is a complex component for a wide range of applications, which can cause many problems in the shared storage scenario. For example, every storage access operation needs to go through the file system, resulting in a deep storage stack. At the same time, the file system itself consumes a certain amount of computing resources and has high maintenance costs. For example, the file system is oriented to general scenarios, with many factors to consider and high flexibility in the storage expansion timing, resulting in poor storage expansion stability. For example, metadata information of logs needs to be synchronized between multiple nodes, but synchronizing metadata information through the file system is costly and uncontrollable. Another example is that the writing of binary logs depends on the cache of the file system, and these caches consume a large amount of memory resources. In particular, when the resources of the database instance are tense, the cleaning of the file system cache will be triggered, which will bring a large number of I / O (Input / Output) operations in a short time, and in severe cases, the entire database instance will become unresponsive (hang) and unavailable for service.

[0032] In view of the above technical problems, in some embodiments of the present application, a solution is provided, which can implement the storage of binary log files without relying on the file system. The following will detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.

[0033] Figure 2 FIG. is a schematic flowchart of a log storage method provided by an exemplary embodiment of the present application, and the method may include the following Figure 1 steps as shown:

[0034] Step 201, obtain the transaction log of the target transaction by using a storage engine component.

[0035] Step 202, write the transaction log into a target online log file; the target online log file is an online log file that supports write operations at any position.

[0036] Step 203, persistently store the transaction log in the target online log file into a corresponding target block device.

[0037] This embodiment is applied to a database node in a distributed database system. Among them, the architecture of the distributed database system is as shown in Figure 3As shown, it may include multiple database nodes, namely a master node and multiple standby nodes. The storage engine components running on each of the multiple database nodes form a storage engine cluster. Among them, any database node can be implemented as a database instance. The execution subject of this embodiment can be the master node, which is a write node for processing data write operations. In this embodiment, the database node can use the storage engine component to directly perform the storage operation of the transaction log without relying on the file system. Among them, a transaction is a unit of concurrency control and is an operation sequence defined by the user. A transaction has atomicity, consistency, isolation, and durability. The target transaction can be any transaction processed by the distributed database system.

[0038] Before the transaction is committed, it is necessary to persistently store the transaction log to re-execute the modification operations of the transaction when the database fails. Among them, the transaction log may at least include: at least one of the ID (Identity document) of the transaction, the operation type of the transaction (such as insert, operation, or update), the operation object of the transaction (such as the name and identifier of the database object), the operation timestamp, and the operation content (such as the updated value or the deleted record, etc.).

[0039] In this embodiment, without relying on the file system, the storage engine component on the database node can use the block storage method to persistently store the transaction log of the target transaction into the target block device. Among them, a block device is a device with a storage space of a fixed size. The unit of each read and write operation of the block device is a data block, and it does not support random read and write. Based on this, in this embodiment, after the storage engine component obtains the transaction log of the target transaction, it can write the obtained transaction log into the target online log file, and when the set conditions are met, store the transaction log written in the target online log file into the block device. Among them, the set conditions may include: the target transaction is committed, the target online log is full, or the target online log file is closed.

[0040] In this embodiment, the target online log file refers to any online binary log (Online Binlog) file provided by the distributed database system, which is abbreviated as the online log file in this article. Among them, the online log file has the characteristic of being able to perform write operations at any position. Furthermore, based on the characteristic that the online log file can be written at any position, the write operation at any position of the block device is indirectly realized, so that the database node can directly store the transaction log in the form of block storage without relying on the file system. In the distributed database system, the number of online log files is usually specified before the database instance is initialized, and the specified number of online log files are created when the database instance is initialized. Among them, the online log files are maintained by the database file component. The online log files can be recycled, so the storage engine of the database node does not need to perform the operation of creating online log files.

[0041] As Figure 3 shown, the database file component can maintain multiple online binary files, including the online log file OLB1, the online log file OLB1…, and the online log file OLB6. Among them, each online log file has a unique identifier (ID), an index number (Index), a start status, and an archive status. Among them, the start status is used to describe whether the online log file is in a writable state, and the value of the start status can be described by 0 (not started) and 1 (started). If an online log file is in the started state, the transaction log generated by the transaction can be written into the online log file. When the online log file is full or the user actively triggers the rotation operation, the online log file is updated to the closed state, and the next new online log file is opened. Among them, during the writing of any online log file, the value of its start status remains in the started state. The transaction logs are written into the online log files in sequence. Therefore, the database allows one online log file to be in the started state at the same time.

[0042] Among them, the archive status is used to describe whether the online log file has been archived, and the value of the archive status can be described by 0 (not archived) and 1 (archived). After the transaction logs in the online log file have been persistently processed, the archive operation can be performed on the online log file for other applications to use. Optionally, the online log file can be archived to the specified storage space, which can be the local database or the remote archive storage node. After the online log file is archived, the value of the archive status of the online log file can be updated to 1 to update the online log file to the archived state. The archived online log file can be overwritten and written in a cyclic manner.

[0043] Among them, the index number is used to identify the sequence number of the online log file generated by the database node. The index number increases sequentially according to the order in which the online log files are started. When the same online log file is recycled, the index number of the online log file is different in different recycling rounds. The following will be described with reference to the accompanying drawings.

[0044] As Figure 3 shown, the ID value of the online log file OLB1 is 1, the index number is 10, the value of the start status is 0 (i.e., not started), and the value of the archiving status is 1 (i.e., already archived). Based on the above information, it can be known that the online log file OLB1 with the index number 10 has been archived, and the online log file OLB1 has been put into recycling. As Figure 3 shown, the ID value of the online log file OLB2 is 2, the index number is 11, the value of the start status is 1 (i.e., already started), and the value of the archiving status is 0 (i.e., not archived). Based on the above information, it can be known that the online log file OLB2 is currently in a writable state. Assume that the ID value of the online log file OLB3 is 3, the index number is 6, the value of the start status is 0 (i.e., not started), and the value of the archiving status is 1 (i.e., already archived). During log rotation, if the online log file OLB2 is closed, the online log file OLB3 can be started, the value of the start status of the online log file OLB3 is updated to 1 (i.e., already started), the value of the archiving status is updated to 0 (i.e., not archived), and a new index number 12 is generated for the online log file OLB3.

[0045] The target online log file described in this embodiment may include: the initial online log file created during database node initialization or the online log file that is put into recycling after being archived. Thus, the recycling of the online binary log file is realized without relying on the storage engine component to execute the creation operation of the online binary log file, saving resource overhead.

[0046] After writing the transaction log of the target transaction into the target online log file, the storage engine component of the database node can perform persistent storage on the transaction log in the target online log file. Performing persistent storage on the transaction log in the target online log file includes: writing the information in the target online log file into the target block device associated with the target online log file in a block storage manner. Among them, the target block device can be a physical block device or a virtual block device, which is not limited in this embodiment. Based on the block device, block storage of the online binary log can be performed. Compared with the file storage method, the block storage method does not require an additional layer depending on the file system and has more efficient data reading efficiency and more reliable data reading results.

[0047] Among them, each online log file in the database file component can be built on at least one block device. In the application scenario of cloud computing, the block device associated with the online log file can be a virtual block device provided by the database base. As Figure 3 shown, the storage base can provide an elastic block storage service, which can virtualize the storage space into multiple virtual block devices for block storage use. Generally, a block device has a relatively fixed storage capacity size. For example, the storage capacity size of a block device is 1G. That is, any online log file has a relatively fixed data volume size. Among them, an online log file can store the transaction logs of multiple transactions; the transaction log of a transaction is allowed to be written into an online log file.

[0048] Based on this, in some embodiments, during the process of the storage engine writing the transaction log of the target transaction into the target online log file, the storage engine can determine the first online log file in the startup state and judge whether the data volume of the transaction log of the target transaction is greater than the remaining writable data volume of the first online log file. If the data volume of the transaction log of the target transaction is greater than the remaining writable data volume of the first online log file, the storage engine can trigger the rotation operation of the online log. Among them, the rotation operation means closing the currently startup online log file and starting the next online log file. That is, the storage engine can close the first online log file, start a new second online log file as the target online log file, and write the transaction log of the target transaction into the target online log file. Among them, before closing the first online log file, an archiving operation can be performed on the transaction log in the first online log file to avoid losing log data. If the data volume of the transaction log of the target transaction is less than or equal to the remaining writable data volume of the first online log file, the storage engine can use the first online log file as the target online log file and write the transaction log of the target transaction into the target online log file.

[0049] Based on this implementation method, by comparing the remaining writable data volume of the opened online log file with the data volume of the transaction log of the transaction, the risk of writing the transaction logs of the same transaction into different online log files is reduced, and the reliability of the log storage result is improved.

[0050] In this embodiment, the online log file can be recycled. Each online log file has a unique identity. During the recycling process, after being opened each time, an index number of the online log file can be generated to identify the sequence of the generated log data. The online log file can be archived and can be put into the recycling use after archiving. When being recycled again, a new index number can be generated for the online log file to distinguish different log data through the index number.

[0051] In some exemplary embodiments, after the transaction logs in the target online log file are persistently stored, the storage engine can archive the target online log file to a specified storage space and update the target online log file to the archived state, so that the target online log file can be recycled after archiving. When the online log file is archived, the target format recognizable by the upstream / downstream applications of the database can be obtained, and the transaction logs in the online log file can be stored according to the target format. Furthermore, even when the persistent format in the online log file is different from the persistent format of the ordinary online log file used by the file system, the upstream / downstream applications of the database can use the transaction logs based on the archived online log file, thereby reducing the risk of incompatibility of the transaction logs with the original upstream / downstream applications.

[0052] In this embodiment, based on the online log file that supports write operations at any position, the database node can directly store the transaction logs in a block storage manner based on its own storage engine component without relying on the file system, reducing the resource overhead caused by introducing the file system and the file storage method. In addition, when the storage engine component writes the transaction logs into the online log file, the characteristic that the online log file can be recycled can be fully utilized, and there is no need to perform the operation of creating binary logs, further reducing the complexity of the log storage operation and the resource overhead.

[0053] In some alternative embodiments, before persistently storing the target online log file, the storage engine can expand the target block device corresponding to the target online log file according to the data volume of the target online log.

[0054] Optionally, during the process of writing the transaction logs of the target transaction into the target online log file, the storage engine can determine the first block device already associated with the target online log file. If the data volume of the transaction logs of the target transaction is greater than the first block device, the storage engine can apply to the storage management component for allocating a second block device and associate the target online log file with the second block device to expand the target online log file. The second block device can include one or more block devices.

[0055] The storage management component can manage the association relationships among the target online log file, the first block device, and the second block device. As Figure 3 shown, in the database meta-file managed by the storage management component, the corresponding relationships between the online log file OLB1 and the block device 01, between the online log file OLB2 and the block device 02, between the online log file OLB3 and the block device 03, etc. can be stored. Furthermore, when the storage engine persistently stores the transaction logs in the target online log file, the transaction logs in the target online log file can be persistently stored in the first block device and the second block device according to the association relationships managed by the storage management component.

[0056] In this embodiment, the storage engine can implement the expansion of the online log file based on the storage management component. Compared with the solution of using the file system for expansion, the expansion operation complexity of the storage engine is lower and the reliability is higher.

[0057] In some alternative embodiments, during the process of the database node performing persistent storage on the online log, the storage engine of the database node can implement reading and writing to any position of the block device based on the memory cache. The following will give an exemplary illustration.

[0058] Optionally, in this embodiment, a memory cache corresponding to the online binary log file can be set in the memory of the database node. The memory cache can be divided into multiple memory cache blocks, and the multiple memory cache blocks can respectively correspond to multiple online binary files. Optionally, the storage space size of any memory cache block is the same as that of the block device.

[0059] Based on this, during the process of the storage engine writing the transaction log of the target transaction into the target online log file, the transaction log of the target transaction can be first written into the target memory cache block corresponding to the target online log file, and the transaction log written in the target memory cache block can be synchronized to the target block device associated with the target online log file according to the log synchronization parameter.

[0060] Optionally, the log synchronization parameter can be set fixedly or determined dynamically according to the environmental data where the log synchronization operation is located. Among them, the environmental data where the log synchronization operation is located is used to predict the pressure generated on the database and / or the block device by the transaction log synchronization operation between the database and the block device. For example, the above environmental data can include at least one of the load information of the distributed database, the throughput capacity information of the target block device, and the backlog information of the online log file to be synchronized in the distributed database. In some embodiments, the storage engine can determine the log synchronization parameter according to the above at least one piece of information.

[0061] Optionally, the log synchronization parameters may include: the log synchronization frequency and / or the amount of synchronized data each time. For example, if the load information of the database indicates that the database has a large load pressure, a lower log synchronization frequency and a larger amount of synchronized data may be set to reduce the load pressure generated by the database performing the log synchronization operation. For example, if the throughput capacity information of the target block device indicates that the block device has a strong throughput capacity, a lower log synchronization frequency and a smaller amount of log synchronization data may be set to make full use of the hardware resources. For another example, if the backlog information of the online log files to be synchronized in the database indicates that the backlog rate of the online log files is high, a higher log synchronization frequency and a larger amount of log synchronization data may be set to alleviate the backlog situation and accelerate the recycling of the online log files to meet the usage requirements of the online log files.

[0062] In this implementation manner, the storage engine can implement write operations at any position in the memory cache block, thereby indirectly implementing write operations at any position in the block device. Correspondingly, when reading data from the block device, the data in the block device can be read into the memory cache block, and the storage engine can implement read operations at any position in the memory cache block. Compared with the method of using a file system for reading / writing, with the storage engine directly performing the read / write operations of the log, the overhead generated by the file system can be saved, and the problem of read / write amplification can be alleviated.

[0063] In addition to recording transaction logs, the online log files can also maintain the metadata information of the online log files themselves. The metadata information mainly includes: the size of the logged data that has been written. Among them, the metadata information needs to be persistently stored for subsequent use. Compared with the file system, during the process of writing the transaction log of the target transaction into the online log file, the storage engine component of the database node can accurately perceive the end boundary of the transaction log, and then, after the transaction log is completely written, update the metadata information of the transaction log to record the write completion identifier of the transaction log.

[0064] Based on this, in some exemplary embodiments, the database node can use the storage engine component to perform transaction-level or transaction-group-level persistent storage operations on the metadata information of the target online log file. Among them, the transaction-level persistent operation of the metadata information refers to persistently storing the size of the logged data that has been written after the transaction is committed. The transaction-group-level persistent operation of the metadata information refers to persistently storing the size of the logged data that has been written after the transaction group to which the transaction belongs is committed. The following will give an exemplary description.

[0065] In some alternative embodiments, after writing the transaction log of a target transaction to a target online log file, the storage engine may store the metadata information of the target online log file as incremental logs in the redo log of the database. In this implementation, during the process of persisting the transaction log of the target transaction, in response to the commit instruction of the target transaction, the storage engine may write the metadata information of the target online log file to the redo log file of the target transaction. During the waiting process for the transaction log to be persisted, the storage engine may write the commit event of the target transaction to the redo log file of the target transaction until the persistent storage of the transaction log is completed, and then complete the commit operation of the target transaction.

[0066] In the above process, on the one hand, converting the caching operation of metadata information into a write operation to the redo log facilitates the persistent storage of metadata information through the persistent operation of the redo log, reducing the number of additional block devices required for subsequent separate persistent storage of metadata information. On the other hand, during the waiting process for the transaction log to be persisted, the storage engine can asynchronously perform the write operations of metadata information and commit events in the redo log file, which can reduce the waiting time for transaction commit to a certain extent.

[0067] Optionally, after the target transaction is committed, the storage engine may perform a persistent operation on the redo log file of the target transaction to perform a transaction-level persistent operation on the metadata information of the target online log file; or, after the transaction group to which the target transaction belongs is committed, the storage engine may perform a persistent operation on the redo log file of the target transaction to perform a transaction-group-level persistent operation on the metadata information of the target online log file. The transaction group to which the target transaction belongs includes the target transaction and one or more transactions with adjacent commit times to the target transaction. Based on this implementation, the frequency of metadata persistence can be further reduced, thereby saving the overhead caused by metadata synchronization operations.

[0068] Optionally, the redo log refers to the Redo log of the database. The Redo log is a physical log, and the form of a log record in the Redo log can be expressed as: changing a certain data at a certain offset of a certain data page to a new data. The data volume of a single record in Redo is relatively small, so it has less pressure on the database. When writing the metadata information of the target online log file to the redo log, a log record of more than a dozen bytes can be written in the Redo log. Furthermore, the read node in the distributed database can obtain the metadata information of the target online log file from the redo log file. Compared with the method of persisting metadata using block devices, the method of persisting the metadata of any online log file using incremental logs realizes the conversion from writing a block device to writing a log record, further reducing the storage cost of metadata.

[0069] In this case, the persisted metadata information is at the transaction level or transaction group level. Compared with the method of performing metadata persistence operations for each log write operation, on the one hand, the overhead can be reduced based on the metadata persistence operations at the transaction level or transaction group level, thereby reducing the impact on the performance of the database; on the other hand, the persisted metadata is bound to the transaction log boundary, which can reduce the uncertainty of the metadata information and improve the availability of the database. On the other hand, compared with the traditional method of persisting the metadata information of the transaction log first and then committing the transaction, in this embodiment, after the storage engine internally drives the transaction to complete the commit, it waits for the persistence of the transaction log, enabling the database to more finely control the data persistence process and improve the performance of the database.

[0070] It should be noted that the execution subject of each step of the method provided in the foregoing embodiments can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 101 to 104 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and so on.

[0071] In addition, in some of the processes described in the foregoing embodiments and the accompanying drawings, a plurality of operations appear in a specific order, but it should be clearly understood that these operations can be executed not in the order in which they appear in this article or in parallel. The operation numbers such as 101, 102, etc. are only used to distinguish different operations, and the numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and these operations can be executed in sequence or in parallel. It should be noted that the descriptions such as "first" and "second" in this article are used to distinguish different messages, devices, modules, etc., do not represent a sequence, and do not limit that "first" and "second" are of different types.

[0072] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0073] In addition to the log storage method described in the foregoing embodiments, the embodiments of the present application also provide a distributed database system, which will be described exemplarily below with reference to the accompanying drawings.

[0074] Figure 4The structural schematic diagram of the distributed database system provided by an exemplary embodiment of the present application is as follows Figure 4 As shown, the distributed database system may include: a plurality of database nodes 401, a database file component 402, and a storage component 403. Among them, the plurality of database nodes 401 are deployed in a primary-standby mode.

[0075] Among them, the storage component 403 is used to provide at least one block device. Any one of the block devices may be a physical block device or a virtual block device, which is not limited in this embodiment.

[0076] Among them, the database file component 402 is used to provide at least one online log file; among them, the at least one online log file can be recycled. Each online log file has a unique identity. During the recycling process, after being opened each time, an index number of the online log file can be generated to identify the sequence of generation of the log data. The online log file can be archived and can be put into the recycling process after archiving. When recycled again, a new index number can be generated for the online log file to distinguish different log data through the index number.

[0077] Among them, any one of the database nodes 401 is used to: obtain the transaction log of the target transaction by using the storage engine; write the transaction log into the target online log file in the at least one online log file; the target online log file is an online log file that supports write operations at any position; and persistently store the transaction log in the target online log file on the target block device associated with the target online log in the at least one block device.

[0078] In some optional embodiments, when the database node 401 writes the transaction log into the target online log file, it is specifically used to: determine the first online log file in the startup state by using the storage engine; judge whether the data volume of the transaction log is greater than the remaining writable data volume of the first online log file; if so, close the first online log file and start a new second online log file as the target online log file; and write the transaction log into the target online log file.

[0079] In some alternative embodiments, when writing the transaction log to the target online log file, the database node 401 is specifically configured to: use a storage engine to write the transaction log to a target memory cache block corresponding to the target online log file; optionally, the size of the target memory cache block is the same as the storage space size of the target block device. Correspondingly, when persistently storing the transaction log in the target online log file to the corresponding target block device, the database node 401 is specifically configured to: synchronize the transaction log written in the target memory cache block to the target block device associated with the target online log file according to a log synchronization parameter.

[0080] In some alternative embodiments, the database node 401 is further configured to: use a storage engine to determine the log synchronization parameter according to at least one of the load information of the distributed database, the throughput capacity information of the target block device, and the backlog information of the online log files to be synchronized in the distributed database.

[0081] In some alternative embodiments, the distributed database system further includes: a storage management component 404. The database node 401 is further configured to: during the process of writing the transaction log to the target online log file, use a storage engine to determine a first block device already associated with the target online log file; if the data volume of the transaction log is greater than that of the first block device, send a block device application to the storage management component 404. The storage management component 404 is configured to: allocate a second block device for the target online log from the at least one block device. Correspondingly, the database node 401 is further configured to: use a storage engine to associate the target online log file with the second block device to expand the target online log file.

[0082] In some alternative embodiments, the database node 401 is further configured to: during the process of persisting the transaction log, in response to a commit instruction of the target transaction, write the metadata information of the target online log file to the redo log file of the target transaction; and write the commit event of the target transaction to the redo log file of the target transaction; and after completing the persistent storage of the transaction log, complete the commit operation of the target transaction.

[0083] Optionally, the database node 401 is further configured to: after the target transaction is committed, perform a persistence operation on the redo log file of the target transaction to perform a transaction-level persistence operation on the metadata information of the target online log file; or after the transaction group to which the target transaction belongs is committed, perform a persistence operation on the redo log file of the target transaction to perform a transaction group-level persistence operation on the metadata information of the target online log file.

[0084] Optionally, when the database node 401 performs a transaction-level persistent storage operation on the metadata information of the target online log file, it is specifically used for: after the target transaction is committed, using the storage engine to store the metadata information of the target online log file as incremental logs into the redo log file of the distributed database system; or, after the transaction group to which the target transaction belongs is committed, using the storage engine to store the metadata information of the target online log file as incremental logs into the redo log file of the distributed database system.

[0085] Optionally, the target online log file includes: an initial online log file created during database node initialization or an online log file that is recycled after being archived.

[0086] In some alternative embodiments, after the database node 401 persistently stores the transaction logs in the target online log file to the corresponding target block device, it is further used for: using the storage engine to obtain the current index number of the target online log file; adding an archive flag to the target online log file corresponding to the current index number; clearing the transaction logs in the target online log file and updating the index number of the target online log file; and restoring the target online log file with the updated index number to an available online log file.

[0087] In this embodiment, a single database node in the distributed database system can directly store transaction logs in a block storage manner based on its own storage engine component without relying on the file system, reducing the resource overhead caused by introducing the file system and the file storage method. The storage engine component writes the transaction logs into the online log file, and can make full use of the characteristic that the online log file can be recycled, without the need to perform the creation operation of binary logs, further reducing the complexity and resource overhead of the log storage operation. In addition, the database node can use its own characteristic of being able to sense whether a transaction ends to perform a transaction-level persistent storage operation on the metadata information of the target online log file, which can ensure that the persisted metadata information is the complete metadata information of the target transaction, improving the controllability and reliability of the persistent storage operation of the metadata information.

[0088] Figure 5 Schematically shows a schematic structural diagram of a server provided by an exemplary embodiment of the present application, and this server is applicable to the log storage method provided by the foregoing embodiment. As Figure 5 shown, this server includes: a memory 501, a processor 502, and a communication component 503.

[0089] A memory 501 for storing computer programs and configurable to store various other data to support operations on the server. Examples of such data include instructions for any application or method operating on the server.

[0090] A processor 502, coupled to the memory 501, for executing the computer program in the memory 501 to: obtain a transaction log of a target transaction using a storage engine component; write the transaction log to a target online log file; the target online log file being an online log file that supports write operations at any location; and persistently store the transaction log in the target online log file to a corresponding target block device.

[0091] Optionally, when writing the transaction log to the target online log file, the processor 502 is specifically configured to: determine a first online log file in a startup state using a storage engine component; determine whether the data volume of the transaction log is greater than the remaining writable data volume of the first online log file; if so, close the first online log file and start a new second online log file as the target online log file; and write the transaction log to the target online log file.

[0092] Optionally, when writing the transaction log to the target online log file, the processor 502 is specifically configured to: write the transaction log to a target memory cache block corresponding to the target online log file using a storage engine component; correspondingly, when persistently storing the transaction log in the target online log file to a corresponding target block device, the processor 502 is specifically configured to: synchronize the transaction log written in the target memory cache block to the target block device associated with the target online log file according to log synchronization parameters using a storage engine component.

[0093] Optionally, the processor 502 is further configured to: determine the log synchronization parameters according to at least one of the load information of the distributed database, the throughput capacity information of the target block device, and the backlog information of the online log files to be synchronized in the distributed database using a storage engine component.

[0094] Optionally, the storage space sizes of the target memory cache block and the target block device are the same.

[0095] Optionally, the processor 502 is further configured to: determine a first block device already associated with the target online log file using a storage engine component during the process of writing the transaction log to the target online log file; if the data volume of the transaction log is greater than the first block device, apply to a storage management component for allocation of a second block device; and associate the target online log file with the second block device to expand the target online log file.

[0096] Optionally, the processor 502 is further configured to: during the process of persisting the transaction log, in response to a commit instruction of the target transaction, write metadata information of the target online log file into the redo log file of the target transaction; and write a commit event of the target transaction into the redo log file of the target transaction; and after the persistent storage of the transaction log is completed, complete the commit operation of the target transaction.

[0097] Optionally, the processor 502 is further configured to: after the target transaction is committed, perform a persistence operation on the redo log file of the target transaction to perform a transaction-level persistence operation on the metadata information of the target online log file; or after the transaction group to which the target transaction belongs is committed, perform a persistence operation on the redo log file of the target transaction to perform a transaction-group-level persistence operation on the metadata information of the target online log file.

[0098] Optionally, the target online log file includes: an initial online log file created during database node initialization or an online log file that is archived and then put into circular use.

[0099] Optionally, after the processor 502 persists the transaction log in the target online log file to the corresponding target block device, the processor 502 is further configured to: archive the target online log file to a specified storage space and update the target online log file to an archived state, so that the target online log file can be put into circular use after being archived.

[0100] Furthermore, as Figure 5 shown, the server further includes: other components such as a power supply component 504. Figure 5 Only some components are schematically shown, and it does not mean that the server only includes Figure 5 the components shown.

[0101] Among them, the memory 501 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0102] Among them, the communication component 503 is configured to facilitate communication between the device where the communication component is located and other devices in a wired or wireless manner. The device where the communication component is located can access a wireless network based on a communication standard, such as Wi-Fi (wireless network communication technology), 2G (such as Global System for Mobile Communications (GSM), etc.), 3G (such as Wideband Code Division Multiple Access (WCDMA)), 4G (such as Long Term Evolution (LTE), etc.), 4G+ (such as LTE-Advanced (LTE-A), etc.) or 5G (5th Generation Mobile Communication Technology), or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can be implemented based on near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wide band (UWB) technology, Bluetooth (BT) technology and other technologies.

[0103] Among them, the power supply component 504 is used to provide power for various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device where the power supply component is located.

[0104] In this embodiment, the database node can directly store the transaction log in a block storage manner based on its own storage engine component without relying on the file system, reducing the resource overhead caused by introducing the file system and the file storage method. The storage engine component writes the transaction log into the online log file, and can make full use of the characteristic that the online log file can be recycled, without performing the operation of creating the binary log, further reducing the complexity and resource overhead of the log storage operation. In addition, the database node can utilize its own characteristic of being able to sense whether the transaction ends, and perform a transaction-level persistent storage operation on the metadata information of the target online log file, which can ensure that the persistent metadata information is the complete metadata information of the target transaction, improving the controllability and reliability of the persistent operation of the metadata information.

[0105] Correspondingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed, it can implement each step executable by the server in the above method embodiment.

[0106] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM (Compact Disc Read-Only Memory), optical storage, etc.) containing computer-usable program code.

[0107] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for realizing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0108] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction means that implements the function specified in one or more of the procedures Figure 1 or more procedures and / or blocks Figure 1 or more blocks specified in the block.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide steps for implementing the function specified in one or more of the procedures Figure 1 or more procedures and / or blocks Figure 1 or more blocks specified in the block.

[0110] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), an input / output interface, a network interface, and memory.

[0111] The memory may include non-permanent memory in the computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

[0112] Computer-readable media includes permanent and non-permanent, removable and non-removable media implemented by any method or technology for storing information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (Parallel Random Access Machine, PRAM), static random access memory (SRAM), dynamic random access memory (Dynamic Random Access Memory, DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (Digital Video Disc, DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0113] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising the said element.

[0114] The above are only examples of the present application and are not intended to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the scope of the claims of the present application.

Claims

1. A log storage method, applicable to any database node in a distributed database, characterized in that, Including: Obtaining the transaction log of the target transaction by using a storage engine component; Writing the transaction log into a target online log file; The target online log file is an online log file that supports write operations at any position; Persistently storing the transaction log in the target online log file into a corresponding target block device.

2. The method according to claim 1, wherein Writing the transaction log into the target online log file includes: Determining a first online log file in a startup state; Judging whether the data volume of the transaction log is greater than the remaining writable data volume of the first online log file; If so, closing the first online log file and starting a new second online log file as the target online log file; Writing the transaction log into the target online log file.

3. The method according to claim 1, characterized in that Writing the transaction log into the target online log file includes: Writing the transaction log into a target memory cache block corresponding to the target online log file; Persistently storing the transaction log in the target online log file into a corresponding target block device includes: Synchronizing the transaction log written in the target memory cache block to a target block device associated with the target online log file according to log synchronization parameters.

4. The method according to claim 3, characterized in that It also includes: Determining the log synchronization parameters according to at least one of the load information of the distributed database, the throughput capacity information of the target block device, and the backlog information of the online log files to be synchronized in the distributed database.

5. The method according to claim 3, characterized in that, The storage space sizes of the target memory cache block and the target block device are the same.

6. The method according to any one of claims 1-5, characterized in that, It also includes: During the process of writing the transaction log into the target online log file, determining a first block device already associated with the target online log file; If the data volume of the transaction log is greater than the first block device, applying to a storage management component for allocating a second block device; Associating the target online log file with the second block device to expand the target online log file.

7. The method according to any one of claims 1-5, characterized in that, It also includes: During the process of persisting the transaction log, in response to a commit instruction of the target transaction, writing metadata information of the target online log file into a redo log file of the target transaction; And, Writing a commit event of the target transaction into the redo log file of the target transaction; And, After completing the persistent storage of the transaction log, completing the commit operation of the target transaction.

8. The method according to claim 7, wherein It also includes: After the target transaction is committed, performing a persistence operation on the redo log file of the target transaction to perform a transaction-level persistence operation on the metadata information of the target online log file; Or, After a transaction group to which the target transaction belongs is committed, performing a persistence operation on the redo log file of the target transaction to perform a transaction group-level persistence operation on the metadata information of the target online log file.

9. The method according to any one of claims 1-5, characterized in that, The target online log file includes: an initial online log file created during database node initialization or an online log file that is put into circular use after being archived.

10. The method according to any one of claims 1-5, characterized in that, After persistently storing the transaction log in the target online log file into a corresponding target block device, it also includes: Archive the target online log file to a specified storage space and update the target online log file to an archived state, so that the target online log file can be recycled after archiving.

11. A distributed database system, characterized in that, Including: Multiple database nodes, a database file component, and a storage component; wherein, The storage component is used to provide at least one block device; The database file component is used to provide at least one online log file; Any one of the multiple database nodes is used to: obtain the transaction log of the target transaction by using the storage engine component; write the transaction log into the target online log file in the at least one online log file; the target online log file is an online log file that supports write operations at any position; persistently store the transaction log in the target online log file to the target block device associated with the target online log in the at least one block device.

12. The system according to claim 11, wherein The system further includes: a storage management component; The storage engine component is further used to: determine the first block device associated with the target online log file during the process of writing the transaction log into the target online log file; if the data volume of the transaction log is greater than the first block device, send a block device application to the storage management component, so that the storage management component allocates a second block device for the target online log from the at least one block device; associate the target online log file with the second block device to expand the target online log file.

13. A server, characterized in that, Including: A memory and a processor; The memory is used to store one or more computer instructions; The processor is used to execute the one or more computer instructions for: performing the steps in the method according to any one of claims 1-10.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement the log storage method according to any one of claims 1-10.

Images