Method, device and system for processing pre-written logs

Abstract

The embodiment of the present specification provides a method, device and system for processing pre-written logs, wherein the method for processing pre-written logs comprises: providing a publishing interface and a subscribing interface based on a message queue communication protocol; receiving a pre-written log writing request through the publishing interface; obtaining a pre-written log record according to data carried by the pre-written log writing request; storing the pre-written log record into a message queue, so that a subscribing end consumes the log record through the subscribing interface; wherein a log serial number in the pre-written log record is determined based on a corresponding rule between a message offset and the log serial number, the message offset is a message offset of the pre-written log record in the message queue, and the log serial number is used to make the subscribing end determine a corresponding message offset of the pre-written log record to be played back based on the corresponding rule between the message offset and the log serial number.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a method, apparatus and system for processing write-ahead logs. Background Technology

[0002] With the development of computer technology, databases, as crucial foundational software providing data storage and retrieval capabilities, have been widely adopted. Among these technologies, write-ahead logging (WAL) is a technique used in database systems to provide atomicity and durability. This technique ensures that any changes to the database are written to the log before taking effect.

[0003] In existing database architectures, the write-ahead log (WAL) functionality is implemented within the database's storage engine. The database's computation engine writes data to the storage engine and loads it into memory as needed for forwarding to external applications via HTTP requests. The storage engine then persists the data written by the computation engine to the WAL. The data persisted to the WAL is further updated in the data storage module. However, this architecture not only places a significant processing burden on both the computation and storage engines but also introduces substantial data redundancy, leading to a decline in database performance. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a method for processing a write-ahead log. One or more embodiments of this specification also relate to an apparatus for processing a write-ahead log, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a method for processing write-ahead logs is provided, comprising: providing a publish interface and a subscribe interface based on a message queue communication protocol; receiving a write-ahead log request through the publish interface; obtaining a write-ahead log record according to the data carried by the write-ahead log request; storing the write-ahead log record in a message queue, and enabling a subscriber to consume the log record through the subscription interface; wherein, the log sequence number in the write-ahead log record is determined based on a correspondence rule between message offset and log sequence number, the message offset is the message offset of the write-ahead log record in the message queue, and the log sequence number is used to enable the subscriber to determine the message offset corresponding to the write-ahead log record to be replayed based on the correspondence rule between message offset and log sequence number.

[0006] Optionally, receiving a write-ahead log request through the publishing interface includes: receiving a write-ahead log request issued by the database's computing engine through the publishing interface, wherein the write-ahead log request is issued by the computing engine in response to receiving a data write request, after converting the data carried in the data write request into a write-ahead log format.

[0007] Optionally, storing the write-ahead log record into the message queue includes: storing the write-ahead log record into the message queue under the message topic according to the message topic to which the database that published the write-ahead log record is mapped, wherein there is a one-to-one mapping relationship between the database and the message topic.

[0008] Optionally, the content of a single log record in the write-ahead log record includes: description information for each of the multiple fields and log data for each of the multiple fields.

[0009] Optionally, it also includes: using a distributed consistency protocol to maintain consistency of the write-ahead log with other write-ahead log component nodes.

[0010] According to a second aspect of the embodiments of this specification, an apparatus for processing write-ahead logs is provided, comprising: an interface providing module configured to provide a publish interface and a subscribe interface based on a message queue communication protocol; a request receiving module configured to receive write-ahead log requests through the publish interface; a record acquisition module configured to acquire write-ahead log records based on data carried by the write-ahead log request; and a record storage module configured to store the write-ahead log records in a message queue, enabling subscribers to consume the log records through the subscription interface. The log sequence number in the write-ahead log record is determined based on a correspondence rule between message offsets and log sequence numbers. The message offset is the message offset of the write-ahead log record in the message queue, and the log sequence number is used to enable the subscriber to determine the message offset corresponding to the write-ahead log record to be replayed based on the correspondence rule between message offsets and log sequence numbers.

[0011] According to a third aspect of the embodiments of this specification, a system for processing write-ahead logs is provided, comprising: a database and a write-ahead log component implemented using the method for processing write-ahead logs as described in any embodiment of this specification. The database includes: a computing engine configured to send a write-ahead log write request to the write-ahead log component through a publishing interface provided by the write-ahead log component in response to a data write request; and a storage engine configured to act as a subscriber, consuming write-ahead log records through a subscription interface provided by the write-ahead log component, and, in the event of a storage exception, determining the message offset corresponding to the write-ahead log record that needs to be replayed based on a correspondence rule between message offsets and log sequence numbers.

[0012] Optionally, the database can be any database in a database cluster, which includes a primary database cluster and a backup database cluster.

[0013] Optionally, it also includes an external application other than the database. The external application is configured to consume write-ahead log records as a subscriber through the subscription interface provided by the write-ahead log component.

[0014] According to a fourth aspect of the embodiments of this specification, a computing device is provided, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the steps of the method for processing write-ahead logs as described in any embodiment of this specification.

[0015] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the method for processing a write-ahead log as described in any embodiment of this specification.

[0016] According to a sixth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the method for processing write-ahead logs as described in any embodiment of this specification.

[0017] This specification implements an embodiment of a method for processing write-ahead logs. Since the method provides a publish interface and a subscribe interface based on a message queue communication protocol, it receives write-ahead log write requests through the publish interface, obtains write-ahead log records based on the data carried in the write-ahead log write request, stores the write-ahead log records in a message queue, and allows subscribers to consume the log records through the subscription interface. Furthermore, the log sequence number in the write-ahead log record is determined based on the correspondence rule between message offset and log sequence number. The message offset is the message offset of the write-ahead log record in the message queue. Therefore, through this log sequence number, the subscriber can determine the message offset corresponding to the write-ahead log record that needs to be replayed based on the correspondence rule between message offset and log sequence number. This not only ensures the functionality of the write-ahead log, but also allows subscribers, such as external programs or database data storage modules, to obtain data simply by subscribing based on the message queue communication protocol. This reduces the processing operations of the database's computing engine, reduces redundant data storage, and effectively improves database performance. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a method for processing a write-ahead log according to one embodiment of this specification;

[0019] Figure 2 This is a schematic diagram of the system architecture in a database application scenario provided by one embodiment of this specification;

[0020] Figure 3 This is a schematic diagram of a database cluster system architecture provided in one embodiment of this specification;

[0021] Figure 4 This is a schematic diagram of timing data recording provided in one embodiment of this specification;

[0022] Figure 5 This is a schematic diagram of timing data organization logic provided in one embodiment of this specification;

[0023] Figure 6 This is a schematic diagram of the structure of an apparatus for processing write-ahead logs according to one embodiment of this specification;

[0024] Figure 7 This is a schematic diagram of the structure of a system for processing write-ahead logs according to one embodiment of this specification;

[0025] Figure 8 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0026] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0027] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0028] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0029] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0030] Write-Ahead Log (WAL) is a technique used in database systems to provide atomicity and durability, and it is an important component of database systems. WAL ensures that any changes to the database are written to the log before they actually take effect.

[0031] The Log Sequence Number (LSN) is the unique identifier for each write-ahead log record. Database write-ahead logs use LSNs to ensure the order of changes and to enable data recovery based on write-ahead logs during database failure recovery.

[0032] The publish / subscribe pattern is a message transmission paradigm in software architecture. In this pattern, the party sending a message (called the "publisher") does not directly communicate with the party receiving the message (called the "subscriber") and they do not need to be aware of each other's existence. Typically, a message queue middleware product and its dependent producers and consumers constitute a publish / subscribe pattern.

[0033] Message queues, also known as "message middleware," are fundamental software components in distributed systems that handle message sending and receiving. They enable platform-independent data exchange through efficient and reliable message passing mechanisms and facilitate the integration of distributed systems based on data communication. By providing message passing and message queue models, they can extend process communication in distributed environments.

[0034] A message offset is a sequence number used to locate a specific message in a message queue. The message offset increments as messages are written and is typically unique globally or within a specific data shard.

[0035] This specification provides a method for processing a write-ahead log, and also relates to an apparatus for processing a write-ahead log, a system for processing a write-ahead log, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0036] See Figure 1 , Figure 1 A flowchart is shown of a method for processing a write-ahead log according to an embodiment of this specification, which specifically includes the following steps.

[0037] Step 102: Provide publish and subscribe interfaces based on the message queue communication protocol.

[0038] The specific implementation of the message queue is not limited. For example, the message queue can be a persistent memory-based message queue. In a persistent memory-based message queue, data remains in the message queue after being consumed, so that the data can be consumed again. For example, the message queue communication protocol can be the Kafka Protocol or other similar message queue communication protocols. For example, APIs implemented based on the Kafka communication protocol include, but are not limited to:

[0039] -Produce

[0040] -Fetch

[0041] -ListOffsets

[0042] -OffsetFetch

[0043] -OffsetCommit

[0044] -JoinGroup

[0045] -......

[0046] The Produce API is used to implement the publish interface, which receives write-ahead log requests. The remaining APIs can be used to implement the subscribe interface so that subscribers can consume data.

[0047] Step 104: Receive a write-ahead log request through the publish interface.

[0048] The publishing end that issues the write-ahead log (BAG) request is not limited. For example, in a database application scenario, the database's computing engine can send a BAG write request through the publishing interface. This BAG write request can be generated according to a message queue communication protocol and carries the data to be written.

[0049] Step 106: Obtain the write-ahead log record based on the data carried in the write-ahead log request.

[0050] The format of the log data carried in the write-ahead request is not limited. For example, the log data can be in a non-write-ahead format. After parsing the log data from the write-ahead request, the log data is converted to the write-ahead format to generate a write-ahead log record. Alternatively, the log data can be obtained by the computing engine from the application and then converted to the write-ahead format. Therefore, after parsing the log data from the write-ahead request, the write-ahead log component can directly obtain the write-ahead log record.

[0051] Step 108: Store the write-ahead log record into a message queue so that the subscriber can consume the log record through the subscription interface.

[0052] The log sequence number in the write-ahead log record is determined based on the correspondence rule between message offset and log sequence number. The message offset is the message offset of the write-ahead log record in the message queue. The log sequence number is used to enable the subscriber to determine the message offset corresponding to the write-ahead log record that needs to be replayed based on the correspondence rule between message offset and log sequence number.

[0053] The correspondence between the message offset and the log sequence number can be set according to the needs of the implementation scenario. For example, in one or more embodiments, the message offset can be directly used as the log sequence number to identify the write-ahead log record.

[0054] The subscriber can be any subscriber that requires pre-written log records. For example, the subscriber can include the storage engine of a database, or it can include an external application other than the database. The subscriber only needs to subscribe based on a message queue communication protocol.

[0055] Because the method provides publish and subscribe interfaces based on the message queue communication protocol, it receives write-ahead log requests through the publish interface, retrieves write-ahead log records based on the data carried in the write-ahead log requests, and stores the write-ahead log records in the message queue. Subscribers then consume these log records through the subscription interface. Furthermore, the log sequence number in the write-ahead log record is determined based on the correspondence between message offsets and log sequence numbers. The message offset is the message offset of the write-ahead log record in the message queue. This allows subscribers to determine the message offset corresponding to the write-ahead log record that needs to be replayed, based on the correspondence between message offsets and log sequence numbers. This not only ensures the functionality of the write-ahead log but also allows subscribers, such as external programs or database data storage modules, to obtain data simply by subscribing based on the message queue communication protocol. This reduces the processing operations of the database's computing engine, reduces redundant data storage, and effectively improves database performance.

[0056] Furthermore, since the method provided in the embodiments of this specification provides publish and subscribe interfaces based on message queue communication protocols, it realizes the capabilities of message queues, thus truly achieving a publish / subscribe data model. This aligns with the commonly existing hybrid data subscription application scenarios, achieving better real-time performance. This is because the essence of the data consumed in real-time by the subscriber is to promptly sense data changes. Therefore, separating the WAL component from the database storage engine and providing services externally elevates the consumption target, allowing the subscriber to be closer to the consumed data, resulting in better real-time performance. Moreover, the subscription capabilities provided by message queue communication protocols such as the Kafka Protocol facilitate the access of various data links.

[0057] The application scenarios of the write-ahead log (WAL) processing methods provided in the embodiments of this specification are not limited. The write-ahead log component implemented according to the methods provided in the embodiments of this specification can be an independent component, decoupling the write-ahead log from the database engine. In database application scenarios, the methods provided in the embodiments of this specification can realize the inherent sequential write log capability of WAL, mapping the concept of log sequence number in the WAL log to the concept of message offset in the message queue, and exposing the write-ahead log subscription capability to the database storage engine and other applications through, for example, the Kafka Protocol. By enabling external services through the WAL component, the database data write chain no longer needs to focus on data forwarding, thereby achieving further centralized utilization of server resources in write scenarios and effectively improving database performance. The high availability of the WAL component itself ensures the stability of the data subscription chain. Logically, it guarantees that as long as data is successfully written to the time-series database, it can definitely be consumed by the subscriber. This achieves the separation of write-ahead logs and data storage in the database engine, more closely integrating write-ahead logs with the stream computing ecosystem.

[0058] For example, such as Figure 2The system architecture diagram in a database application scenario is shown, including: a database, a WAL component, and one or more external applications acting as subscribers. The database includes a computing engine and a data storage module acting as a subscriber. The data storage module corresponds to the storage engine. The data writing process includes: the computing engine receiving write requests from project applications, sending the corresponding data along with the write-ahead log (WAL) write request to the WAL component, and the WAL component generating the corresponding write-ahead log record and storing it in a message queue. The data storage module, acting as a subscriber to the WAL component, consumes the log data, thus saving the corresponding data in an in-memory MemTable at the storage layer. The storage layer can asynchronously persist the data in the MemTable to data files using methods commonly used by database engines. It is understood that all subscribers can consume log data in groups based on, for example, the "Kafka protocol." It is also understood that when the storage engine applies WAL data to the MemTable, it needs to record the log sequence number corresponding to the data write to ensure the consistency of the order when the storage engine flushes the data from the MemTable to storage. Only after the storage engine has officially flushed the data corresponding to the log sequence number to storage can the log be considered truly consumed. Since consumption in message queue mode is based on offsets, when a database partition crashes, the range of message offsets of the write-ahead logs that need to be replayed can be determined based on the correspondence between message offsets and log sequence numbers, the log sequence number of the write-ahead log, and the log sequence number flushed to disk by the storage engine. Data can then be replayed from the message queue of the WAL component.

[0059] In database application scenarios, for example, receiving a write-ahead log (WAP) write request through the publishing interface may include: receiving a WAP write request issued by the database's computing engine through the publishing interface, wherein the WAP write request is issued by the computing engine in response to receiving a data write request, after converting the data carried in the data write request into a WAP format. In this embodiment, converting log data into a WAP format through the database's computing engine can fully utilize the computing power of the computing engine and avoid processing pressure on the WAP component.

[0060] Since a database is a unit of data organization and access isolation, message topics can be used to isolate log data from different databases within a message queue. Write-ahead log records from different databases can be stored in the message queues corresponding to those topics. Each message topic has its own message queue. Therefore, storing write-ahead log records in a message queue can include: storing the write-ahead log records in the message queue under the message topic to which the database publishing the log records is mapped, where there is a one-to-one mapping between databases and message topics. For example, in a WAL component, a database can be mapped to a unit of WAL log organization, corresponding to the Topic concept in the Kafka Protocol.

[0061] To enable subscribers to easily identify the components of a write-ahead log record without requiring additional metadata, the method provided in this embodiment allows the content of a single log record to include descriptive information for multiple fields and the log data for each field. For example, in this embodiment, the write-ahead log record can be a relatively fixed semi-structured record. This record can clearly mark the structural description of the log data at the data structure level, such as the metric, tags, timestamp, and fields for time-series data. Because this record structure is self-describing, subscribers can easily identify the components of the time-series data without requiring additional metadata.

[0062] For example, the write-ahead log record can be a JSON string conforming to the RFC 7159 specification. The log data can be inserted between the "${" and "}" in the JSON string according to the described information. For example, the JSON structure is as follows:

[0063]

[0064] The above-described JSON string format for write-ahead logging is only used to illustrate the method provided in the embodiments of this specification. In practical applications, the format of write-ahead logging can also be set according to actual needs, such as according to serialization / deserialization protocols such as ProtocolBuffer and Avro.

[0065] It should be noted that the write-ahead log (WAL) component implemented according to the methods provided in the embodiments of this specification can be deployed on the same node as the database's computing and storage engines, or it can be deployed separately. The WAL component itself can also support high-availability deployment, that is, using multiple WAL nodes. The consistency of the write-ahead logs between multiple WAL nodes can be achieved through distributed consistency protocols such as the Raft protocol. Therefore, the method provided in the embodiments of this specification may further include: maintaining consistency of the write-ahead logs with other write-ahead log component nodes using distributed consistency protocols.

[0066] A write-ahead log component deployed in a high-availability manner can enable the deployment of a high-availability database cluster system and bidirectional writes between master and standby clusters. For example, ... Figure 3 As shown in the schematic diagram of the database cluster system architecture, the computing engine of the database in the primary database cluster receives write requests, converts the data into a log recording format, and sends it to the WAL component. The WAL component persists the logs using a message queue approach and utilizes a distributed consistency protocol to ensure WAL log consistency among WAL component nodes. Therefore, the WAL component achieves high availability based on log replication. The data storage module (corresponding storage engine) of the primary database cluster acts as a log consumer, consuming the logs from the WAL component and writing the data to a MemTable. The standby database cluster supports bidirectional writes. The storage engine of the standby database cluster, as another log consumer group, also consumes the same WAL logs, ensuring logical consistency of data between the standby and primary database clusters.

[0067] The methods provided in the embodiments of this specification are not limited to the type of database. For example, the method can be applied to the processing of write-ahead logs in time-series databases, and the system architecture can be as follows: Figure 2 , Figure 3 As shown, details will not be elaborated further here. Time-series databases, as crucial foundational software providing IoT data storage and query capabilities, are seeing increasingly specialized and segmented solutions as data-intensive applications continue to expand. While time-series databases can handle time-series storage and querying, they often face scenarios where time-series data is mixed with other data sources. Therefore, it's necessary to ensure that data stored in time-series databases can be subscribed to in real-time on multiple data links and mixed with data stored in other heterogeneous data sources. Thus, real-time data subscription is a typical application scenario for time-series database products.

[0068] To make the above application scenarios easier to understand, the following is an illustrative explanation of time series databases.

[0069] In Internet of Things (IoT) applications, devices continuously collect data over time. Therefore, the data collected by devices in IoT application systems typically needs to be stored uniformly to facilitate subsequent querying, analysis, and display. Data collected in this type of application is often referred to as "time-series data." Time-series data is a semi-structured data model with the following characteristics: each record has a timestamp; the same device can support the collection of one or more data metrics; data metrics collected at different times may not include all data metrics that the device can collect; data points can be distinguished from data points from other sources by a set of labels indicating their origin. For example, ... Figure 4 As shown in the diagram of a time-series data record, a single time-series data record can include multiple fields. These include fields such as "metric," "tags," "timestamp," "fields," and "values." In different time-series database implementations, the various components of a time-series data record, i.e., each field (also called a tag), can be configured as needed. Characteristics of time-series databases include: frequent access to hot data; relatively stable data write frequency, which is much greater than the data read frequency; and data typically queried according to time windows.

[0070] According to the method provided in the embodiments of this specification, data subscription capabilities are combined with WAL capabilities, and write-ahead logs are decoupled from the time-series database engine. In terms of data organization within the time-series database, such as... Figure 5 The time-series data organization logic diagram shown typically follows the following logical hierarchy:

[0071] Cluster: Represents a time-series database cluster, which can be a single-node cluster, for example.

[0072] Database: A logical concept that isolates data access.

[0073] Timeseries: The basic unit of time series data organization. A Timeseries represents a series of time series data reported from the same data source.

[0074] Datapoint: Time-series data record.

[0075] In this context, a database is a unit of data organization and access. Therefore, in the WAL component, a database can be mapped to a unit of WAL log organization, corresponding to the Topic concept in the Kafka Protocol.

[0076] Because time-series databases possess the following two characteristics: data changes only involve new writes; and the data model is relatively fixed, the write-ahead log (LAB) can be designed as a relatively fixed semi-structured record, combining one or more of the above embodiments. This clearly marks the time-series data's metrics, tags, timestamps, and fields at the data structure level. Since the log record structure is self-describing, the subscriber can easily identify the various components of the time-series data without requiring additional metadata.

[0077] By combining the method for processing write-ahead logs provided in one or more embodiments of this specification with the real-time subscription scenario requirements of time-series data, a write-ahead log component can be implemented in a time-series database product, thereby expanding the functional scope of write-ahead logs in time-series databases and enabling write-ahead logs to provide native data subscription capabilities on the basis of satisfying the persistence of time-series database data.

[0078] Corresponding to the above method embodiments, this specification also provides embodiments of an apparatus for processing write-ahead logs. Figure 6 A schematic diagram of an apparatus for processing a write-ahead log according to one embodiment of this specification is shown. Figure 6 As shown, the device includes: an interface providing module 602, a request receiving module 604, a record acquisition module 606, and a record storage module 608.

[0079] This interface provides module 602, which can be configured to provide publish and subscribe interfaces based on the message queue communication protocol.

[0080] The request receiving module 604 can be configured to receive write-ahead log requests through the publishing interface.

[0081] The record acquisition module 606 can be configured to acquire write-ahead log records based on the data carried by the write-ahead log write request.

[0082] The record storage module 608 can be configured to store the write-ahead log records into a message queue, so that the subscriber can consume the log records through the subscription interface.

[0083] The log sequence number in the write-ahead log record is determined based on the correspondence rule between message offset and log sequence number. The message offset is the message offset of the write-ahead log record in the message queue. The log sequence number is used to enable the subscriber to determine the message offset corresponding to the write-ahead log record that needs to be replayed based on the correspondence rule between message offset and log sequence number.

[0084] Because the device provides publish and subscribe interfaces based on the message queue communication protocol, it receives write-ahead log requests through the publish interface, retrieves write-ahead log records based on the data carried in the write-ahead log requests, and stores the write-ahead log records in the message queue. Subscribers then consume these log records through the subscription interface. Furthermore, the log sequence number in the write-ahead log record is determined based on the correspondence between message offsets and log sequence numbers. The message offset is the message offset of the write-ahead log record in the message queue. This log sequence number allows subscribers to determine the message offset corresponding to the write-ahead log record that needs to be replayed, based on the correspondence between message offsets and log sequence numbers. This not only ensures the functionality of the write-ahead log but also allows subscribers, such as external programs or database data storage modules, to obtain data simply by subscribing based on the message queue communication protocol. This reduces the processing operations of the database's computing engine, reduces redundant data storage, and effectively improves database performance.

[0085] In one or more embodiments of this specification, the interface providing module 602 can be configured to receive a write-ahead log request issued by the computing engine of the database through the publishing interface, wherein the write-ahead log request is issued by the computing engine in response to receiving the data write request, after converting the data carried in the data write request into the format of the write-ahead log.

[0086] In one or more embodiments of this specification, the record storage module 608 can be configured to store the write-ahead log record in a message queue under the message topic according to the message topic to which the database that publishes the write-ahead log record is mapped, wherein there is a one-to-one mapping relationship between the database and the message topic.

[0087] The content of a single log record in the write-ahead log record may include: description information of each of the multiple fields and log data of each of the multiple fields.

[0088] In addition, the device can also use a distributed consensus protocol to maintain consistency of the write-ahead log with other write-ahead log component nodes.

[0089] The above is an illustrative scheme of an apparatus for processing write-ahead logs according to this embodiment. It should be noted that the technical solution of this apparatus for processing write-ahead logs and the technical solution of the above-described method for processing write-ahead logs belong to the same concept. For details not described in detail in the technical solution of the apparatus for processing write-ahead logs, please refer to the description of the technical solution of the above-described method for processing write-ahead logs.

[0090] Corresponding to the above method embodiments, this specification also provides system embodiments for processing write-ahead logs. Figure 7A schematic diagram of the structure of a system for processing write-ahead logs according to one embodiment of this specification is shown. Figure 7 As shown, the system includes: a database 702 and a write-ahead log component 704 implemented using the method for processing write-ahead logs as described in any embodiment of this specification.

[0091] The database 702 includes:

[0092] The computing engine 7022 can be configured to send a write-ahead log request to the write-ahead log component through the publishing interface provided by the write-ahead log component in response to a data write request.

[0093] Storage engine 7024 can be configured as a subscriber to consume write-ahead log records through the subscription interface provided by the write-ahead log component. When a storage exception occurs, it determines the message offset corresponding to the write-ahead log record that needs to be replayed based on the correspondence rule between message offset and log sequence number.

[0094] Because the system's write-ahead log component provides publish and subscribe interfaces based on the message queue communication protocol, it receives write-ahead log requests through the publish interface, retrieves write-ahead log records based on the data carried in the write-ahead log requests, and stores the write-ahead log records in the message queue. Subscribers then consume these log records through the subscription interface. Furthermore, the log sequence number in the write-ahead log record is determined based on the correspondence between message offsets and log sequence numbers. The message offset is the message offset of the write-ahead log record in the message queue. This log sequence number allows subscribers to determine the message offset corresponding to the write-ahead log record that needs to be replayed, based on the correspondence between message offsets and log sequence numbers. This not only ensures the functionality of the write-ahead log but also allows subscribers, such as external programs or database data storage modules, to obtain data simply by subscribing based on the message queue communication protocol. This reduces the processing operations of the database's computing and storage engines, reduces redundant data storage, and effectively improves database performance.

[0095] Wherein, database 702 can be any database in a database cluster, which includes a primary database cluster and a backup database cluster. For example, ... Figure 3 The diagram shows the database cluster system architecture. Database 702 can be any one or more databases in the primary database cluster or the backup database cluster.

[0096] Additionally, the system may also include an external application 706 other than the database 702. The external application 706 can be configured as a subscriber to consume write-ahead log records through the subscription interface provided by the write-ahead log component. For example, as... Figure 2The system architecture diagram shown depicts several external applications acting as subscribers to the WAL component, consuming write-ahead log records.

[0097] The above is an illustrative scheme of a system for processing write-ahead logs according to this embodiment. It should be noted that the technical solution of this system for processing write-ahead logs and the technical solution of the method for processing write-ahead logs described above belong to the same concept. For details not described in detail in the technical solution of the system for processing write-ahead logs, please refer to the description of the technical solution of the method for processing write-ahead logs described above.

[0098] Figure 8 A structural block diagram of a computing device 800 according to one embodiment of this specification is shown. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0099] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0100] In one embodiment of this specification, the above-described components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0101] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 800 can also be a mobile or stationary server.

[0102] The processor 820 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described method for processing write-ahead logs.

[0103] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-described method for processing write-ahead logs belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-described method for processing write-ahead logs.

[0104] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described method for processing write-ahead logs.

[0105] The above is an illustrative embodiment of a computer-readable storage medium. It should be noted that the technical solution of this storage medium and the technical solution of the above-described method for processing write-ahead logs belong to the same concept. Details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the above-described method for processing write-ahead logs.

[0106] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described method for processing write-ahead logs.

[0107] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the above-described method for processing write-ahead logs belong to the same concept. Details not described in detail in the computer program's technical solution can be found in the description of the technical solution of the above-described method for processing write-ahead logs.

[0108] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0109] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0110] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0111] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0112] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for processing write-ahead logs, comprising: Provide publish and subscribe interfaces based on the message queue communication protocol; Receive write-ahead log requests through the published interface; The write-ahead log record is obtained based on the data carried in the write-ahead log request. The write-ahead log records are stored in a message queue, and the subscriber consumes the log records through the subscription interface. The write-ahead log records are stored in the message queue using a write-ahead log component that is independent of the database. The log sequence number in the write-ahead log record is determined based on the correspondence rule between message offset and log sequence number. The message offset is the message offset of the write-ahead log record in the message queue. The log sequence number is used to enable the subscriber to determine the message offset corresponding to the write-ahead log record that needs to be replayed based on the correspondence rule between message offset and log sequence number.

2. The method according to claim 1, wherein receiving the write-ahead log write request through the publishing interface includes: The database receives write-ahead log requests from its computing engine via the publishing interface. These write-ahead log requests are issued by the computing engine in response to receiving a write-ahead request, after the engine converts the data carried in the write-ahead request into a write-ahead log format.

3. The method according to claim 1, wherein storing the write-ahead log record into the message queue comprises: Based on the message topic to which the database that publishes the write-ahead log record is mapped, the write-ahead log record is stored in the message queue under the message topic, wherein there is a one-to-one mapping relationship between the database and the message topic.

4. The method according to claim 1, wherein the content of a single log record in the write-ahead log record includes: The description information of each of the multiple fields and the log data of each of the multiple fields.

5. The method according to claim 1, further comprising: The distributed consistency protocol is used to maintain consistency between the write-ahead log and other write-ahead log component nodes.

6. An apparatus for processing a write-ahead log, comprising: The interface provider module is configured to provide publish and subscribe interfaces based on the message queue communication protocol. The request receiving module is configured to receive write-ahead log requests through the publishing interface; The record acquisition module is configured to acquire write-ahead log records based on the data carried by the write-ahead log write request. The record storage module is configured to store the write-ahead log records into a message queue, so that the subscriber can consume the log records through the subscription interface, wherein the write-ahead log records are stored in the message queue using a write-ahead log component independent of the database; The log sequence number in the write-ahead log record is determined based on the correspondence rule between message offset and log sequence number. The message offset is the message offset of the write-ahead log record in the message queue. The log sequence number is used to enable the subscriber to determine the message offset corresponding to the write-ahead log record that needs to be replayed based on the correspondence rule between message offset and log sequence number.

7. A system for processing a write-ahead log, comprising: The database and the write-ahead log component implemented by the method of processing write-ahead logs as described in any one of claims 1-5; The database includes: The computing engine is configured to send a write-ahead log request to the write-ahead log component through the publish interface provided by the write-ahead log component in response to a data write request. The storage engine is configured to consume write-ahead log records through the subscription interface provided by the write-ahead log component as a subscriber. When a storage exception occurs, it determines the message offset corresponding to the write-ahead log record that needs to be replayed based on the correspondence rule between message offset and log sequence number.

8. The system according to claim 7, wherein the database is any database in a database cluster, and the database cluster includes a primary database cluster and a backup database cluster.

9. The system according to claim 7, further comprising: External applications other than the database; The external application is configured to consume write-ahead log records as a subscriber through the subscription interface provided by the write-ahead log component.

10. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method for processing write-ahead logs as described in any one of claims 1 to 5.

11. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the method for processing a write-ahead log as described in any one of claims 1 to 5.

12. A computer program product comprising computer instructions that, when executed by a processor, implement the steps of the method for processing a write-ahead log as described in any one of claims 1 to 5.

Images