A data synchronization method for a database master-slave cluster and a related product

Abstract

The application provides a data synchronization method for a database master-slave cluster and related products. The data synchronization method for the database master-slave cluster comprises the following steps: in response to prewrite log data being temporarily stored in a log buffer of a master database, initiating a persistence process for the prewrite log data; and before the persistence process is completed, in response to the prewrite log data meeting a flow replication reading condition in the log buffer, initiating a process of directly reading the prewrite log data from the log buffer and performing flow replication to at least one slave database. According to the scheme of the application, a memory area meeting the flow replication reading condition in the log buffer is locked in a shared reading mode, so that a flow replication process can directly refer to a memory address of the area for data reading, without the need of copying data from the memory buffer to a user buffer, thereby reducing the number of system call times of network sending operations and reducing protocol stack encapsulation overhead.

Description

Technical Field

[0001] This invention relates to database technology, and in particular to a data synchronization method and related products for database master-slave clusters. Background Technology

[0002] Database master-slave clusters ensure data consistency through write-ahead log data. The master database must wait for the write-ahead log data to be written to its local disk before it can read the persisted log content from the disk file and send it. After receiving the write-ahead log data, the slave database first persists it and then reads the log content from the disk file for replay. This results in increased synchronization latency, deteriorated transaction response time, and limited throughput. Summary of the Invention

[0003] One objective of this invention is to improve the synchronization performance of database master-slave clusters and shorten data synchronization latency.

[0004] A further objective of this invention is to reduce the waiting overhead in the streaming replication path while ensuring data persistence and consistency.

[0005] Specifically, the present invention provides a data processing method for a database master-slave cluster, comprising: In response to the write-ahead log data being temporarily stored in the log buffer of the main database, a persistence process for the write-ahead log data is initiated. Before the persistence process is completed, in response to the write-ahead log data satisfying the streaming replication read condition in the log buffer, an initiation is made to directly read the write-ahead log data from the log buffer and stream replicate it to at least one slave database, wherein the execution period of the streaming replication at least partially overlaps with the execution period of the persistence process.

[0006] Optionally, the step of initiating persistence processing for the write-ahead log data includes: Determine that the asynchronous replication link associated with the master database is active; Based on the transmission limit supported by the asynchronous replication link, the write-ahead log data in the log buffer is encapsulated into continuous data transmission units; The data transmission unit is injected into the asynchronous replication link according to the timing sequence to perform the persistence process.

[0007] Optionally, before initiating the persistence processing of the write-ahead log data, the method further includes: Monitor the data filling status of the log buffer until it reaches a preset capacity threshold; All write-ahead log data located between the start position of the log buffer and the capacity threshold position is identified as the data block to be processed; The data block to be processed is used to perform persistent processing.

[0008] Optionally, the step of responding to the write-ahead log data satisfying the conditions for streaming copy reading in the log buffer includes: Monitor the first capacity of the write-ahead log data in the log buffer that is in a state of pending persistence; The first capacity is compared with a preset capacity threshold; If the first capacity reaches or exceeds the preset capacity threshold, it is determined that the write-ahead log data in the log buffer meets the streaming copy read condition.

[0009] Optionally, the step of initiating the direct reading of the write-ahead log data from the log buffer and streaming replication to at least one slave database includes: Within the log buffer, the memory region containing the write-ahead log data that satisfies the streaming copy read conditions is locked in a shared read manner; Based on the lock, the write-ahead log data is read from the memory region using a zero-copy method to form a continuous data stream, and the data stream is pushed to the at least one slave database.

[0010] Optionally, the step of reading the write-ahead log data from the memory region using a zero-copy method includes: The memory region is mapped to a directly accessible linear address space; The memory descriptor is determined based on the linear address space; Based on the memory descriptor, the write-ahead log data is read via direct memory access.

[0011] Optionally, after pushing the data stream to the at least one slave database, the method further includes: Monitor the second capacity of the write-ahead log data marked as pushed in the memory region; If the second capacity reaches the preset reclamation threshold and the corresponding write-ahead log data has been persisted, the memory space of the second capacity is released and the write-ahead log data is reallocated.

[0012] According to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data processing methods for a database master-slave cluster.

[0013] According to another aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a processor, implements the steps of any of the above-described data processing methods for a database master-slave cluster.

[0014] According to another aspect of the present invention, a computer device is also provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any data processing method for a database master-slave cluster.

[0015] The data processing method for database master-slave clusters of the present invention advances the triggering time of the streaming replication sending operation from waiting for the write-ahead log data to complete the local disk write to when the write-ahead log data still resides in the log buffer and meets the streaming replication reading conditions. This allows the master database to start data transmission without waiting for the local disk I / O to complete. The log persistence processing and streaming replication operation are designed to overlap at least partially, eliminating the need to wait for persistence to complete before starting streaming replication. This significantly shortens the log transmission time from the master database to the slave database, effectively compresses the data synchronization latency of the master-slave cluster, and improves the real-time performance of streaming replication.

[0016] Furthermore, the data processing method for database master-slave clusters of the present invention confirms whether the asynchronous replication link is active before initiating data transmission, thereby avoiding transmission failure or data loss due to link failure; at the same time, according to the temporal order of the pre-written log data generated in the log buffer, the data transmission units are injected into the link in an orderly manner, strictly maintaining the temporal semantics of the log, ensuring that the slave database can receive and replay the logs in the correct order, fundamentally guaranteeing the consistency of the database master-slave cluster.

[0017] Furthermore, the data processing method for database master-slave clusters of the present invention locks the memory region that meets the reading conditions of streaming replication in the log buffer using a shared read method. This allows the streaming replication process to directly reference the memory address of that region to read data, eliminating the need to copy data from the memory buffer to the user buffer or to perform cross-memory data movement within the database process. This significantly reduces the resource consumption of write-ahead log data sending operations. Directly reading write-ahead log data from the memory region and organizing it into a continuous data stream logically forms sequential and compact sending units from potentially discretely stored memory pages, reducing the number of system calls for network sending operations and lowering protocol stack encapsulation overhead.

[0018] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0019] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings: Figure 1 This is a schematic flowchart of a data processing method for a database master-slave cluster according to an embodiment of the present invention; Figure 2 This is a schematic flowchart of the persistence processing in a database master-slave cluster data processing method according to an embodiment of the present invention; Figure 3 This is a schematic flowchart illustrating the preliminary operations of persistence processing in a database master-slave cluster data processing method according to an embodiment of the present invention; Figure 4 This is a schematic flowchart illustrating the determination of streaming replication read conditions in a database master-slave cluster data processing method according to an embodiment of the present invention; Figure 5 This is a schematic flowchart of the streaming replication operation in a database master-slave cluster data processing method according to an embodiment of the present invention; Figure 6 This is a schematic flowchart illustrating the zero-copy method for data reading in a database master-slave cluster data processing method according to an embodiment of the present invention; Figure 7 This is a schematic flowchart illustrating the process after streaming replication in a database master-slave cluster data processing method according to an embodiment of the present invention; Figure 8 This is a schematic diagram of a computer program product according to an embodiment of the present invention; Figure 9 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention; Figure 10 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0020] Database master-slave clusters ensure transaction durability and data consistency through a write-ahead logging (WAL) mechanism. According to this mechanism, the master database must write the corresponding write-ahead log data to its local disk before committing a transaction to ensure data recovery through the log in case of system failure. The slave database receives the write-ahead log data transmitted by the master database in real time via a streaming replication protocol, writes it to its local disk, and then applies the data changes to the slave database's data pages through log replay, thereby achieving eventual consistency between master and slave data.

[0021] In the above operations, the write-ahead log data must go through a complete serial link from generation in the primary database to the application in the secondary database: "Generate → Write to disk → Read → Transmit → Write to disk → Read → Playback". The primary database's log sending process (Walsender) must wait for the write-ahead log data to complete the local disk write before it can read the persisted log content from the disk file and encapsulate it into a network packet for transmission. After receiving the write-ahead log data, the secondary database's log receiving process (Walreceiver) first writes the write-ahead log data to the local disk file for persistence, and then reads the log content from the disk file through the log playback process (Startup) for playback.

[0022] This design forces each write-ahead log data synchronization to include two disk write operations (master database write to disk, slave database write to disk) and two disk read operations (master database read to send, slave database read to replay). Furthermore, all four disk I / O operations occur on the critical path between transaction commit and synchronization confirmation. With the significant improvement in memory access speed and network bandwidth in modern hardware environments, disk I / O has become a major bottleneck for streaming replication performance. In synchronous replication mode, the master database transaction commit requires waiting for the slave database to confirm the write-ahead log data has been written to disk. These redundant disk I / O operations further amplify synchronization latency, leading to a significant deterioration in transaction response time and severely limited system throughput.

[0023] Therefore, there is an urgent need to provide a solution that can effectively eliminate redundant disk I / O in the critical path of streaming replication without compromising data persistence and consistency.

[0024] The following describes, with reference to the accompanying drawings, a data synchronization method for a database master-slave cluster and related products thereof according to embodiments of the present invention.

[0025] Figure 1 This is a schematic flowchart of a data processing method for a database master-slave cluster according to an embodiment of the present invention, as shown below. Figure 1 As shown, the data processing method of this database master-slave cluster can generally include: Step S101: In response to the write-ahead log data being temporarily stored in the log buffer of the main database, initiate the persistence processing of the write-ahead log data. Step S102: Before the persistence process is completed, in response to the write-ahead log data satisfying the streaming replication read condition in the log buffer, the write-ahead log data is directly read from the log buffer and stream-replicated to at least one slave database, wherein the execution period of streaming replication at least partially overlaps with the execution period of persistence process.

[0026] In this embodiment, when a transaction generates write-ahead log data records during execution, the write-ahead log data is serialized and temporarily stored in the log buffer in the shared memory of the main database. The main database's logger writer process or background writer process continuously monitors the data filling status of the log buffer. Once it detects that there is write-ahead log data in the log buffer that has not yet been persisted, it triggers the persistence process. The log writer adds dirty log pages (i.e., memory pages in the log buffer that have been written with new write-ahead log data but have not yet been written to disk for persistent storage) to the persistence task queue and wakes up an independent persistence write thread. This persistence write thread retrieves the write-ahead log data from the persistence task queue asynchronously and writes the write-ahead log data to disk through interfaces such as pwrite or fsync provided by the operating system. The write operation does not involve network transmission, and the persistence operation is decoupled from the subsequent streaming replication operation.

[0027] In this embodiment, within the time window during which the initiated persistence process has not yet completed, the streaming replication and sending process continuously monitors the status of the write-ahead log data in the log buffer. When it is detected that the write-ahead log data meets the transmission ready conditions, the subsequent transmission process is triggered.

[0028] In this embodiment, the transmission readiness conditions include: the write-ahead log data constitutes one or more logically continuous and complete data blocks in memory; the memory page where the data block is located has been removed from the volatile state and has the ability to maintain its content unchanged when accessed concurrently; and a positive difference is formed between the range of the write-ahead log data sequence corresponding to the data block and the current transmission progress.

[0029] In this embodiment, if the transmission readiness condition is met, the streaming replication sending process requests concurrent access authorization for the memory region from the log buffer. This concurrent access authorization is granted in shared access mode, allowing multiple processes to read the memory region simultaneously without mutual exclusion. Based on the obtained authorization, the sending process obtains a logical access identifier pointing to the memory region. This logical access identifier encapsulates metadata such as the starting address, length, and access permissions of the memory region, and can serve as a unified handle for subsequent data read operations.

[0030] In this embodiment, the streaming replication sending process extracts the original content of the write-ahead log data from the memory region through a direct memory access path based on a logical access identifier. This access path bypasses the existing log buffer replication process, avoiding multiple data transfers between user space and kernel space. The extracted write-ahead log data is sent to the protocol adaptation layer, which performs format conversion, boundary delimitation, and verification information appending on the original data according to the data synchronization protocol specifications agreed upon between the master and slave clusters, encapsulating it into semantically complete data packets that can be independently transmitted and parsed.

[0031] In this embodiment, the encapsulated data packets are delivered to the transmission management layer, which maintains pre-established, long-lived logical communication channels with each slave database. Based on the current channel load and a preset priority strategy, the transmission management layer injects the data packets into the communication channels in an orderly manner and pushes them to the log receiver of the slave database.

[0032] In this embodiment, the execution period of streaming replication and the execution period of persistence processing overlap at least partially. This overlap can manifest in various ways, including but not limited to: persistent writing and streaming replication sending alternately occupying processor resources at different execution stages; persistent writing operations and streaming replication sending operations occupying different I / O channels and not blocking each other; and in the same batch of write-ahead log data, streaming replication sending of later data and persistent writing of earlier data occur simultaneously.

[0033] Based on the above steps, by advancing the triggering time of the streaming replication sending operation from after waiting for the write-ahead log data to complete the local disk write to when the write-ahead log data still resides in the log buffer and meets the streaming replication read conditions, the master database can start data transmission without waiting for the local disk I / O to complete. The log persistence processing and streaming replication operation are designed to overlap at least partially, without waiting for persistence to complete before starting streaming replication. This significantly shortens the log transmission time from the master database to the slave database, effectively compresses the data synchronization latency of the master-slave cluster, and improves the real-time performance of streaming replication.

[0034] Figure 2 This is a schematic flowchart illustrating the persistence processing in a database master-slave cluster data processing method according to an embodiment of the present invention, as follows: Figure 2 As shown, the steps to initiate persistent processing of write-ahead log data include: Step S201: Determine that the asynchronous replication link associated with the primary database is active; Step S202: Based on the transmission limit supported by the asynchronous replication link, the write-ahead log data in the log buffer is encapsulated into continuous data transmission units; Step S203: Inject the data transmission unit into the asynchronous replication link according to the timing sequence to perform persistence processing.

[0035] In this embodiment, before initiating the persistence processing of write-ahead log data, the streaming replication sending process of the master database first performs link availability verification. This availability verification process includes: detecting the status of the pre-established streaming replication session with the slave database to confirm that the streaming replication session has not been interrupted or timed out; checking the health of the underlying transport layer connection and obtaining bidirectional reachability confirmation of the link through protocol layer heartbeats or TCP keep-alive mechanisms; and verifying that the replication slot is active and capable of receiving incremental write-ahead log data. Only when all of the above conditions are met simultaneously does the streaming replication sending process determine that the current asynchronous replication link has the capability to carry data transmission and mark the link as active and available.

[0036] In this embodiment, in response to the asynchronous replication link being confirmed as active, the streaming replication sending process obtains the transmission limit supported by the asynchronous replication link. The transmission limit includes at least one of the following or a combination thereof: the maximum single payload length limit defined by the maximum transmission unit (MTU) of the asynchronous replication link, the maximum effective data payload specified by the streaming replication protocol message format, or the transmit buffer capacity threshold configured by the operating system network stack.

[0037] In this embodiment, the streaming replication sending process, constrained by the transmission limit, logically slices the write-ahead log data in the log buffer that meets the streaming replication read conditions according to their storage order in memory, ensuring that the data volume of each slice does not exceed the single-pass capacity allowed by the transmission limit. Each slice is independently encapsulated into a data transmission unit conforming to the streaming replication protocol specification. This data transmission unit includes at least: a sequence number field identifying the start position of the write-ahead log data, a field identifying the length of the write-ahead log data, the original write-ahead log data payload, and a checksum field for integrity verification. The encapsulated data transmission units are logically connected end-to-end, forming a continuous, non-overlapping, and gapless data sequence.

[0038] In this embodiment, the streaming replication sending process submits data transmission units sequentially to the sending queue of the asynchronous replication link according to the time order in which the write-ahead log data is generated in the log buffer. This sequential submission operation ensures that the log stream received from the database is strictly consistent with the transaction order actually executed by the master database. After the data transmission units are injected into the asynchronous replication link, the underlying transport protocol stack is responsible for delivering the data transmission units to the slave database. This delivery process is executed in parallel on the timeline with the local disk write operations of the same batch of write-ahead log data on the master database side.

[0039] Based on the above steps, the data processing method for database master-slave clusters of the present invention confirms whether the asynchronous replication link is active before initiating data transmission, thereby avoiding transmission failure or data loss due to link failure; at the same time, according to the temporal order of the pre-written log data generated in the log buffer, the data transmission units are injected into the link in an orderly manner, strictly maintaining the temporal semantics of the log, ensuring that the slave database can receive and replay the logs in the correct order, fundamentally guaranteeing the consistency of the database master-slave cluster.

[0040] Figure 3 This is a schematic flowchart illustrating the preliminary operations of persistence processing in a database master-slave cluster data processing method according to an embodiment of the present invention, such as... Figure 3 As shown, before initiating the persistence processing of write-ahead log data, the following steps are also included: Step S301: Monitor the data filling status of the log buffer to ensure it reaches a preset capacity threshold; Step S302: All write-ahead log data located between the start position of the log buffer and the capacity threshold position is identified as data blocks to be processed, wherein the data blocks to be processed are used to perform persistence processing.

[0041] Based on the above steps, a dynamic feedback mechanism between buffer pressure and persistence frequency is established by continuously monitoring the data filling status of the log buffer and triggering persistence processing when it reaches a preset capacity threshold. This allows persistence operations to adaptively match the transaction write load. Instead of processing pre-written log records piecemeal, the entire contiguous memory region from the start of the log buffer to the capacity threshold is treated as a single data block, significantly reducing disk seek time and I / O interrupts.

[0042] Figure 4 This is a schematic flowchart illustrating the determination of streaming replication read conditions in a database master-slave cluster data processing method according to an embodiment of the present invention, as follows: Figure 4 As shown, the steps in response to the condition that the write-ahead log data in the log buffer satisfies the requirement for streaming copy reading include: Step S401: Monitor the first capacity of the write-ahead log data in the log buffer that is in a state of pending persistence. Step S402: Compare the first capacity with a preset capacity threshold; Step S403: If the first capacity reaches or exceeds the preset capacity threshold, it is determined that the write-ahead log data in the log buffer meets the streaming copy reading conditions.

[0043] Based on the above steps, by monitoring the first capacity of the write-ahead log data in the pending persistence processing state and comparing it with a preset capacity threshold, a trigger determination mechanism independent of the persistence completion state is constructed for the streaming replication sending operation. This trigger determination mechanism does not need to wait for the persistence operation to complete, nor does it need to be aware of the disk I / O progress, thus achieving decoupling of write-ahead log data sending and write-ahead log data disk writing in the control logic.

[0044] Figure 5 This is a schematic flowchart of the streaming replication operation in a database master-slave cluster data processing method according to an embodiment of the present invention, as shown below. Figure 5 As shown, the steps to initiate direct reading of write-ahead log data from the log buffer and stream replication to at least one slave database include: Step S501: In the log buffer, lock the memory area where the write-ahead log data that meets the streaming copy read conditions is located in the shared read mode; Step S502: Based on locking, zero-copy method is used to read write-ahead log data from the memory area to form a continuous data stream, and the data stream is pushed to at least one slave database.

[0045] In this embodiment, in response to the write-ahead log data being determined to meet the streaming replication read conditions, the streaming replication sending process initiates a concurrent access authorization request to the log buffer, requesting read permission for the memory region where the write-ahead log data resides. After the concurrent access authorization request is verified as valid, a shared read lock flag is set for the memory region. This lock allows any number of streaming replication sending processes to read the memory region simultaneously, but is mutually exclusive with an exclusive write lock currently performing log buffer page content reclamation, overwriting, or persistence confirmation and subsequent clearing operations. If the memory region is currently held by an exclusive write lock, the streaming replication sending process either waits or bypasses to the next available page, thereby avoiding read-write conflicts. The holding scope of the shared read lock is strictly limited to the entire read operation process, and it is released immediately after the read is completed to avoid log buffer deadlock or decreased concurrency caused by long-term locking.

[0046] In this embodiment, the streaming replication sending process reads the write-ahead log data through a direct memory access mechanism, that is, it directly maps or references the write-ahead log data in the memory region to the network sending buffer, avoiding multiple data transfers between user space and kernel space. The read write-ahead log data is then continuously assembled into a sequence of messages conforming to the streaming replication protocol specification, according to its physical storage order in the log buffer. Each message includes a start log sequence number, data payload length, original log content, and a checksum field. The messages are linked end-to-end to form a logically continuous and semantically complete data stream.

[0047] Based on the above steps, by locking the memory region that meets the read conditions for streaming replication in the log buffer using shared read mode, the streaming replication process can directly reference the memory address of this region to read data. This eliminates the need to copy data from the memory buffer to the user buffer or to perform cross-memory data movement within the database process, significantly reducing the resource consumption of write-ahead log sending operations. Directly reading write-ahead log data from the memory region and organizing it into a continuous data stream logically forms sequential and compact sending units from potentially discretely stored memory pages, reducing the number of system calls for network sending operations and lowering protocol stack encapsulation overhead.

[0048] Figure 6 This is a schematic flowchart illustrating data reading using a zero-copy method in a database master-slave cluster data processing method according to an embodiment of the present invention, as shown below. Figure 6 As shown, the steps for reading the write-ahead log data from the memory region using a zero-copy method include: Step S601: Map the memory region to a directly accessible linear address space; Step S602: Determine the memory descriptor based on the linear address space; Step S603: Based on the memory descriptor, read the write-ahead log data via direct memory access.

[0049] Based on the above steps, an end-to-end zero-copy data channel is established for the streaming copy sending process through the mechanism of "memory region mapping → memory descriptor determination → direct memory access reading". This zero-copy data channel enables the write-ahead log data to travel from the log buffer to the network interface without undergoing any cross-address space data copying. This eliminates the overhead of multiple memory copying between user mode and kernel mode, and between process private buffers and shared buffers in the prior art, and significantly reduces CPU utilization and memory bandwidth consumption.

[0050] Figure 7 This is a schematic flowchart illustrating the streaming replication operation in a database master-slave cluster data processing method according to an embodiment of the present invention, as shown below. Figure 7 As shown, after pushing the data stream to at least one database, it also includes: Step S701: Monitor the second capacity of write-ahead log data marked as pushed in the memory region; Step S702: If the second capacity reaches the preset reclamation threshold and the corresponding write-ahead log data has been persisted, release the memory space of the second capacity and reallocate the write-ahead log data.

[0051] Based on the above steps, by continuously monitoring the second capacity occupied by the write-ahead log data marked as "pushed" after the data stream is pushed, and triggering memory release operations according to a preset reclamation threshold, a complete closed loop of the buffer lifecycle is constructed. This mechanism ensures that write-ahead log data that no longer needs to be retained in the log buffer can be identified and reclaimed in a timely manner, avoiding long-term retention of memory pages due to inconsistencies between the progress of the sending process and the persistence process, and preventing the risk of buffer resource exhaustion and system instability.

[0052] The flowchart provided in this embodiment is not intended to indicate that the operations of the method will be performed in any particular order, or that all operations of the method are included in every case. Furthermore, the method may include additional operations. Within the scope of the technical concept provided by the method in this embodiment, additional variations can be made to the above method.

[0053] It should be understood that in some embodiments, the components may be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods may be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0054] This embodiment also provides a computer program product 10, a computer-readable storage medium 20, and a computer device 30. Figure 8 This is a schematic diagram of a computer program product 10 according to an embodiment of the present invention. Figure 9 This is a schematic diagram of a computer-readable storage medium 20 according to an embodiment of the present invention. Figure 10 This is a schematic diagram of a computer device 30 according to an embodiment of the present invention. The computer program product 10 includes a computer program 11, which, when executed by the processor 32, implements the steps of any of the above-described methods. A computer-readable storage medium 20 stores the computer program 11 thereon, which, when executed by the processor 32, implements the steps of the data processing method for a database master-slave cluster according to any of the above embodiments. The computer device 30 may include a memory 31, a processor 32, and the computer program 11 stored on the memory 31 and running on the processor 32.

[0055] The computer program 11 used to perform the operations of this invention may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​and procedural programming languages. The computer program 11 may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer. In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing status information of the computer-readable program instructions.

[0056] For the purposes of this embodiment, computer program product 10 is a related product that includes computer program 11.

[0057] For the purposes of this embodiment, computer-readable storage medium 20 is a tangible device capable of holding and storing a computer program 11. It can be any device capable of containing, storing, communicating, propagating, or transmitting the program 11 for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage medium 20 include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanical encoding device, and any suitable combination thereof.

[0058] Computer device 30 can be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer device 30 can be a cloud computing node. Computer device 30 can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer device 30 can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0059] Computer device 30 may include a processor 32 adapted to execute stored instructions and a memory 31 that provides temporary storage space for the operation of said instructions during operation. The processor 32 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. The memory 31 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0060] Computer device 30 may also include a network adapter / interface and an input / output (I / O) interface. The I / O interface allows external devices that can be connected to the computer device to input and output data. The network adapter / interface provides communication between the computer device and a network, typically represented as a communication network.

[0061] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A data processing method for a database master-slave cluster, characterized in that, include: In response to the write-ahead log data being temporarily stored in the log buffer of the main database, a persistence process for the write-ahead log data is initiated. Before the persistence process is completed, in response to the write-ahead log data satisfying the streaming replication read condition in the log buffer, an initiation is made to directly read the write-ahead log data from the log buffer and stream replicate it to at least one slave database, wherein the execution period of the streaming replication at least partially overlaps with the execution period of the persistence process.

2. The data processing method for a database master-slave cluster according to claim 1, characterized in that, The steps to initiate the persistence processing of the write-ahead log data include: Determine that the asynchronous replication link associated with the master database is active; Based on the transmission limit supported by the asynchronous replication link, the write-ahead log data in the log buffer is encapsulated into continuous data transmission units; The data transmission unit is injected into the asynchronous replication link according to the timing sequence to perform the persistence process.

3. The data processing method for a database master-slave cluster according to claim 1, characterized in that, Before initiating the persistence process for the write-ahead log data, the following steps are also included: Monitor the data filling status of the log buffer until it reaches a preset capacity threshold; All write-ahead log data located between the start position of the log buffer and the capacity threshold position is identified as the data block to be processed; The data block to be processed is used to perform persistent processing.

4. The data processing method for a database master-slave cluster according to claim 1, characterized in that, The steps for responding to the write-ahead log data satisfying the conditions for streaming copy reading in the log buffer include: Monitor the first capacity of the write-ahead log data in the log buffer that is in a state of pending persistence; The first capacity is compared with a preset capacity threshold; If the first capacity reaches or exceeds the preset capacity threshold, it is determined that the write-ahead log data in the log buffer meets the streaming copy read condition.

5. The data processing method for a database master-slave cluster according to claim 1, characterized in that, The steps of initiating direct reading of the write-ahead log data from the log buffer and streaming replication to at least one slave database include: Within the log buffer, the memory region containing the write-ahead log data that satisfies the streaming copy read conditions is locked in a shared read manner; Based on the lock, the write-ahead log data is read from the memory region using a zero-copy method to form a continuous data stream, and the data stream is pushed to the at least one slave database.

6. The data processing method for a database master-slave cluster according to claim 5, characterized in that, The steps for reading the write-ahead log data from the memory region using a zero-copy method include: The memory region is mapped to a directly accessible linear address space; The memory descriptor is determined based on the linear address space; Based on the memory descriptor, the write-ahead log data is read via direct memory access.

7. The data processing method for a database master-slave cluster according to claim 5, characterized in that, After pushing the data stream to the at least one slave database, the process further includes: Monitor the second capacity of the write-ahead log data marked as pushed in the memory region; If the second capacity reaches the preset reclamation threshold and the corresponding write-ahead log data has been persisted, the memory space of the second capacity is released and the write-ahead log data is reallocated.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method for a database master-slave cluster as described in any one of claims 1 to 7.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the data processing method for a database master-slave cluster as described in any one of claims 1 to 7.

10. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the data processing method for a database master-slave cluster as described in any one of claims 1 to 7.

Images