A database uniform waiting method, device and medium based on WAL flush events

Abstract

The embodiment of the application discloses a database unified waiting method and device based on a WAL flush event, and a medium, belongs to the technical field of databases, and solves the problems that the existing polling mechanism is prone to causing resource waste and unstable event response delay when waiting for a WAL flush event. The method comprises the following steps: establishing a WAL flush event center in a database kernel, and registering a global waiting item in the WAL flush event center by a sub-process; the WAL flush event center arranges the registered waiting items to generate a global waiting queue; when the WAL disk flushing is completed and a new log flush position is generated, the WAL flush event center wakes up the sub-process corresponding to the waiting item in the global waiting queue, wherein the target log sequence number of the waiting item is not greater than the latest log flush position; the awakened sub-process acquires the current log flush position, detects the waiting condition again based on the current log flush position, and performs business logic or reenters the waiting state based on the detection result.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a unified database waiting method, device and medium based on WAL refresh events. Background Technology

[0002] In modern database systems, write-ahead logging (WAL) is a core mechanism for ensuring data persistence and implementing master-slave replication. When a transaction is committed, the changes must first be written to the WAL and ensured to be safely persisted before a successful commit is returned to the client. This persisted log position is typically identified by a Log Sequence Number (LSN). Several critical subsystems within the database, such as streaming replication, logical replication, WAL archiving, and incremental backup processes, all rely on the confirmation that the required WAL logs have been flushed to the specified positions for normal operation. Therefore, these subprocesses generally have a need to wait for WAL flush events.

[0003] Currently, database systems generally adopt a polling wait mechanism. The process is as follows: the waiting process first obtains the current WAL refresh position, and then determines whether it has reached the target LSN; if it has not reached it, it actively sleeps, and then repeats the above steps until the condition is met.

[0004] However, a large number of waiting processes are idle most of the time, consuming valuable computing resources. Furthermore, even if the WAL (Write-Ahead Log) has been actually refreshed, the waiting processes may still be in a sleep cycle and cannot be immediately woken up to execute subsequent tasks. Therefore, the existing polling mechanism's waiting for WAL refresh events easily leads to wasted CPU resources and unstable event response delays. Summary of the Invention

[0005] This application provides a database unified waiting method, device, and medium based on WAL refresh events to solve the following technical problem: the existing polling mechanism for waiting for WAL refresh events easily leads to wasted CPU resources and unstable event response delays.

[0006] The embodiments of this application adopt the following technical solutions: This application provides a unified database waiting method based on WAL refresh events. It includes: establishing a WAL refresh event center in the database kernel; multiple child processes within the database that need to wait for WAL refresh to the target log sequence number registering global wait items with the WAL refresh event center; wherein each global wait item includes at least one of the following: target log sequence number, child process identifier, and wake-up handle; the WAL refresh event center arranges all registered wait items in ascending order of target log sequence number to generate a global wait queue; when WAL flushing is completed and a new log refresh position is generated, the WAL refresh event center sequentially traverses the global wait queue from the head of the queue according to the latest log refresh position, waking up only the child processes corresponding to wait items in the global wait queue whose target log sequence number is not greater than the latest log refresh position; the awakened child process obtains the current log refresh position, re-checks the waiting conditions based on the current log refresh position, and executes business logic or re-enters the waiting state based on the check result.

[0007] In one implementation of this application, generating a global wait queue specifically includes: constructing a global wait queue using a doubly linked list as the underlying data structure; wherein each node in the doubly linked list corresponds to a global wait item, and the node data is set with pointers to the predecessor and successor nodes; when a child process initiates a registration request, the WAL refreshes the event center to acquire a lightweight lock acting on the doubly linked list, and starts traversing from the head of the doubly linked list to determine the first node whose target log sequence number is greater than the target log sequence number of the new wait item; inserting the new node before the node, and releasing the lightweight lock.

[0008] In one implementation of this application, the WAL refresh event center sequentially traverses the global waiting queue from the head based on the latest log refresh position. It only wakes up the child processes corresponding to waiting items in the global waiting queue whose target log sequence number is not greater than the latest log refresh position. Specifically, this includes: after each WAL flush, the WAL refresh event center reads and saves the latest log refresh position as the wake-up baseline; starting from the head of the global waiting queue, it continuously wakes up the child processes corresponding to waiting items whose target log sequence number is not greater than the baseline value; when a waiting item with a target log sequence number greater than the baseline value is detected, it stops the current traversal and records the breakpoint position of the current queue; when the next WAL flush event is triggered, a new wake-up traversal operation begins at the breakpoint position.

[0009] In one implementation of this application, after waking up the child process corresponding to the waiting item in the global waiting queue whose target log sequence number is not greater than the latest log refresh position, the method further includes: the WAL refresh event center determining the difference between the target log sequence number of the head waiting item and the latest log refresh position, and determining that the current wake-up event has a long-term wait when the duration of the difference is greater than a preset duration threshold; after the next WAL flush event is triggered, the WAL refresh event center selectively wakes up the child process corresponding to the head waiting item to implement timeout processing; wherein, the latest log refresh position when timeout processing is started has not yet reached the target log sequence number of the head waiting item.

[0010] In one implementation of this application, after waking up the child process corresponding to the waiting item in the global wait queue whose target log sequence number is not greater than the latest log refresh position, the method further includes: when the child process has multiple target log sequence numbers to wait for, creating local waiting items for each target log sequence number in its process local address space; wherein, a bidirectional association pointer is set between each local waiting item and the corresponding global waiting item in the global wait queue; the awakened database child process determines the subset of local waiting items to be checked based on the internally set baseline log sequence number; wherein, the baseline log sequence number is the largest log sequence number that the child process has confirmed has been processed, and the subset of local waiting items includes all local waiting items whose target log sequence number is greater than the baseline log sequence number. Waiting items; The awakened database subprocess traverses the local waiting item subset and checks it based on the locally set baseline log sequence number and the latest log refresh position. If there is a waiting item in the local waiting item subset with a target log sequence number not greater than the latest log refresh position, the awakened database subprocess sends a notification to the WAL refresh event center to deregister the corresponding global waiting item in the global waiting queue. After processing the business corresponding to the waiting item, the baseline log sequence number is updated to the target log sequence number of the waiting item. If there is a waiting item in the local waiting item subset with a target log sequence number greater than the latest log refresh position, the corresponding global waiting item is retained in the global waiting queue, and the awakened subprocess re-enters the waiting state.

[0011] In one implementation of this application, after detecting the subset of local waiting items, the method further includes: marking local waiting items whose target log sequence number is not greater than the current latest log refresh position as items to be merged; merging items with consecutively adjacent target log sequence numbers in the items to be merged to construct a continuous sequence number interval, and recording the continuous sequence number interval in the list of intervals to be processed; for each continuous sequence number interval in the list of intervals to be processed, the subprocess sends a batch deregistration request to the WAL refresh event center; wherein, the batch deregistration request includes the identifiers of the global waiting items associated with each item to be merged in the continuous sequence number interval; the WAL refresh event center removes the corresponding multiple global waiting items from the global waiting queue according to the identifiers; after the batch business operation corresponding to the continuous sequence number interval is completed, the subprocess updates the base log sequence number to the end log sequence number of the continuous sequence number interval.

[0012] In one implementation of this application, after creating local wait items for each target log sequence number, the method further includes: a subprocess registering pre-computation conditions associated with the target log sequence number with the WAL refresh event center; when the WAL refresh event center detects the generation of a new WAL record, if it determines that the log sequence number corresponding to the record is less than the target log sequence number of any registered wait item, and the record content meets the pre-computation conditions registered by the wait item, it sends a pre-computation trigger signal to the corresponding subprocess; based on the location information in the pre-computation trigger signal, the subprocess asynchronously reads the corresponding WAL record and performs pre-computation operations related to the final business logic, and caches the pre-computation results; when a WAL flush event occurs, and the wait item is formally awakened and confirmed to meet the pre-computation conditions, the subprocess uses the cached pre-computation results to execute the business logic to reduce computational overhead.

[0013] In one implementation of this application, the awakened child process obtains the current log refresh position, re-checks the waiting conditions based on the current log refresh position, and executes business logic or re-enters the waiting state based on the check result. Specifically, this includes: after the child process is awakened by the WAL refresh event center, it obtains the latest log refresh position from the database kernel; the child process compares the latest log refresh position with its corresponding target log sequence number; if the latest log refresh position is not less than the target log sequence number, the waiting condition is determined to be met, and a success signal is sent to the WAL refresh event center to cancel the corresponding waiting item; if the latest log refresh position is less than the target log sequence number, the waiting condition is determined to be unmet, and a re-entry signal is sent to the WAL refresh event center to re-add its corresponding waiting item to the waiting queue, causing the child process to re-enter the waiting state and wait for the next wake-up.

[0014] This application provides a unified database waiting device based on WAL refresh events, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: register global wait items with the WAL refresh event center for multiple sub-processes within the database that need to wait for WAL refresh to the target log sequence number; wherein the global wait item includes at least one of the following: target log sequence number, sub-process identifier, and wake-up handle; the WAL refresh event center arranges all registered wait items in ascending order of target log sequence number to generate a global wait queue; when WAL flushing is completed and a new log refresh position is generated, the WAL refresh event center sequentially traverses the global wait queue from the head of the queue according to the latest log refresh position, and only wakes up the sub-processes corresponding to the wait items in the global wait queue whose target log sequence number is not greater than the latest log refresh position; the woken-up sub-processes obtain the current log refresh position, re-detect the waiting conditions based on the current log refresh position, and execute business logic or re-enter the waiting state based on the detection result.

[0015] This application provides a non-volatile computer storage medium storing computer-executable instructions. These instructions are configured as follows: multiple sub-processes within the database that need to wait for the WAL (Write-Ahead Log) update to the target log sequence number register global wait items with the WAL update event center. Each global wait item includes at least one of the following: target log sequence number, sub-process identifier, and wake-up handle. The WAL update event center arranges all registered wait items in ascending order of the target log sequence number, generating a global wait queue. When the WAL update is completed and a new log update position is generated, the WAL update event center sequentially traverses the global wait queue from the head of the queue based on the latest log update position, waking up only the sub-processes corresponding to wait items in the global wait queue whose target log sequence number is not greater than the latest log update position. The awakened sub-process obtains the current log update position, re-checks the waiting conditions based on the current log update position, and executes business logic or re-enters the waiting state based on the check results.

[0016] The above-mentioned technical solutions adopted in this application embodiment can achieve the following beneficial effects: Firstly, this application embodiment registers waiting items with the WAL refresh event center through subprocesses, eliminating the drawbacks of traditional solutions where each module polls independently, thus achieving architectural uniformity and reducing code redundancy and system complexity. Secondly, this application embodiment generates a global queue in ascending order of the target log sequence number, improving processing efficiency. The WAL refresh event center only wakes up waiting items whose target LSN is not greater than that LSN after disk flushing, starting from the head of the queue based on the latest LSN. This avoids a large number of invalid wake-ups and checks in traditional polling, transforming CPU consumption from active periodic idle time to event-driven precise response, saving computing resources, and making the delay from WAL disk flushing completion to the start of subsequent tasks extremely small and stable. Finally, the step of the woken subprocess re-acquiring and verifying the LSN can handle boundary cases such as concurrent wake-ups and spurious wake-ups, ensuring that business logic is only executed when the conditions are truly met, thereby improving performance while guaranteeing the absolute correctness of the system. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart of a database unified waiting method based on WAL refresh events is provided for embodiments of this application; Figure 2 A flowchart illustrating the waiting process for a WAL refresh event, provided as an embodiment of this application; Figure 3 A flowchart illustrating the waiting process for a database kernel WAL refresh event is provided in this embodiment of the application. Figure 4 This is a schematic diagram of the structure of a unified database waiting device based on WAL refresh events, provided in an embodiment of this application.

[0018] Figure label: 200: Database unified waiting device based on WAL refresh event; 201: Processor; 202: Memory. Detailed Implementation

[0019] This application provides a database unified waiting method, device, and medium based on WAL refresh events.

[0020] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0021] Figure 1 A flowchart of a database unified waiting method based on WAL refresh events provided in this application embodiment is shown below. Figure 1 As shown, the unified database waiting method based on WAL refresh events includes the following steps: Step 101: Multiple child processes within the database that need to wait for the WAL to be refreshed to the target log sequence number register global wait items with the WAL refresh event center.

[0022] In one implementation of this application, when any child process within the database, such as a streaming replication log sending process, a logical replication worker process, a WAL archiving process, or an incremental backup process, reaches a point where its business logic requires waiting for the WAL to be refreshed to a specific target log sequence number, the process enters a wait registration process. The process first dynamically allocates a memory structure in its operating system process context or thread-local storage to hold the core information for this wait; this memory structure is a wait item. The wait item contains at least the following data: the target LSN (the WAL refresh position the process needs to wait for); a process identifier pointing to the control block of the waiting process for precise wake-up; a wait event handle; a module type indicating the subsystem it belongs to; a wait status, including waiting, awakened, and unregistered; and predecessor / successor pointers for forming a doubly linked list queue.

[0023] Step 102: The WAL refreshes the event center, arranging all registered wait items in ascending order of the target log sequence number, and generating a global wait queue.

[0024] In one implementation of this application, a doubly linked list is used as the underlying data structure to construct a global wait queue. Each node in the doubly linked list corresponds to a global wait item, and the node data is configured with pointers to the predecessor and successor nodes. When a child process initiates a registration request, the WAL refreshes the event center, acquires a lightweight lock on the doubly linked list, and begins traversing the list from the head to determine the first node whose target log sequence number is greater than the target log sequence number of the new wait item. The new node is then inserted before the node, and the lightweight lock is released.

[0025] Specifically, the WAL refresh event center first defines a linked list head structure in the database kernel as the anchor point of the entire linked list, containing two pointer fields, pointing to the first and last valid nodes of the linked list, respectively. During initialization, both pointers are set to null, indicating that the linked list is empty. Simultaneously, the node structure of the linked list is defined. Each node represents a global wait item, and its structure members include at least: a data field, used to store or point to a specific wait item information block, which at least includes the target log sequence number, the identifier of the child process that registered this wait item, and a handle used to wake up the child process; a pointer to the predecessor node, used to point to the previous node; if this node is the first node, this pointer points to the head of the linked list or is null; and a pointer to the successor node, used to point to the next node; if this node is the tail node, this pointer points to the head of the linked list or is null.

[0026] Furthermore, after receiving a registration request from a child process, the WAL refresh event center extracts the target log sequence number from the registration parameters. Then, the event center accesses its internal doubly linked list, sorted in ascending order of target log sequence number—the global wait queue. Before performing an insertion operation, it acquires a lightweight lock on this queue. While holding the lock, the WAL refresh event center traverses the list from the head, comparing the target log sequence number of the new wait item with the target log sequence numbers of existing nodes in the list until it finds the first node in the list whose target log sequence number is greater than the new wait item's target log sequence number. Next, the event center creates a new linked list node, stores the wait item information passed from the child process into this node, and inserts this new node before the previously found node. If, after traversing to the end of the list, no node with a target value greater than the new node is found, then the new node should be inserted at the end of the list.

[0027] Furthermore, a predecessor pointer of the new node is set to point to the node preceding the target position, and a successor pointer of the new node is set to point to the current node at the target position. The successor pointer of the node preceding the target position is then modified to point to the new node, and the predecessor pointer of the current node at the target position is modified to point to the new node. If the insertion is at the head or tail of the linked list, only the head pointer and the corresponding pointers of adjacent nodes need to be adjusted. This pointer adjustment process ensures that the new node is seamlessly integrated into the linked list, and the doubly linked relationship of the entire linked list remains intact after insertion.

[0028] Once the pointer adjustment is complete and the new node has been successfully added to the linked list, the WAL refresh event center immediately releases the lightweight lock.

[0029] This insertion operation ensures that, regardless of the number of processes or the order in which they register, the global wait queue always maintains its nodes linked in ascending order of the target log sequence number, so that the node at the head of the queue is always the one with the smallest target log sequence number among all the wait items.

[0030] Step 103: When the WAL disk flushing is completed and a new log refresh position is generated, the WAL refresh event center sequentially traverses the global waiting queue from the head of the queue according to the latest log refresh position, and only wakes up the child processes corresponding to the waiting items in the global waiting queue whose target log sequence number is not greater than the latest log refresh position.

[0031] In one implementation of this application, after each WAL flush, the WAL refresh event center reads and saves the latest log refresh position as the baseline value for this wake-up. Starting from the head of the global wait queue, the child processes corresponding to wait items whose target log sequence number is not greater than the baseline value are continuously woken up. When a wait item with a target log sequence number greater than the baseline value is detected, the current traversal stops, and the breakpoint position of the current queue is recorded. When the next WAL flush event is triggered, a new wake-up traversal operation begins at the breakpoint position to avoid repeatedly judging wait items that have already been determined not to meet the conditions.

[0032] Specifically, after the database kernel's WAL write module completes a log flush operation, it obtains the latest log refresh position. The WAL refresh event center uses this value as the decision-making basis for this wake-up operation. The WAL refresh event center internally sets up a traversal breakpoint pointer, which points to the head of the queue during initialization. At the start of each wake-up process, the event center first checks this breakpoint pointer. If the pointer is valid and points to a node in the queue, the traversal starts from the node pointed to by the pointer. If the pointer is invalid or the queue is empty, the traversal starts from the actual head node of the queue.

[0033] For each node visited, the event center compares the target log sequence number stored in its data field with the wake-up benchmark. If the target log sequence number of the node is less than or equal to the current wake-up benchmark, the WAL log required by the waiting item has been securely persisted, and its waiting condition has been met. At this point, the event center immediately executes the wake-up operation. Using the child process identifier and wake-up handle stored in the node's data field, a wake-up signal is sent to the corresponding database child process. This wake-up operation transitions the child process from a dormant state to a runnable state, ready to re-examine and execute business logic. After sending the wake-up signal, the event center does not immediately remove the node but continues to visit the next node in the linked list, repeating the comparison and wake-up process. During sequential traversal, once a waiting item node is visited whose target log sequence number is greater than the current wake-up benchmark, the current traversal is immediately stopped. Simultaneously, the WAL refresh event center updates the traversal breakpoint pointer, making it point to the first node that does not meet the condition. When the next WAL flush event occurs, the new wake-up traversal can start directly from the record breakpoint, thus avoiding repeated and invalid traversal judgments on the first half of the queue that is known to not meet the conditions.

[0034] In one implementation of this application, after waking up the child process corresponding to the waiting item in the global waiting queue whose target log sequence number is not greater than the latest log refresh position, the WAL refresh event center determines the difference between the target log sequence number of the head waiting item and the latest log refresh position. If the duration of the difference exceeds a preset duration threshold, it is determined that the current wake-up event has a long wait. After the next WAL flush event is triggered, the WAL refresh event center selectively wakes up the child process corresponding to the head waiting item to implement timeout handling; wherein, the latest log refresh position when timeout handling is initiated has not yet reached the target log sequence number of the head waiting item.

[0035] Specifically, after each WAL refresh event center completes a round of precise wake-up operations based on the latest log refresh position, it checks the status of the global waiting queue. If the queue is not empty, it retrieves the waiting item node at the head of the queue and reads its target log sequence number. Simultaneously, it retrieves the latest log refresh position that triggered this wake-up process again and calculates the difference between these two values. This difference represents the longest-waiting item in the current queue, whose target log sequence number exceeds the confirmed safe persistent log position. Its magnitude reflects the lag between the WAL flushing progress and the highest waiting demand. To distinguish between normal short-term lag and abnormal long-term blocking, this embodiment sets a timer in the WAL refresh event center. Whenever the calculated difference is greater than zero, meaning the head item is not satisfied, the event center uses this timer to check how long this difference state has persisted. If the duration exceeds the preset time threshold, the current head of the queue will be determined to be in a long-term waiting state. At this time, an unexpected anomaly may have occurred, such as an extremely slow receiving end of a backup database, a partially disconnected network connection, or a temporary unavailability of the archive storage system, which will prevent the business logic of the corresponding subprocess from proceeding, and thus its registered waiting item will remain at the head of the queue for a long time.

[0036] When a subsequent WAL flush event is triggered again, before the WAL flush event center begins a new round of standard wake-up traversal, it first checks the status of its internal records. If there are any wait items marked for timeout probing, and these wait items are still at the head of the current global wait queue, the event center obtains the latest log refresh position generated after this flush and compares it with the target log sequence number of the marked item still at the head of the queue. At this point, there is a very high probability that the latest log refresh position has not yet reached its target log sequence number. This comparison step confirms that the long-term wait state still exists. After confirmation, the event center sends a wake-up signal to the child process corresponding to the marked head-of-the-queue wait item, allowing the process to resume execution from long-term blocking and execute the timeout handling routine.

[0037] In one implementation of this application, when a subprocess has multiple target log sequence numbers to wait for, a local wait entry is created for each target log sequence number in its process local address space. A bidirectional association pointer is set between each local wait entry and the corresponding global wait entry in the global wait queue. The awakened database subprocess determines a subset of local wait entries to be checked based on an internally set baseline log sequence number. The baseline log sequence number is the largest log sequence number that the subprocess has confirmed has been processed. The local wait entry subset includes all local wait entries whose target log sequence number is greater than the baseline log sequence number. The awakened database subprocess traverses the local wait entry subset and checks it based on the locally set baseline log sequence number and the latest log refresh position. If there is a wait entry in the local wait entry subset whose target log sequence number is not greater than the latest log refresh position, the awakened database subprocess sends a notification to the WAL refresh event center to deregister the corresponding global wait entry in the global wait queue. After processing the business corresponding to the wait entry, the baseline log sequence number is updated to the target log sequence number of the wait entry. If there is a waiting item in the local waiting item subset whose target log sequence number is greater than the latest log refresh position, then the corresponding global waiting item is kept in the global waiting queue, and the awakened child process re-enters the waiting state.

[0038] Specifically, when a database subprocess, such as a streaming replication process, needs to send logs at different progress levels to multiple backup databases simultaneously, and its logic determines that it needs to wait for multiple target log sequence numbers, it creates a private data structure within its own process address space. This structure is typically an ordered linked list or a hash table with the target log sequence number as the key, to manage all its local wait requests. For each specific target log sequence number that needs to be waited for, the process dynamically allocates a new node in its private structure as a local wait item node. This local wait item node records the target log sequence number itself, the pointer field bound to the global queue, and other information. Subsequently, the process calls a unified registration interface, submitting information including this target log sequence number to the WAL refresh event center. After creating the corresponding global wait item node in the global ordered wait queue, the WAL refresh event center returns the memory address of the global node to the process. Upon receiving this address, the process fills it into the associated pointer field of the local wait item node. Simultaneously, a back pointer is also set within the global wait item node, pointing to the process that created it and its corresponding local wait item node.

[0039] When a process is awakened by the WAL refresh event center, it obtains the latest log refresh position and compares all locally maintained wait items with the baseline log sequence number. It then identifies wait items whose target log sequence number is greater than the current baseline log sequence number, forming a subset of local wait items to be checked. This process ensures that the check scope is incremental; as the baseline value advances, the number of items to be checked dynamically decreases, avoiding the overhead of a full traversal that might occur as the process runtime increases. Since items within the subset are usually organized in order of their target log sequence number, the traversal starts with the item with the smallest target value. For each local wait item in the subset, the latest log refresh position obtained from the database kernel is compared numerically with the target log sequence number recorded for that wait item. If the latest log refresh position is greater than or equal to the target log sequence number, the condition for this local wait item is considered met. Conversely, if the latest log refresh position is still less than the target log sequence number, the condition for this local wait item is considered not yet met.

[0040] Furthermore, for all local wait items deemed to meet the conditions, the process uses the bidirectional association pointer recorded in the local wait item to find its corresponding global node identifier in the global queue, and sends a request to the event center to cancel the specific global wait item. Simultaneously or immediately after sending the cancellation notification, the process begins executing the actual business logic corresponding to the wait item, and updates its core state after successful execution. The local baseline log sequence number is modified to the value of the target log sequence number just processed. If multiple consecutive items simultaneously meet the conditions, the baseline log sequence number is updated to the largest target value among them.

[0041] If a process encounters a local wait item that is determined to be unmet, no business processing action is taken on it. The global wait items associated with these wait items remain in the global queue of the WAL refresh event center, maintaining a waiting state.

[0042] In one implementation of this application, after detecting the subset of local waiting items, the process further includes: marking local waiting items whose target log sequence number is not greater than the current latest log refresh position as items to be merged. Items with consecutively adjacent target log sequence numbers in the items to be merged are merged to construct a continuous sequence number interval, and this continuous sequence number interval is recorded in the pending interval list. For each continuous sequence number interval in the pending interval list, the subprocess sends a batch deregistration request to the WAL refresh event center; the batch deregistration request includes the identifiers of the global waiting items associated with each item to be merged within the continuous sequence number interval. Based on the identifiers, the WAL refresh event center removes the corresponding multiple global waiting items from the global waiting queue. After the batch business operations corresponding to the continuous sequence number interval are completed, the subprocess updates the base log sequence number to the end log sequence number of that continuous sequence number interval.

[0043] Specifically, after the child process is awakened and obtains the latest log refresh position, it iterates through the subset of local waiting items whose target log sequence number is greater than the base log sequence number. During the traversal, when it detects that the target log sequence number of a local waiting item is not greater than the latest log refresh position, the waiting item is marked as a pending item. The process continues to traverse subsequent items in the subset, repeating this marking process for all items that meet the condition to build a waiting item list. Scanning begins from the first item in the list, using its target log sequence number as both the start and end value of the current interval. If the target log sequence number of the next item is exactly equal to the end log sequence number of the current interval plus a unit increment (i.e., they are consecutively adjacent), then the end value of the current interval is updated to this larger target log sequence number. This process is repeated, continuously attempting to merge the next consecutive item into the current interval. When the scan encounters a target log sequence number that is not consecutive with the end value of the previous interval, it is determined that the construction of the current consecutive interval is complete. The currently completed interval is recorded as a consecutive log sequence number processing interval defined by the start and end log sequence numbers, and uploaded to the pending interval list. Then, using the current discontinuous item as a new starting point, the next continuous interval is constructed until all items to be merged have been scanned. Ultimately, the list of intervals to be processed contains several non-overlapping continuous log sequence number intervals, each interval representing a logically continuous segment of WAL logs that can be processed in one go.

[0044] Furthermore, the process iterates through each target log sequence number within the interval and, using the established bidirectional association pointers, finds the local waiting item node corresponding to each target log sequence number, thereby obtaining the unique identifier of the global waiting item associated with that local node. These identifiers of all global waiting items belonging to the same interval are grouped together to form a batch deregistration request, which is then sent to the WAL refresh event center. For each identifier in the batch deregistration request, the event center locates the corresponding waiting item node in its global data structure. Since these nodes were inserted in order of target log sequence number during registration and have been determined to meet the conditions in this wake-up call, they may already be in a state of being woken up but not removed in the queue. The event center iterates through each identifier and performs a node removal operation. Batch processing avoids the overhead of repeatedly traversing the queue to find nodes, thus the entire batch removal operation can be completed efficiently. After the batch business operations corresponding to the consecutive sequence number interval are completed, the locally maintained baseline log sequence number is set to the end log sequence number of the just completed consecutive interval. That is, if there are multiple intervals in the list of intervals to be processed, the process processes them in the order of their starting points, and updates the baseline log sequence number immediately after each interval is processed, or updates the baseline log sequence number to the end value of the last interval after all intervals are processed.

[0045] In one implementation of this application, after creating local wait items for each target log sequence number, the process further includes a child process registering pre-calculation conditions associated with the target log sequence number with the WAL refresh event center. When the WAL refresh event center detects a new WAL record, if it determines that the log sequence number corresponding to the record is less than the target log sequence number of any registered wait item, and the record content meets the pre-calculation conditions registered by the wait item, it sends a pre-calculation trigger signal to the corresponding child process. Based on the location information in the pre-calculation trigger signal, the child process asynchronously reads the corresponding WAL record and performs pre-calculation operations related to the final business logic, caching the pre-calculation results. When a WAL flush event occurs, and the wait item is formally awakened and confirmed to meet the pre-calculation conditions, the child process uses the cached pre-calculation results to execute the business logic, thereby reducing computational overhead.

[0046] Specifically, after a child process creates a local wait entry for a target log sequence number in its local address space, if the business logic associated with that wait entry allows for pre-processing, the child process constructs a pre-computed condition description structure. This structure defines the characteristics of the expected WAL record to match, such as the object identifier of a specific table, the transaction operation type, or the data modification mode. The child process then passes the target log sequence number along with the aforementioned pre-computed condition description structure to the WAL refresh event center. The event center creates a corresponding global wait entry node in the global wait queue and associates the pre-computed condition description structure or its index with that node, thereby establishing a binding relationship between the wait entry and the pre-computed requirement. The WAL refresh event center monitors new log records generated by the WAL write module in real time through kernel hooks or callback mechanisms. Whenever a new WAL record is written to the buffer but not yet persisted, the event center obtains the log sequence number and record content corresponding to that record. The event center traverses the global wait queue. For each wait item node registered with pre-computation conditions, it performs condition evaluation. First, it compares the log sequence number of the new record with the target log sequence number of the wait item. If the new record's log sequence number is less than the target log sequence number, it further extracts the content of the new record and calls a matching function to determine whether its content conforms to the pre-computation condition description structure associated with the wait item. If the new WAL record simultaneously satisfies the condition that its log sequence number is less than the target log sequence number and its content conforms to the pre-computation conditions, the event center determines to trigger pre-computation. The event center locates the corresponding child process and its local wait item through the bidirectional association pointer stored in the global wait item node. Subsequently, the event center sends a pre-computation trigger signal to the child process. This signal contains precise location information of the new WAL record, such as the offset and length of the record in the WAL buffer, or a pointer to the memory address of the record.

[0047] Provided its waiting state remains intact, the child process receives and processes pre-computation trigger signals via an independent low-priority thread or coroutine. The processing routine directly reads the corresponding raw record data from the WAL buffer based on the location information in the signal. The child process performs pre-processing operations strongly related to the final business logic, such as logically decoding records to generate SQL statements, filtering irrelevant fields according to subscription rules, or compressing and serializing the payload. After pre-computation, the child process stores the processing result using the log sequence number of the new WAL record as the key and marks the cached entry as valid. When a subsequent WAL flush event occurs, the child process is formally awakened and enters the re-checking phase. The child process obtains the current latest log refresh position and confirms that its target log sequence number has been satisfied. When preparing to execute business logic to process the data corresponding to the target log sequence number, if a query is made for the pre-computation result using that target log sequence number as the key, the child process directly reads the pre-processed data from the cache. If the query fails, the child process falls back to the normal processing path.

[0048] Step 104: The awakened child process obtains the current log refresh position, re-checks the waiting conditions based on the current log refresh position, and executes business logic or re-enters the waiting state based on the check results.

[0049] In one implementation of this application, after the child process is awakened by the WAL refresh event center, it obtains the latest log refresh position from the database kernel. The child process compares the latest log refresh position with its corresponding target log sequence number. If the latest log refresh position is not less than the target log sequence number, the waiting condition is deemed met, and a success signal is sent to the WAL refresh event center to cancel the corresponding waiting item. If the latest log refresh position is less than the target log sequence number, the waiting condition is deemed not met, and a reentry signal is sent to the WAL refresh event center to re-add its corresponding waiting item to the waiting queue, causing the child process to re-enter the waiting state and wait for the next wake-up.

[0050] Specifically, after the child process is awakened from the operating system scheduler's waiting state, its execution flow first jumps to the re-verification logic entry point. The process accesses a global variable maintained in the database kernel's shared memory area through an atomic memory read operation, representing the latest log refresh position of the currently persisted WAL (Wait-and-See) position. This atomic operation ensures that the process obtains a momentarily consistent snapshot value, avoiding value changes due to concurrent disk flushing during the read process, thus providing a deterministic basis for subsequent judgments. The process obtains the target log sequence number stored in its corresponding local wait item. This target value represents the minimum WAL progress required for this wait. The process compares the obtained snapshot value with the target log sequence number to determine if the latest log refresh position is greater than or equal to the target log sequence number. This comparison operation is completed in the process's local memory without any locking mechanism, ensuring the efficiency and lock-free nature of the re-verification process. If the comparison result shows that the latest log refresh position is not less than the target log sequence number, the process determines that the waiting condition has been met. At this time, the process locates the corresponding global wait item identifier in the global wait queue through an internal bidirectional associative pointer. The process sends a deregistration request signal to the WAL refresh event center, which includes the identifier of this global wait item. After sending, the process immediately begins executing the actual business logic bound to the wait item; the completion of the business logic marks the end of the wait item's lifecycle. If the comparison result shows that the latest log refresh position is less than the target log sequence number, the process determines that the wait condition is not met. This is an expected and normal branch, usually caused by spurious wakeups or concurrent wakeups. The process sends a reentrancy signal to the WAL refresh event center. This signal indicates that the event center does not need to make any modifications to the global queue, and the corresponding global wait item remains in its original position. After sending the signal, the process calls the kernel scheduling function to put its thread back into a sleep state and releases CPU resources, waiting for the next wakeup triggered by a WAL flush event.

[0051] Figure 2 is a waiting flow chart of a WAL flush event provided by an embodiment of the present application, as Figure 2 shown, the waiting flow for a WAL flush event includes: 1. Start waiting for WAL LSN: This module indicates that a certain process inside the database starts to perform the WAL flush waiting operation. In a database system, many subsystems need to wait for WAL to be flushed to a specified position, for example: log sender process, logical replication process, WAL archiving process, and incremental backup process. When these processes need to confirm that WAL has been flushed to a target Log Sequence Number (Target LSN), they enter this waiting flow.

[0052] 2. Obtain current Flush LSN: This module is configured to obtain the WAL position that has been flushed to disk in the current database system. Usually, the database kernel maintains a global WAL flush pointer (Flush LSN), which is used to indicate the log position that has been safely written to disk. The waiting process determines the current WAL flush progress by reading this pointer.

[0053] 3. Determine whether Flush LSN ≥ Target LSN: This module is configured to determine whether the current WAL flush position has reached the target log sequence number. The specific judgment logic is: if Flush LSN ≥ Target LSN, it indicates that the required log has been safely flushed to disk; if Flush LSN < Target LSN, it indicates that WAL has not been flushed to the required position. This is the core judgment condition of the entire waiting flow.

[0054] 4. End waiting and continue execution: When it is detected that Flush LSN has reached or exceeded Target LSN, the waiting process ends the waiting state and continues to perform subsequent operations. Subsequent operations of different database modules may include: sending WAL data to the standby database, sending logical replication data to the subscriber, performing WAL archiving, and performing incremental backup processing. There is no need to enter the waiting mechanism at this time.

[0055] 5. Enter WAL Flush Event waiting: When it is determined that Flush LSN has not reached Target LSN, the waiting process enters the WAL Flush Event waiting state. In this state: the process calls the condition variable waiting mechanism provided by the database system, the process enters the sleep state until the WAL flush event occurs. This mechanism avoids CPU consumption caused by traditional polling waiting.

[0056] 6. WAL disk flushing completion event: This module indicates that an event notification is triggered after the WAL Writer or log flushing module completes the WAL flushing operation. When the database flushes the WAL to disk, the system notifies all processes waiting for WAL flushing through an event broadcast mechanism. This broadcast operation will wake up all processes waiting on the WAL Flush Event.

[0057] 7. Awakened and LSN rechecked: When a waiting process receives a wake-up signal, it re-enters the Flush LSN check process. This design handles the following situations: the WAL has been flushed to the target position, the WAL has not yet reached the target position, or multiple processes are woken up simultaneously. Therefore, after being woken up, the process needs to check again (Flush LSN ≥ Target LSN). If the condition is still not met, it re-enters the waiting state. This mechanism ensures that the system can still operate correctly when faced with spurious wake-ups or concurrent wake-ups.

[0058] Figure 3 A flowchart illustrating the waiting process for a database kernel WAL refresh event, as provided in this application embodiment, is shown below. Figure 3 As shown, the waiting process for a database kernel WAL refresh event includes the following steps: The process begins with a WAL write and flush operation. After this operation occurs, the control flow enters the WAL refresh event center, which captures the underlying physical flush action and converts it into an upper-level logical event. Next, the event center broadcasts or distributes the refresh event to downstream processing processes, namely the streaming replication process, the logical replication process, and the archiving process. All three types of processes rely on the WAL log for data synchronization, distribution, or backup. Upon receiving the event notification, each process converges on the event node indicating it is currently in a blocked or pre-emptive state, waiting for a specific log sequence number to be reached. When a specific condition is met, the process returns to the WAL refresh event wake-up node, triggering the wake-up mechanism of the corresponding process and resuming it from waiting. Finally, the woken-up process enters the bottom re-checking log sequence number node to perform local condition verification, ensuring that the logic remains correct before processing, thereby guaranteeing data consistency and accuracy.

[0059] Figure 4 This is a schematic diagram illustrating the structure of a unified database waiting device based on WAL refresh events, provided as an embodiment of this application. Figure 4As shown, a unified database waiting device 200 based on WAL refresh events includes: at least one processor 201; and a memory 202 communicatively connected to the at least one processor 201. The memory 202 stores instructions executable by the at least one processor 201. These instructions, when executed by the at least one processor 201, enable the at least one processor 201 to: register global wait entries with the WAL refresh event center for multiple sub-processes within the database that need to wait for the WAL refresh to the target log sequence number; wherein each global wait entry includes at least one of the following: target log sequence number, sub-process identifier, and wake-up handle. The WAL refresh event center arranges all registered waiting items in ascending order of target log sequence number, generating a global waiting queue. When WAL disk flushing is completed and a new log refresh position is generated, the WAL refresh event center sequentially traverses the global waiting queue from the head of the queue based on the latest log refresh position, waking up only the child processes corresponding to waiting items in the global waiting queue whose target log sequence number is not greater than the latest log refresh position. The awakened child process obtains the current log refresh position, re-checks the waiting conditions based on the current log refresh position, and executes business logic or re-enters the waiting state based on the check results.

[0060] This application provides a non-volatile computer storage medium storing computer-executable instructions. These instructions are configured as follows: multiple sub-processes within the database that need to wait for the WAL (Write-Ahead Log) update to the target log sequence number register global wait items with the WAL update event center. Each global wait item includes at least one of the following: target log sequence number, sub-process identifier, and wake-up handle. The WAL update event center arranges all registered wait items in ascending order of the target log sequence number, generating a global wait queue. When the WAL update is completed and a new log update position is generated, the WAL update event center sequentially traverses the global wait queue from the head of the queue based on the latest log update position, waking up only the sub-processes corresponding to wait items in the global wait queue whose target log sequence number is not greater than the latest log update position. The awakened sub-process obtains the current log update position, re-checks the waiting conditions based on the current log update position, and executes business logic or re-enters the waiting state based on the check results.

[0061] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0062] The above descriptions are merely embodiments of this application and are not intended to limit the scope of this application. For those skilled in the art, various modifications and variations can be made to the embodiments of this application. These modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions in the embodiments of this application.

Claims

1. A unified database waiting method based on WAL refresh events, characterized in that, The method for establishing a WAL refresh event center in the database kernel includes: Multiple child processes within the database that need to wait for the WAL to be refreshed to the target log sequence number register global wait items with the WAL refresh event center; wherein, the global wait item includes at least one of the following: target log sequence number, child process identifier, and wake-up handle; The WAL refresh event center arranges all registered waiting items in ascending order of the target log sequence number, generating a global waiting queue; When the WAL disk flushing is completed and a new log refresh position is generated, the WAL refresh event center sequentially traverses the global waiting queue from the head of the queue according to the latest log refresh position, and only wakes up the child processes corresponding to the waiting items in the global waiting queue whose target log sequence number is not greater than the latest log refresh position. The awakened child process obtains the current log refresh position, re-detects the waiting conditions based on the current log refresh position, and executes business logic or re-enters the waiting state based on the detection result. After waking up the child processes corresponding to the waiting items in the global waiting queue whose target log sequence number is not greater than the latest log refresh position, the method further includes: When a subprocess has multiple target log sequence numbers to wait for, a local wait entry is created for each target log sequence number in its process local address space; wherein, a bidirectional association pointer is set between each local wait entry and the corresponding global wait entry in the global wait queue; The awakened database subprocess determines a subset of local wait items to be checked based on an internally set baseline log sequence number; wherein, the baseline log sequence number is the largest log sequence number that the subprocess has confirmed has been processed, and the subset of local wait items includes all local wait items whose target log sequence number is greater than the baseline log sequence number; The awakened database subprocess traverses the local wait item subset and detects the local wait item subset based on the locally set baseline log sequence number and the latest log refresh position; If, in the subset of local waiting items, there exists a waiting item whose target log sequence number is not greater than the latest log refresh position, the awakened database subprocess sends a notification to the WAL refresh event center to deregister the corresponding global waiting item in the global waiting queue, and after processing the business corresponding to the waiting item, updates the baseline log sequence number to the target log sequence of the waiting item. If there is a waiting item in the subset of local waiting items whose target log sequence number is greater than the latest log refresh position, then the corresponding global waiting item is still kept in the global waiting queue, and the awakened child process re-enters the waiting state.

2. The database unified waiting method based on WAL refresh events according to claim 1, characterized in that, The generation of the global waiting queue specifically includes: A global waiting queue is constructed using a doubly linked list as the underlying data structure; each node in the doubly linked list corresponds to a global waiting item, and the node data is set with pointers to the predecessor and successor nodes; When a child process initiates a registration request, the WAL refresh event center acquires a lightweight lock applied to the doubly linked list and begins traversing the doubly linked list from its head to determine the first node whose target log sequence number is greater than the target log sequence number of the new waiting item. Insert the new node before the node and release the lightweight lock.

3. The database unified waiting method based on WAL refresh events according to claim 1, characterized in that, The WAL refresh event center sequentially traverses the global waiting queue from the head based on the latest log refresh position, waking up only the child processes corresponding to waiting entries in the global waiting queue whose target log sequence number is not greater than the latest log refresh position. Specifically, this includes: After each WAL disk flush is completed, the WAL refresh event center reads and saves the latest log refresh position as the baseline value for this wake-up. Starting from the head of the global wait queue, continuously wake up the child processes corresponding to the wait items whose target log sequence number is not greater than the baseline value; When a waiting item with a target log sequence number greater than the baseline value is detected, the current traversal is stopped, and the breakpoint position of the current queue is recorded. When the next WAL flush event is triggered, a new wake-up traversal operation will begin at the breakpoint.

4. The database unified waiting method based on WAL refresh events according to claim 1, characterized in that, After waking up the child processes corresponding to the waiting items in the global waiting queue whose target log sequence number is not greater than the latest log refresh position, the method further includes: The WAL refresh event center determines the difference between the target log sequence number of the first waiting item in the queue and the latest log refresh position, and determines that the current wake-up event has a long wait when the duration of the difference is greater than a preset duration threshold. After the next WAL flush event is triggered, the WAL refresh event center selectively wakes up the child process corresponding to the first waiting item in the queue to implement timeout processing; wherein, the latest log refresh position when the timeout processing is started has not yet reached the target log sequence number of the first waiting item.

5. The database unified waiting method based on WAL refresh events according to claim 1, characterized in that, After detecting the subset of local waiting items, the method further includes: Mark local waiting items whose target log sequence number is not greater than the current latest log refresh position as items to be merged; Items with consecutively adjacent target log sequence numbers in the items to be merged are merged to form a continuous sequence number interval, and the continuous sequence number interval is recorded in the list of intervals to be processed. For each consecutive sequence number interval in the list of intervals to be processed, the subprocess sends a batch deregistration request to the WAL refresh event center; wherein, the batch deregistration request includes the identifiers of the global waiting items associated with all items to be merged within the consecutive sequence number intervals; The WAL refresh event center removes the corresponding global wait items from the global wait queue based on the identifier; After the batch business operation corresponding to the consecutive sequence number range is completed, the subprocess updates the baseline log sequence number to the end log sequence number of the consecutive sequence number range.

6. The database unified waiting method based on WAL refresh events according to claim 4, characterized in that, After creating a local wait item for each of the target log sequence numbers, the method further includes: The subprocess registers pre-computed conditions associated with the target log sequence number with the WAL refresh event center; When the WAL refresh event center detects the generation of a new WAL record, if it determines that the log sequence number corresponding to the record is less than the target log sequence number of any registered waiting item, and the record content meets the pre-calculation conditions registered by the waiting item, it sends a pre-calculation trigger signal to the corresponding child process. The subprocess asynchronously reads the corresponding WAL record based on the location information in the pre-calculated trigger signal, performs pre-calculation operations related to the final business logic, and caches the pre-calculation results. When a WAL flush event occurs, the waiting item is officially awakened and confirmed to meet the pre-calculation conditions. The child process then uses the cached pre-calculation results to execute business logic, thereby reducing computational overhead.

7. The database unified waiting method based on WAL refresh events according to claim 1, characterized in that, The awakened child process obtains the current log refresh position, re-checks the waiting conditions based on the current log refresh position, and executes business logic or re-enters the waiting state based on the check result, specifically including: After the child process is awakened by the WAL refresh event center, it obtains the latest log refresh position from the database kernel; The subprocess compares the latest log refresh position with its corresponding target log sequence number; If the latest log refresh position is not less than the target log sequence number, the waiting condition is determined to be met, and a success signal is sent to the WAL refresh event center to cancel the corresponding waiting item; If the latest log refresh position is less than the target log sequence number, the waiting condition is determined not to be met, and a reentry signal is sent to the WAL refresh event center to re-place the corresponding waiting item into the waiting queue, so that the child process re-enters the waiting state and waits for the next wake-up.

8. A unified database waiting device based on WAL refresh events, characterized in that, The device includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to perform the method described in any one of claims 1-5.

9. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are capable of performing the method described in any one of claims 1-7.

Images