Method of organizing prewritten logs in a database and computer readable storage medium

Abstract

This invention provides a method for organizing write-ahead logs in a database and a computer-readable storage medium. The method for organizing write-ahead logs in the database includes: creating a preset number of physical files for write-ahead logs during database cluster initialization; the physical files are named in a fixed format and their names are independent of the logical sequence of the write-ahead logs; establishing and maintaining a mapping relationship between physical files and logical IDs, with the logical IDs continuously increasing as physical files are reused; setting a status flag for each physical file to distinguish whether it can be reused; allocating the physical files to multiple preset log spaces, managing the physical files independently on a log space basis, and triggering checkpoints or state transitions based on the number of available physical files within a log space. This invention reduces file system metadata operations and I / O contention, stabilizes RTO metrics, avoids disk space surges, and improves database performance and reliability.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a method for organizing write-ahead logs in a database and a computer-readable storage medium. Background Technology

[0002] Write-Ahead Logging (WAL) is the core mechanism by which databases ensure transaction durability and data integrity. The core principle of WAL is that any modification to data files, such as stored tables and indexes, can only be made after the corresponding WAL record has been persisted to storage. This means that when a transaction is committed, the database does not need to immediately flush all data pages to disk; it only needs to ensure that the relevant WAL records have been written to disk. In the event of a system crash, the database can recover to a consistent state using the WAL records in the Redo Log (REDO) that were not applied to data pages.

[0003] WAL logs effectively reduce disk writes because the cost of writing WAL files sequentially to disk is far lower than the cost of flushing scattered data pages to disk. Furthermore, WAL technology enables databases to support online backups and point-in-time recovery (PITR).

[0004] In terms of physical storage, the WAL log is stored in a subdirectory of the database cluster data directory. The WAL log is physically divided into multiple segment files, each typically 16MB in size. Each segment file is further divided into multiple 8KB pages. Segment files are named using incrementing hexadecimal numbers, such as 000000010000000000000001, which reflect the segment's position within the abstract WAL sequence.

[0005] WAL log recycling is achieved by reclaiming and renaming old, no longer needed WAL segment files, which prevents the disk space occupied by the WAL directory from growing indefinitely. The system typically maintains only a certain number of WAL segment files. When a new WAL segment file is needed, it checks and reclaims those segment files whose content is located before the last checkpoint. The system considers that the modifications contained in these segment files have been fully applied to the data pages and that database consistency has been ensured, therefore they are no longer needed. The reclamation process involves renaming these no longer needed old segment files to higher segment numbers, thus allowing them to be reused as new WAL segment files.

[0006] Renaming and reusing old WAL files to achieve log circulation is a concise design. However, this mechanism, which relies on file system metadata operations, exposes two significant drawbacks in high-performance, high-pressure, or specific storage environments: performance and latency issues introduced by the "renaming" operation, and unpredictable resource management and "segment file starvation."

[0007] The performance and latency issues introduced by the "rename" operation are the most critical drawback, specifically manifested in the following four aspects: First, metadata operation overhead. A rename operation is essentially a file system metadata operation. It requires modifying directory entries and updating information in the file's inode. While this operation itself is fast, compared to a pure data append write, it is still an additional, synchronous metadata update request. Under database loads with extremely frequent WAL writes (e.g., numerous small transactions or batch data loading), this continuous, intensive rename operation becomes a small but not negligible burden on the file system, potentially introducing additional latency.

[0008] Second, there is a potential bottleneck in synchronous writes. To ensure crash consistency, WAL writes must be synchronous or controlled by a group commit policy. Although the rename operation itself occurs after the file content is written, this metadata update operation may compete for resources with critical transaction log flush operations when the file needs to be reclaimed to allocate new WAL segments, potentially becoming a bottleneck on some systems with limited I / O capabilities.

[0009] Third, performance is sensitive to specific file systems, as different file systems handle metadata operations at different efficiencies. For example, on mechanical hard drives, metadata operations may cause the read / write head to seek, impacting performance. Even on solid-state drives, if the file system itself is inefficient in metadata management, or if the storage device itself has high latency, frequent renaming operations may amplify its negative impact.

[0010] Fourth, when a file system fails, the metadata in the cache needs to be redone and repaired. When there is too much metadata, the recovery efficiency will be greatly reduced, affecting the recovery time objective (RTO).

[0011] Regarding the unpredictability of resource management and "segment file starvation," the reclamation mechanism relies on the availability of "qualified" old files for reclamation. This can lead to two problems in high-concurrency write scenarios: First, the strong coupling between checkpointing and reclamation means a WAL segment file is only considered "reclaimable" after all modifications it contains have been flushed to the data file by the checkpoint. If, for some reason, such as a small `max_wal_size` setting, improper checkpoint parameter configuration, or system I / O bottlenecks causing the checkpoint process to lag, there may not be enough reclaimable segment files. Second, segment file starvation and WAL surges occur when no reclaimable segment files are available, forcing the creation of new segment files. This can cause the WAL directory to expand rapidly, potentially exhausting the allocated disk space instantly, leading to database read-only access or system shutdown, posing a serious threat to system stability. This "suddenness" and unpredictability in resource management is a design weakness of this reclamation mechanism.

[0012] In summary, while the WAL (Write-Ahead Logging) mechanism is simple and effective, its core "rename and reuse" strategy and "archive first, reclaim later" principle inherently have drawbacks. At the performance level, frequent metadata renaming operations can become a bottleneck under high pressure. At the resource management level, its strong coupling with checkpoints leads to an unpredictable surge in disk space usage. Summary of the Invention

[0013] One object of the present invention is to provide a method for organizing write-ahead logs in a database that can solve any of the above problems.

[0014] A further objective of this invention is to address the performance issues such as metadata operation overhead and synchronization bottlenecks caused by frequent renaming in existing WAL log recycling mechanisms, reduce recovery latency during file system failures, and ensure RTO stability.

[0015] Another further objective of this invention is to avoid the risk of WAL log surges and disk space exhaustion caused by "segment file starvation," thereby achieving smoother resource management and improving the stability and reliability of the database system.

[0016] Specifically, the present invention provides a method for organizing write-ahead logs in a database, comprising: creating a preset number of physical files for write-ahead logs at once when initializing a database cluster; the physical files are named in a fixed format and their names are independent of the logical sequence of the write-ahead logs; establishing and maintaining a mapping relationship between physical files and logical IDs, wherein the logical IDs continuously increase as physical files are reused; setting a status flag for each physical file to distinguish whether it can be reused; and allocating the physical files to multiple preset log spaces, managing the physical files independently on a log space basis, and triggering checkpoints or state transitions based on the number of available physical files in a log space.

[0017] Optionally, the steps for maintaining the mapping relationship between physical files and logical IDs include: maintaining the mapping relationship between physical files and logical IDs through a hash structure in the shared memory of the database, wherein the hash structure stores the unique identifier of the physical file, the logical ID, the physical file name, and the absolute path.

[0018] Optionally, when writing to the write-ahead log, the database determines the target physical file through the mapping relationship in shared memory and directly appends the log data to the target physical file without creating a new physical file or modifying the physical file name.

[0019] Optionally, each running node of the database corresponds to a metadata description file. The metadata description file is used to persist the mapping information in the hash structure. When the running node starts, it initializes the mapping relationship in the shared memory based on the corresponding metadata description file.

[0020] Optionally, during write-ahead log archiving, the logical ID corresponding to the physical file is obtained based on the mapping relationship initialized by the metadata description file. The logical ID is used as the name of the archived file. When switching files, an archive identifier file is created. The archive identifier file records the absolute path of the corresponding physical file. After the database archiving process detects the archive identifier file, it reads the physical file based on the absolute path and completes the archiving.

[0021] Optionally, during database recovery, archive recovery directly reads the archive files named by logical IDs in the archive directory; crash recovery and standby recovery obtain the physical file name of the target physical file through the mapping relationship in shared memory, and then read the log data.

[0022] Optionally, the status identifiers include active and inactive. Physical files that are in use or cannot be reused in the database are marked as active, while physical files that are not used or can be reused are marked as inactive. When a checkpoint is executed, the status of physical files with logical IDs less than the maximum REDO log logical ID corresponding to the checkpoint is updated to inactive.

[0023] Optionally, the log space management rules include: triggering a checkpoint when the number of available physical files in a single log space is less than a preset threshold; issuing an alarm and automatically switching to other log spaces in the database that have available physical files when no physical files are available in a single log space; and entering a read-only state when no physical files are available in any log space until the checkpoint completes log reclamation.

[0024] Optionally, the conversion relationship between the logical ID and the log sequence number of the write-ahead log is: logid=((lsn)-1) / (wal_segsz_bytes), where logid is the logical ID, lsn is the log sequence number, and wal_segsz_bytes is the size of the segment file of the write-ahead log in bytes.

[0025] According to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the write-ahead log organization method in the database described above.

[0026] The write-ahead log organization method in the database of this invention completely avoids the frequent file renaming operations in the prior art by pre-creating WAL physical files with fixed names and reusing them in a loop. This significantly reduces the overhead of file system metadata modification, avoids the performance bottlenecks and synchronization delays caused by metadata operations under high load, and reduces the complexity of metadata repair when the file system fails, ensuring the stability of RTO indicators. The log space can be deployed on an independent high-speed disk, separated from the data files to reduce I / O contention, further improving the database operating efficiency. Moreover, by managing the log files on the database side, the dependence on the file system is reduced, enhancing system adaptability.

[0027] Furthermore, the write-ahead log organization method in the database of the present invention, through the design of pre-allocating log files and independently managing and reusing them according to log space, completely solves the "segment file starvation" problem, avoids the risk of WAL log surge and disk space exhaustion caused by checkpoint lag, and makes disk space usage more predictable; the hierarchical management mechanism of log space ensures the flexibility and security of resource allocation, reduces the impact of insufficient disk space on system operation, and significantly improves the reliability and continuous service capability of the database system.

[0028] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0029] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings: Figure 1 This is a schematic flowchart of a method for organizing a write-ahead log in a database according to an embodiment of the present invention; Figure 2 This is a logical operation diagram of a method for organizing a write-ahead log in a database according to an embodiment of the present invention; Figure 3 This is a log reuse logic diagram of a write-ahead log organization method in a database according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a computer program product according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention; and Figure 6 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0030] Those skilled in the art should understand that the embodiments described below are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. These partial embodiments are intended to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention. Based on the embodiments provided by the present invention, all other embodiments obtained by those skilled in the art without creative effort should still fall within the scope of protection of the present invention.

[0031] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0032] This embodiment provides a method for organizing write-ahead logs in a database, which can reduce file system metadata operations and I / O contention, stabilize RTO metrics, avoid disk space surges, and improve database performance and reliability. Figure 1 This is a schematic flowchart of a write-ahead log organization method in a database according to an embodiment of the present invention, such as... Figure 1 As shown, the organization method of write-ahead logs in a database generally includes the following steps: In step S102, when initializing the database cluster, a preset number of physical files for write-ahead logs are created at once. The physical files are named in a fixed format and their names are unrelated to the logical sequence of the write-ahead logs.

[0033] Step S104: Establish and maintain the mapping relationship between physical files and logical IDs. The logical IDs continue to grow as physical files are reused.

[0034] Step S106: Set a status flag for each physical file to distinguish whether it can be reused.

[0035] Step S108: Distribute physical files to multiple preset log spaces, manage physical files independently on a log space basis, and trigger checkpoints or state transitions based on the number of available physical files in a log space.

[0036] Specifically, maintaining the mapping relationship between physical files and logical IDs in step S104 may include: maintaining the mapping relationship between physical files and logical IDs through a hash structure in the shared memory of the database, wherein the hash structure stores the unique identifier of the physical file, the logical ID, the physical file name, and the absolute path.

[0037] Specifically, when writing to the write-ahead log, the database determines the target physical file through the mapping relationship in shared memory and directly appends log data to the target physical file without creating a new physical file or modifying the physical file name.

[0038] Specifically, each running node of the database corresponds to a metadata description file. The metadata description file is used to persist the mapping information in the hash structure. When the running node starts, it initializes the mapping relationship in the shared memory based on the corresponding metadata description file.

[0039] Specifically, during write-ahead log archiving, the logical ID corresponding to the physical file is obtained based on the mapping relationship initialized by the metadata description file. The logical ID is used as the name of the archived file. When switching files, an archive identifier file is created. The archive identifier file records the absolute path of the corresponding physical file. After the database archiving process detects the archive identifier file, it reads the physical file based on the absolute path and completes the archiving.

[0040] Specifically, during database recovery, archive recovery directly reads the archive files named by logical IDs in the archive directory; crash recovery and standby recovery obtain the physical file name of the target physical file through the mapping relationship in shared memory, and then read the log data.

[0041] Specifically, the status identifiers in step S106 include active and inactive. Physical files that are in use or cannot be reused in the database are marked as active, while physical files that are not used or can be reused are marked as inactive. When a checkpoint is executed, the status of physical files whose logical ID is less than the maximum REDO log logical ID corresponding to the checkpoint is updated to inactive.

[0042] Specifically, the log space management rules in step S108 include: when the number of available physical files in a single log space is less than a preset threshold, a checkpoint is triggered; when there are no available physical files in a single log space, an alarm is issued and the system automatically switches to other log spaces in the database that have available physical files; when there are no available physical files in all log spaces, the database enters a read-only state until the checkpoint completes log reclamation.

[0043] Specifically, the conversion relationship between logical ID and log sequence number of write-ahead log is: logid=((lsn)-1) / (wal_segsz_bytes), where logid is the logical ID, lsn is the log sequence number, and wal_segsz_bytes is the size of the segment file of write-ahead log in bytes.

[0044] It is important to emphasize that the write-ahead log (WAL) organization method in this embodiment utilizes pre-created logs and reuses them cyclically at runtime. Since there is no need to create or modify new log files, the number of operations on file system metadata is reduced. Self-management of database log files lowers the database's dependence on the file system. Excessive modification of file metadata is avoided at the database level. This prevents the file system from bearing an excessive load, which could negatively impact performance and RTO. This cyclical reuse technique reduces the amount of metadata modification at the WAL end, thus achieving stability in database performance and RTO metrics.

[0045] By generating all necessary log files during data cluster initialization and continuously reusing these files, the waiting issues caused by WAL log file creation can be resolved. Placing the log space on different high-speed disks, separate from data files, reduces I / O contention and effectively improves production performance.

[0046] The database only needs to pre-allocate enough redo log files and use these files in a loop at runtime by allocating log space reasonably, instead of constantly adding new log files, which reduces the impact on the system when disk space is insufficient.

[0047] Figure 2 This is a logical operation diagram of a write-ahead log organization method in a database according to an embodiment of the present invention. Figure 3This is a log reuse logic diagram of a write-ahead log organization method in a database according to an embodiment of the present invention. For example... Figure 2 and Figure 3 As shown, when initializing the data cluster, all WAL logs are initialized at once and allocated to different log spaces. During runtime, this batch of logs is used cyclically. The physical file names of the WAL logs are kept in the format of REDOLOG0, REDOLOG1, REDOLOG2, etc. After using REDOLOG2, the REDOLOG0 file is reused to write WAL logs. It is because of this reuse mechanism that the writing order of the logs cannot be determined solely from the file names.

[0048] To determine the order of the logs, a logical ID needs to be assigned to each file. Each file has a corresponding logical ID, which increases and changes as the file is reused. For example, REDOLOG0 will correspond to logical ID 1 when it is used for the first time, and logical ID 4 when it is reused for the second time.

[0049] Each node maintains a metadata description file, storing the relationship between physical files and logical IDs. A hash structure mapping logical file names to physical file names is maintained in shared memory.

[0050] Since the physical filenames of physical files are no longer directly related to their logids, a structure is needed to map the relationship between physical files and logids. Therefore, a hash structure is built in shared memory to map file relationships. The shared memory structure contains: 1) a unique identifier for each file; and 2) basic physical information about the file, including its logical ID, physical filename, and absolute path. This structure allows mapping the physical and logical relationships between files and also enables the placement of redo log files in different paths, providing greater flexibility. Figure 2 As shown, this hash structure is stored in the shared memory XLogFileCache, working in conjunction with Kingbase's walwrite and walsender processes to achieve the association scheduling of log read / write with physical files. To persist the information in this structure, a file metadata description file, sys_redolog_nodeid, is created for each running node. This file persists the file information of the shared memory structure. This file is updated when the file information in the shared memory structure changes. When the cluster or node starts, this file is used to initialize the file information in shared memory.

[0051] To determine which files can be reused and which cannot, a state is assigned to each file: files currently in use and not reusable are marked as "active," while files that are not in use or can be reused are marked as "inactive." This way, whenever a file is needed, an "inactive" file is searched for. File state information is recorded in the shared memory XLogFileCache. During checkpointing, all files with logical IDs less than the maximum REDO log logical ID corresponding to the checkpoint are set to "inactive" to allow for subsequent reuse. In the event of a shutdown or crash, file states are meaningless and this information does not need to be persisted.

[0052] The conversion relationship between Log Sequence Number (lsn) and Logical ID (logid) is as follows: logid = ((lsn) - 1) / (wal_segsz_bytes). The mapping relationship between physical file names and logids is stored in XLogFileCache and the sys_redolog_nodeid file. Through these two relationships—the conversion relationship between log sequence number and logical ID, and the mapping relationship between logical ID and physical file—the physical storage corresponding to the log can be accurately located.

[0053] Based on this, the concept of a log space is proposed: logs within a log space are managed independently, and a checkpoint request is issued when the number of available WAL log files in a single log space falls below a preset threshold. For example, the preset threshold can be 10%. It should be noted that the specific value of the preset threshold mentioned above is merely an example and is not intended to limit the invention. In other embodiments, the preset threshold can be set to other values ​​according to actual circumstances.

[0054] like Figure 3 As shown, WAL SPACE1 and WAL SPACE2 independently manage the physical files Redolog0, Redolog1, and Redolog2, respectively, achieving load balancing and independent scheduling of log storage. When the WAL logs in a single log space are exhausted, an alarm is issued, a checkpoint is requested, and other log spaces with available logs are automatically used simultaneously. If no logs are available in any log space, the database enters a read-only state until the checkpoint completes log reclamation. It should be noted that this situation is extremely rare.

[0055] The process during database operation is as follows: At startup, the information in the metadata description file is loaded into shared memory. Then, before reading the log, a mapping relationship between physical files and logical IDs is created. After that, checkpoint information can be read normally, and startup or recovery can proceed.

[0056] Before writing to the WAL log, a mapping relationship between physical files and logical IDs needs to be created through XLogFileCache. After that, normal write operations can be performed. When all logs are in an active state and there are no usable logs during writing, an alarm message is issued, prompting the administrator to add log information. At the same time, a checkpoint is initiated to force data pages to be written to disk and WAL logs to be recycled. At this time, writing is blocked, and the database write business is in a read-only state.

[0057] Once log writing is complete, for system reliability, logs need to be archived promptly for permanent storage. When archiving log files to the archive directory, they need to be renamed using logical filenames. When switching files, an archive identifier file named "ready" is created using logical filenames, and the absolute path of the corresponding physical file is written in the file. When the archiving process detects the "ready" file, it reads the physical file path from the file, executes the archiving command, and archives the physical file to the archive directory using the logical filenames.

[0058] When the database starts up, it reads the checkpoint logs. Before reading the logs, it loads the log metadata description file into memory to build a mapping relationship between logical and physical files. Then, it resolves the logical ID through the checkpoint's LSN and obtains the physical file through the mapping relationship for reading.

[0059] Recovery is divided into three methods: archive recovery, crash recovery, and standby recovery. Archive recovery retrieves the recovered files from the archive directory. These files are named using the logical file names and can be read normally without file mapping.

[0060] Both crash recovery and standby recovery require accessing physical files in the local directory to read logs. Therefore, it's necessary to obtain the physical filename of the log file containing the data to be recovered through a mapping relationship before reading the logs.

[0061] In this embodiment, by pre-creating WAL physical files with fixed names and reusing them cyclically, the frequent file renaming operations in the prior art are completely avoided, significantly reducing the overhead of file system metadata modification, avoiding performance bottlenecks and synchronization delays caused by metadata operations under high load, and reducing the complexity of metadata repair when the file system fails, thus ensuring the stability of RTO indicators. The log space can be deployed on an independent high-speed disk, separated from the data files to reduce I / O contention, further improving the database operating efficiency. Moreover, by managing the log files on the database side, the dependence on the file system is reduced, enhancing system adaptability.

[0062] By pre-allocating log files and managing and reusing them independently according to log space, the "segment file starvation" problem is completely solved, avoiding the risk of WAL log surges and disk space exhaustion caused by checkpoint lag, making disk space usage more predictable; the hierarchical management mechanism of log space ensures the flexibility and security of resource allocation, reduces the impact of insufficient disk space on system operation, and significantly improves the reliability and continuous service capability of the database system.

[0063] The flowcharts provided in the above embodiments are not intended to indicate that the operations of the method will be performed in any particular order, or that all operations of the method are included in all every case. Furthermore, the method may include additional operations. Within the scope of the technical concept provided by the methods in the above embodiments, additional variations can be made to the above methods.

[0064] This embodiment also provides a computer program product, a computer-readable storage medium, and a computer device. Figure 4 This is a schematic diagram of a computer program product 500 according to an embodiment of the present invention. Figure 5 This is a schematic diagram of a computer-readable storage medium 300 according to an embodiment of the present invention. Figure 6 This is a schematic diagram of a computer device 400 according to an embodiment of the present invention.

[0065] Computer program product 500 includes computer program 310, which, when executed by processor 410, implements the steps of the write-ahead log organization method in any of the above embodiments. Computer-readable storage medium 300 stores the computer program 310 thereon, which, when executed by processor 410, implements the steps of the write-ahead log organization method in any of the above embodiments. Computer device 400 may include memory 420, processor 410, and computer program 310 stored in memory 420 and running on processor 410.

[0066] The computer program 310 used to perform the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​and procedural programming languages.

[0067] Computer program 310 may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0068] In some embodiments, in order to perform aspects of the present invention, electronic circuits including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs) or programmable logic arrays (PLAs) can execute computer-readable program instructions to personalize the electronic circuits by utilizing state information of computer-readable program instructions.

[0069] For the purposes of this embodiment, computer program product 500 is a related product containing computer program 310. For the purposes of this embodiment, computer-readable storage medium 300 is a tangible device capable of holding and storing computer program 310, and can be any device capable of containing, storing, communicating, propagating or transmitting computer program 310 for use by or in conjunction with an instruction execution system, apparatus or device.

[0070] More specific examples (a non-exhaustive list) of computer-readable storage media 300 include the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, and any suitable combination of the foregoing.

[0071] Computer device 400 may include memory 420, processor 410, and computer program 310 stored on memory 420 and running on processor 410, wherein processor 410 executes computer program 310 to implement the steps of the write-ahead log organization method in the database of any of the above embodiments.

[0072] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any machine-readable storage medium for use by, or in conjunction with, instruction execution systems, apparatuses or devices (such as computer-based systems, processor-based systems or other systems that can fetch and execute instructions from, or instruction execution systems, apparatuses or devices).

[0073] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system.

[0074] Computer device 400 can be, for example, a server, desktop computer, laptop computer, tablet computer, or smartphone. In some examples, computer device 400 can be a cloud computing node. Computer device 400 can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. Computer device 400 can be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0075] Computer device 400 may include a processor 410 adapted to execute stored instructions and a memory 420 that provides temporary storage space for the operation of instructions during operation. The processor 410 may be a single-core processor, a multi-core processor, a computing cluster, or any other configuration. The memory 420 may include random access memory (RAM), read-only memory, flash memory, or any other suitable storage system.

[0076] The processor 410 can be connected via a system interconnect (e.g., PCI, PCI-Express, etc.) to an I / O interface (input / output interface) suitable for connecting the computer device 400 to one or more I / O devices (input / output devices). I / O devices may include, for example, a keyboard and indicating devices, where indicating devices may include a touchpad or touchscreen, etc. I / O devices may be built into the computer device 400 or may be external devices connected to the computing device.

[0077] The processor 410 may also be linked via a system interconnect to a display interface suitable for connecting the computer device 400 to a display device. The display device may include a display screen as a built-in component of the computer device 400. The display device may also include an external computer monitor, television, or projector connected to the computer device 400. Furthermore, a network interface controller (NIC) may be adapted to connect the computer device 400 to a network via a system interconnect. In some embodiments, the NIC may use any suitable interface or protocol (such as an Internet Minicomputer System Interface) to transmit data. The network may be a cellular network, a radio network, a wide area network (WAN), a local area network (LAN), or the Internet, etc. Remote devices may connect to the computing device via the network.

[0078] Therefore, those skilled in the art should recognize that although numerous exemplary embodiments of the present invention have been shown and described in detail herein, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope of the invention. Thus, the scope of the present invention should be understood and construed as covering all such other variations or modifications.

Claims

1. A method for organizing a write-ahead log in a database, comprising: When initializing the database cluster, a preset number of physical files for the write-ahead log are created at once. The physical files are named in a fixed format and their names are unrelated to the logical sequence of the write-ahead log. Establish and maintain the mapping relationship between the physical files and logical IDs, wherein the logical IDs continuously increase as the physical files are reused; A status flag is set for each physical file to distinguish whether it can be reused; as well as The physical files are allocated to multiple preset log spaces, and the physical files are managed independently in each log space. Checkpoints or state switches are triggered based on the number of available physical files in each log space.

2. The method according to claim 1, wherein the step of maintaining the mapping relationship between the physical file and the logical ID includes: The mapping relationship between the physical file and the logical ID is maintained by a hash structure in the shared memory of the database. The hash structure stores the unique identifier of the physical file, the logical ID, the name of the physical file, and the absolute path.

3. The method according to claim 2, wherein, When the database writes to the write-ahead log, it determines the target physical file through the mapping relationship in the shared memory and directly appends log data to the target physical file without creating a new physical file or modifying its name.

4. The method according to claim 2, wherein, Each running node of the database corresponds to a metadata description file, which is used to persist the mapping information in the hash structure. When the running node starts, it initializes the mapping relationship in the shared memory based on the corresponding metadata description file.

5. The method according to claim 4, wherein, During the write-ahead log archiving process, the logical ID corresponding to the physical file is obtained based on the mapping relationship initialized by the metadata description file. The logical ID is used as the name of the archived file. When switching files, an archive identifier file is created. The archive identifier file records the absolute path corresponding to the physical file. After the database archiving process detects the archive identifier file, it reads the physical file based on the absolute path and completes the archiving.

6. The method according to claim 5, wherein, During database recovery, archive recovery directly reads the archive file named with the logical ID in the archive directory; Crash recovery and standby recovery obtain the physical file name of the target physical file through the mapping relationship in the shared memory, and then read the log data.

7. The method according to claim 1, wherein, The status identifiers include active and inactive. Physical files that are in use or cannot be reused in the database are marked as active, while physical files that are unused or can be reused are marked as inactive. When the checkpoint is executed, the status of the physical file whose logical ID is less than the maximum REDO log logical ID corresponding to the checkpoint is updated to inactive.

8. The method according to claim 1, wherein the management rules for the log space include: The checkpoint is triggered when the number of available physical files in a single log space is less than a preset threshold. When a single log space has no available physical file, an alarm is issued and the system automatically switches to another log space in the database that has available physical files. When all the physical files in the log spaces are unavailable, the database enters a read-only state until the checkpoint completes log reclamation.

9. The method according to claim 1, wherein, The conversion relationship between the logical ID and the log sequence number of the write-ahead log is: logid = ((lsn) - 1) / (wal_segsz_bytes), where logid is the logical ID, lsn is the log sequence number, and wal_segsz_bytes is the byte size of the segment file of the write-ahead log.

10. A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the method for organizing a write-ahead log in a database as described in any one of claims 1 to 9.

Images