Ceph metadata disaster recovery acceleration method and system based on persistent memory and RDMA network

Abstract

The invention relates to the technical field of metadata management, in particular to a Ceph metadata disaster recovery acceleration method and system based on a persistent memory and an RDMA network, the Ceph metadata disaster recovery acceleration method based on the persistent memory and the RDMA network comprises the following steps of PMEM-WAL double-write mechanism, metadata snapshot differential synchronization and failover optimization; the method has the beneficial effects that the consistency of main and standby data at any moment is ensured through RDMA atomic double writing. And the standby node directly mounts the PMEM log, and the recovery time is reduced from the minute level to the second level. And the difference snapshot only synchronizes incremental data, so that the total transmission overhead is reduced. And the RDMA hardware unloading and PMEM byte addressing characteristics reduce the CPU load by more than 40%.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of metadata management, in particular to a Ceph metadata disaster backup acceleration method and system based on persistent memory and an RDMA network. BACKGROUND

[0002] RocksDB is an embedded key-value storage database developed by Facebook. It is designed for fast storage and retrieval of large amounts of data, and is particularly suitable for application scenarios that require high-performance storage solutions, such as real-time big data analysis, cache systems, etc. In RocksDB, data is organized in levels, and a process called "compaction" is used to manage and optimize storage space.

[0003] There is a mechanism called log replay in the mechanism of Rocksdb itself. Simply put, when the database is not gracefully exited, the data in memory will be lost; but when the database starts next time, the lost data in memory will be replayed according to the WAL log file, and the data entries recorded in the WAL log will be added to a new memtable, thereby preventing data loss.

[0004] High network latency: metadata logs (WAL) need to be synchronized to standby nodes through TCP / IP network, and transmission delay makes it difficult to optimize RPO (recovery point objective).

[0005] Large log replay overhead: standby nodes need to replay WAL logs one by one to recover the state, and RTO (recovery time objective) is as long as minutes.

[0006] Low efficiency of full synchronization: periodic metadata snapshots need to be transmitted in full, occupying a large amount of bandwidth.

[0007] Existing technologies (such as log replication based on LevelDB and multi-copy storage) cannot balance low latency and high reliability, and there is an urgent need for a hardware and software collaborative optimization disaster recovery solution. SUMMARY

[0008] The purpose of the present application is to provide a Ceph metadata disaster backup acceleration method and system based on persistent memory and an RDMA network to solve the problems raised in the background.

[0009] To achieve the above purpose, the present application provides the following technical scheme: a Ceph metadata disaster backup acceleration method based on persistent memory and an RDMA network, comprising the following steps:

[0010] Step 1: PMEM-WAL dual writing mechanism

[0011] Perform memory registration and virtual address mapping: initialize PMEM memory pool, format PMEM device as persistent memory namespace and create fixed-size virtual address space; register PMEM virtual address space to RDMA network card to generate memory region descriptor; standby node transmits the VA and Key of MR to the master node through a secure channel to establish a remote direct write channel;

[0012] Develop atomic double-write process: CephMDS generates WAL log and writes it to the local NVMeSSD log file through the Rocksdb interface; the master node uses RDMASend / Write operation to asynchronously write WAL log content to the specified virtual address of the standby node PMEM; through the atomic confirmation protocol, it is ensured that the double-write of local NVMe and remote PMEM is completed before the successful response is returned to the client;

[0013] Implement exception handling and degradation mechanism: when the network is interrupted, the master node temporarily stores the unsynchronized WAL log in the local PMEM buffer, and supports breakpoint resume after network recovery; when the standby node PMEM is unavailable, the master node switches to asynchronous replication mode and synchronizes the WAL log to the NVMeSSD of the standby node in batches;

[0014] Step two: metadata snapshot differential synchronization

[0015] Generate consistent snapshot: the master node starts the snapshot process every fixed period, suspends new WAL log writing; use PMDK transactional API to lock PMEM log area, generate consistent snapshot; record snapshot timestamp, start LSN, end LSN and CRC64 checksum value, store in PMEM header metadata area;

[0016] Calculate and compress difference bitmap: use PMEM mmap interface to compare current snapshot with last snapshot byte by byte to generate difference bitmap; apply sparse bitmap encoding and LZ4 algorithm to optimize and compress difference bitmap; master node transmits compressed difference bitmap and corresponding data block to standby node PMEM difference staging area through RDMARead operation;

[0017] Perform incremental synchronization and merge: standby node parses difference bitmap and writes data according to address to PMEM corresponding area; use PMDK pmem_memcpy_persist function to ensure persistence; update local snapshot metadata, record the latest LSN and checksum value;

[0018] Step three: fault switching optimization

[0019] PMEM state monitoring: the master node periodically sends heartbeat packets to the standby node, with the current maximum LSN value; the standby node polls the master node PMEM metadata area to verify log continuity; if the heartbeats are continuously lost, the master node is determined to be faulty, triggering the switching process;

[0020] Performing fast switching process: log integrity verification is performed to ensure that the log is not damaged; the standby node calls the secondary instance of Rocksdb, mounts the PMEM log and applies the log to generate the latest metadata view; the standby node is promoted to the master node and takes over the CephMDS service;

[0021] Implementing parallel recovery mechanism: when there are multiple backup nodes, each node reads the log from PMEM in parallel; a new master is elected through a distributed consensus protocol to speed up cluster recovery.

[0022] Preferably, in the memory registration and virtual address mapping process of the PMEM-WAL dual-write mechanism, when the PMEM memory pool is initialized, the size and attribute parameters of the persistent memory namespace are accurately configured according to the preset storage capacity and performance requirements; when registering the PMEM virtual address space to the RDMA network card, a specific registration strategy is used to ensure that the generation of the memory region descriptor conforms to the efficient transmission requirements of the RDMA network; when the standby node transmits the VA and Key of the MR to the master node, an encryption algorithm is used to encrypt the transmission data to ensure the confidentiality and integrity of the secure channel.

[0023] Preferably, in the consistency snapshot generation process of the metadata snapshot differential synchronization, when the master node starts the snapshot process, an intelligent scheduling algorithm is used to dynamically adjust the snapshot period according to the current load and business priority of the system; when locking the PMEM log area using the transactional API of PMDK, a reasonable lock timeout is set to avoid performance degradation caused by long-term locking; when recording snapshot metadata, a redundant storage method is used to store the snapshot timestamp, start LSN, end LSN, and CRC64 checksum in multiple locations in the PMEM header metadata area to improve data reliability.

[0024] Preferably, in the PMEM state monitoring process of the fault switching optimization, when the master node sends heartbeat packets, an adaptive heartbeat interval adjustment mechanism is used to dynamically adjust the heartbeat interval time according to the network status and system stability; when the standby node polls the master node PMEM metadata area, an incremental polling method is used to only obtain the metadata information that has changed since the last polling, reducing network transmission volume and polling time; when determining that the master node is faulty, a comprehensive judgment is made based on multiple monitoring indicators, considering not only the continuous loss of heartbeats but also the system resource usage of the master node, network connection quality, and other factors to improve the accuracy of fault judgment.

[0025] Preferably, in the parallel recovery mechanism, when multiple backup nodes read logs from PMEM in parallel, a load balancing algorithm is adopted to reasonably allocate log reading tasks according to the processing capacity and current load of each node; when a new master is elected through a distributed consistency protocol, an optimized election strategy is adopted to reduce the communication overhead and time delay in the election process, and to ensure that the new master can be quickly elected and take over the service; after the new master takes over the service, a data synchronization verification mechanism is adopted to comprehensively verify the data between the new master and the backup nodes, and to ensure the consistency and integrity of the data.

[0026] A Ceph metadata disaster recovery acceleration method based on persistent memory and RDMA network system, characterized by comprising:

[0027] (1) PMEM-WAL dual-write module, comprising:

[0028] Memory registration and virtual address mapping submodule: responsible for PMEM memory pool initialization, formatting PMEM device into persistent memory namespace and creating fixed-size virtual address space; registering PMEM virtual address space to RDMA network card to generate memory region descriptor; arranging backup node to transmit VA and Key of MR to master node through secure channel to establish remote direct write channel;

[0029] Atomic dual-write flow submodule: CephMDS generates WAL log and writes it to local NVMeSSD log file through Rocksdb interface; master node uses RDMASend / Write operation to asynchronously write WAL log content to specified virtual address of backup node PMEM; atomic confirmation protocol is adopted to ensure that local NVMe and remote PMEM dual-write are completed before returning success response to client;

[0030] Exception handling and degradation mechanism submodule: when the network is interrupted, the master node temporarily stores the unsynchronized WAL log in the local PMEM buffer, and supports breakpoint resume after network recovery; when the backup node PMEM is unavailable, the master node switches to asynchronous replication mode and periodically synchronizes WAL logs in batches to the NVMeSSD of the backup node;

[0031] (2) Metadata snapshot differential synchronization module, comprising:

[0032] Consistent snapshot generation submodule: the master node starts the snapshot process every fixed period, suspends new WAL log writing; uses the transactional API of PMDK to lock the PMEM log area and generates a consistent snapshot; records the snapshot timestamp, starting LSN, ending LSN and CRC64 checksum value in the PMEM header metadata area;

[0033] Difference bitmap calculation and compression submodule: byte-by-byte comparison of the current snapshot and the last snapshot using the mmap interface of PMEM to generate a difference bitmap; application of sparse bitmap encoding and LZ4 algorithm for optimized compression of the difference bitmap; the master node transmits the compressed difference bitmap and its corresponding data block to the difference staging area of the standby node PMEM through the RDMARead operation;

[0034] Incremental synchronization and merging submodule: the standby node parses the difference bitmap and writes the data according to the address into the corresponding area of PMEM; uses the pmem_memcpy_persist function of PMEM to ensure persistence; updates the local snapshot metadata to record the latest LSN and checksum;

[0035] (3) Fault switching optimization module, comprising:

[0036] PMEM state monitoring submodule: the master node periodically sends a heartbeat packet to the standby node, with the current maximum LSN value; the standby node polls the master node PMEM metadata area to verify the log continuity; if the heartbeats are continuously lost, the master node is determined to be faulty, triggering the switching process.

[0037] Fast switching process submodule: perform log integrity check to ensure that the log is not damaged; the standby node calls the secondary instance of Rocksdb, mounts the PMEM log and applies the log to generate the latest metadata view; promote the standby node to the master node and take over the CephMDS service.

[0038] Parallel recovery mechanism submodule: when there are multiple backup nodes, each node reads the log from PMEM in parallel; elect a new master through a distributed consistency protocol to speed up cluster recovery.

[0039] Preferably, in the memory registration and virtual address mapping submodule of the PMEM-WAL dual-write module, when the PMEM memory pool is initialized, the size, access rights and cache strategy of the persistent memory namespace are accurately configured according to the system's preset storage capacity and performance indicators; when the PMEM virtual address space is registered to the RDMA network card, a specific registration algorithm is used to optimize the generation of memory region descriptors to reduce the delay of RDMA operations; when the standby node transmits the VA and Key of the MR to the master node, multiple encryption technologies are used to encrypt the transmission data to ensure the absolute confidentiality and attack resistance of the secure channel.

[0040] Preferably, in the consistency snapshot generation submodule of the metadata snapshot differential synchronization module, when the master node starts the snapshot process, an intelligent scheduling algorithm is adopted, the current load condition, business priority and historical snapshot generation time are combined, and the snapshot period is dynamically adjusted; when locking the PMEM log area using the transactional API of PMEM, a flexible lock timeout time and conflict resolution strategy are set, so that the system performance is prevented from being reduced due to long-time locking or lock conflict; when recording snapshot metadata, a distributed storage mode is adopted, the snapshot timestamp, starting LSN, ending LSN and CRC64 check value are stored in the PMEM header metadata area of multiple nodes at the same time, and the reliability and availability of data are improved.

[0041] Preferably, in the PMEM state monitoring submodule of the fault switching optimization module, when the master node sends a heartbeat packet, an adaptive heartbeat interval adjustment mechanism is adopted, the heartbeat interval time is adjusted in real time according to the network bandwidth, delay and system stability; when the standby node polls the PMEM metadata area of the master node, an incremental polling and caching technology is adopted, only the metadata information changed after the last polling is obtained, and the metadata information is cached locally, so that the network transmission amount and polling time are reduced; when the master node failure is determined, comprehensive judgment is made in combination with multiple monitoring indexes, in addition to continuous loss of heartbeat, the system resource usage condition, network connection quality, disk I / O performance and other factors of the master node are also considered, and the accuracy and reliability of fault judgment are improved.

[0042] Preferably, in the parallel recovery mechanism submodule of the fault switching optimization module, when multiple backup nodes read logs from PMEM in parallel, a load balancing and task allocation algorithm is adopted, the log reading task is reasonably allocated according to the processing capacity, current load condition and network bandwidth of each node; when a new master is elected through a distributed consistency protocol, an optimized election strategy is adopted, the performance indexes, historical election records and network topology structure of the nodes are combined, the communication overhead and time delay in the election process are reduced, and it is ensured that the new master can be quickly elected and take over the service; after the new master takes over the service, a data synchronization verification and repair mechanism is adopted, the data between the new master and the backup nodes is comprehensively verified, and when data inconsistency is found, the data inconsistency is repaired in time, and the consistency and integrity of data are ensured.

[0043] Compared with the prior art, the method and system have the following beneficial effects:

[0044] The Ceph metadata disaster recovery acceleration method and system based on persistent memory and RDMA network provided by the application ensure the consistency of master and backup data at any moment through RDMA atomic double writing. The backup node directly mounts the PMEM log, and the recovery time is reduced from minutes to seconds. The differential snapshot only synchronizes the incremental data, and the full transmission overhead is reduced. The CPU load is reduced by more than 40% through the RDMA hardware offloading and the PMEM byte addressing characteristics. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 Flow chart of the method of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions of the present application clear, complete description, and the advantages are more clear and obvious, the following will be further described in detail with the help of the accompanying drawings. It should be understood that the specific embodiments described here are part of the embodiments of the present application, not all embodiments, and are only used to explain the embodiments of the present application, and not used to limit the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0047] Embodiment one, please refer to Figure 1 The present application provides a technical solution: a Ceph metadata disaster recovery acceleration method based on persistent memory and RDMA network, characterized by comprising the following steps:

[0048] I. PMEM-WAL double writing mechanism

[0049] 1.1 Memory registration and virtual address mapping

[0050] 1. PMEM memory pool initialization: a) The backup node formats the PMEM device into a persistent memory namespace, creates a virtual address space with a fixed size (such as 1TB), and divides it into multiple logical segments, each of which stores continuous WAL logs. b) Register the PMEM virtual address space to the RDMA network card through RDMA memory registration, generate a memory region descriptor (MR), which contains a virtual address and an access key.

[0051] 2. Master node address synchronization: the backup node transmits the VA and Key of the MR to the master node through a secure channel, and the master node RDMA network card binds the address space to establish a remote direct write channel.

[0052] 1.2 Atomic double writing process

[0053] 1. Log generation and local writing: Ceph MDS generates WAL logs for metadata operations (such as file creation, directory modification), and calls Rocksdb interface to write log files to local NVMe SSD.

[0054] 2. RDMA asynchronous writing PMEM:

[0055] a) The master node writes the WAL log content directly to the specified virtual address of the backup node PMEM through the RDMA Send / Write operation.

[0056] b) Log write in non-blocking mode, master maintains two independent queues: local queue manages NVMe SSD write requests; remote queue manages PMEM write requests of RDMA NIC.

[0057] 3. Atomic acknowledgement protocol:

[0058] a) Master waits for the double write completion signals of local NVMe and remote PMEM (through NVMe CompletionQueue and RDMA Work Completion polling).

[0059] b) Only after both confirm success, return operation success response to client, otherwise trigger retry or degradation process.

[0060] 1.3 Exception handling and degradation mechanism

[0061] 1. Network termination handling:

[0062] a) If RDMA link termination, master node temporarily stores bit-synchronized WAL log to local PMEM buffer, records breakpoint position (Last Synced LSN).

[0063] b) After network recovery, support breakpoint resume through difference.

[0064] 2. Standby node PMEM failure: when standby node PMEM is unavailable, master node switches to asynchronous replication mode, temporarily stores WAL log locally, and periodically synchronizes in batches to standby node's NVMe SSD.

[0065] II. Metadata snapshot differential synchronization

[0066] 2.1 Consistent snapshot generation

[0067] 1. Snapshot freeze window:

[0068] a) Master node starts snapshot process every fixed period (such as 60 seconds), suspends new WAL log write (freeze time <= 10 ms).

[0069] b) Locks PMEM log area through PMDK transactional API to generate consistent snapshot.

[0070] 2. Snapshot metadata record:

[0071] Records snapshot timestamp, start LSN (Log Sequence Number), end LSN, and CRC64 checksum value, and stores in PMEM header metadata area.

[0072] 2.2 Difference bitmap calculation and compression

[0073] 1. Byte-level diff detection: Use mmap interface of PMEM to set current snapshot and last snapshot in memory, compare byte by byte to generate diff bitmap, mark changed regions.

[0074] 2. Bitmap optimized compression:

[0075] a). Apply sparse bitmap encoding to diff bitmap, only record start address and length of continuous changed block.

[0076] b). Further use LZ4 algorithm to reduce transmission volume.

[0077] 2.3 Incremental synchronization and merging

[0078] 1. Diff data transmission: Master node transmits compressed diff bitmap and corresponding data block to backup node PMEM's diff staging area through RDMA Read operation.

[0079] 2. Backup node data merging: Backup node parses diff bitmap, writes data to corresponding area in PMEM according to address, and ensures persistence through pmem_memcpy_persist function of PMEM.

[0080] 3. Snapshot metadata update: Backup node updates local snapshot metadata, records the latest LSN and checksum.

[0081] Three. Fault switching optimization

[0082] 3.1 PMEM state monitoring

[0083] Heartbeat and log tracking: Master node sends heartbeat packet to backup node regularly, with the current maximum LSN value; backup node polls master node PMEM metadata area through RDMA Read to verify log continuity.

[0084] 3.2 Fast switching process

[0085] 1. Fault detection and triggering: If backup node continuously loses heartbeat (such as timeout for 3 times), it determines that the master node has failed and triggers the switching process.

[0086] 2. Log integrity verification: Backup node reads the CRC64 checksum value of the latest snapshot in PMEM and compares it with the locally calculated checksum to ensure that the log is not damaged.

[0087] 3. Direct mounting of PMEM log:

[0088] a). Backup node calls Rocksdb's secondary instance to map the WAL log in PMEM as a read-only database.

[0089] b). Apply journal quickly through DB::TryCatchUpWithPrimary() interface to generate the latest metadata view.

[0090] 4. Service takeover: standby node promotes to new primary node, takes over Ceph MDS service, and client traffic seamlessly switches over.

[0091] 3.3 Parallel recovery mechanism

[0092] Multiple-copy parallel loading: if there are multiple backup nodes, each node can read the journal from PMEM in parallel, elect a new master through a distributed consensus protocol (such as Raft, paxos), and speed up cluster recovery.

[0093] In the embodiment two, on the basis of the embodiment one, a system of a Ceph metadata disaster recovery acceleration method based on persistent memory and RDMA network is proposed, comprising:

[0094] (1) PMEM-WAL dual-write module, comprising:

[0095] Memory registration and virtual address mapping submodule: responsible for PMEM memory pool initialization, formatting PMEM device into persistent memory namespace and creating fixed-size virtual address space; registering PMEM virtual address space to RDMA network card to generate memory region descriptor; arranging standby node to transmit MR's VA and Key to primary node through secure channel to establish remote direct write channel; when PMEM memory pool is initialized, according to the system preset storage capacity and performance index, the size, access permission and cache strategy of the persistent memory namespace are accurately configured; when the PMEM virtual address space is registered to the RDMA network card, a specific registration algorithm is used to optimize the generation of the memory region descriptor to reduce the delay of the RDMA operation; when the standby node transmits the VA and Key of the MR to the primary node, multiple encryption technologies are used to encrypt the transmission data to ensure the absolute confidentiality and attack resistance of the secure channel;

[0096] Atomic dual-write flow submodule: Ceph MDS generates WAL log and writes it to the local NVMeSSD log file through the Rocksdb interface; the primary node uses RDMASend / Write operation to asynchronously write WAL log content to the specified virtual address of the standby node PMEM; an atomic confirmation protocol is used to ensure that the dual-write of the local NVMe and the remote PMEM is completed before a successful response is returned to the client;

[0097] Abnormality processing and degradation mechanism submodule: when the network is interrupted, the master node temporarily stores the unsynchronized WAL log in the local PMEM buffer, and supports breakpoint continuation after network recovery; when the standby node PMEM is unavailable, the master node switches to asynchronous replication mode, and synchronizes the WAL log to the standby node's NVMeSSD in batches at regular intervals;

[0098] (2) Metadata snapshot differential synchronization module, comprising:

[0099] Consistent snapshot generation submodule: the master node starts the snapshot process every fixed period, suspends new WAL log writing; lock the PMEM log area using the transactional API of PMDK, generate a consistent snapshot; record the snapshot timestamp, starting LSN, ending LSN and CRC64 checksum value, and store them in the PMEM header metadata area; when the master node starts the snapshot process, use an intelligent scheduling algorithm to dynamically adjust the snapshot period in combination with the current load of the system, business priority and historical snapshot generation time; when locking the PMEM log area using the transactional API of PMDK, set a flexible lock timeout and conflict resolution strategy to avoid system performance degradation due to long-term locking or lock conflicts; when recording snapshot metadata, use a distributed storage method to store the snapshot timestamp, starting LSN, ending LSN and CRC64 checksum value in the PMEM header metadata area of multiple nodes at the same time, to improve data reliability and availability;

[0100] Difference bitmap calculation and compression submodule: use the mmap interface of PMEM to compare the current snapshot with the previous snapshot byte by byte to generate a difference bitmap; apply sparse bitmap encoding and LZ4 algorithm to optimize and compress the difference bitmap; the master node transmits the compressed difference bitmap and its corresponding data block to the difference temporary storage area of the standby node PMEM through the RDMARead operation;

[0101] Incremental synchronization and merging submodule: the standby node parses the difference bitmap and writes the data according to the address into the corresponding area of PMEM; use the pmem_memcpy_persist function of PMDK to ensure persistence; update the local snapshot metadata to record the latest LSN and checksum value;

[0102] (3) Fault switching optimization module, comprising:

[0103] The PMEM state monitoring submodule: the master node periodically sends a heartbeat packet to the standby node, with the current maximum LSN value; the standby node polls the master node PMEM metadata area to verify log continuity; if the heartbeat is continuously lost, the master node is determined to be faulty, triggering the switching process. When the master node sends the heartbeat packet, an adaptive heartbeat interval adjustment mechanism is used to adjust the heartbeat interval in real time according to the network bandwidth, delay and system stability; when the standby node polls the master node PMEM metadata area, an incremental polling and caching technology is used to only obtain the metadata information that has changed since the last polling, and cache it locally, reducing network transmission and polling time; when the master node is determined to be faulty, a comprehensive judgment is made in combination with multiple monitoring indicators, in addition to the continuous loss of heartbeat, the system resource usage of the master node, network connection quality, disk I / O performance factors are also considered, to improve the accuracy and reliability of fault judgment.

[0104] The fast switching process submodule: perform log integrity check to ensure that the log is not damaged; the standby node calls the secondary instance of Rocksdb, mounts the PMEM log and applies the log to generate the latest metadata view; promote the standby node to the master node and take over the CephMDS service. When multiple backup nodes read logs from PMEM in parallel, a load balancing and task allocation algorithm is used to reasonably allocate log reading tasks according to the processing capacity, current load and network bandwidth of each node; when electing a new master through a distributed consensus protocol, an optimized election strategy is used to reduce communication overhead and time delay in the election process, and ensure that the new master can be quickly elected and take over the service; after the new master takes over the service, a data synchronization check and repair mechanism is used to comprehensively check the data between the new master and the standby node, and repair it in time when the data is inconsistent, to ensure the consistency and integrity of the data.

[0105] The parallel recovery mechanism submodule: when there are multiple backup nodes, each node reads logs from PMEM in parallel; a new master is elected through a distributed consensus protocol to speed up cluster recovery.

[0106] Although embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A Ceph metadata disaster recovery acceleration method based on persistent memory and RDMA network, characterized in that: Includes the following steps: Step 1: PMEM-WAL Dual-Write Mechanism Perform memory registration and virtual address mapping: initialize the PMEM memory pool, format the PMEM device as a persistent memory namespace and create a fixed-size virtual address space; Register the PMEM virtual address space to the RDMA network card and generate a memory region descriptor; the standby node transmits the VA and Key of the MR to the master node through a secure channel to establish a remote direct write channel; To implement an atomic dual-write process: CephMDS generates WAL logs and writes them to the log file of the local NVMe SSD through the Rocksdb interface; the master node uses RDMASend / Write operations to asynchronously write the WAL log content to the specified virtual address of the backup node PMEM; and only returns a success response to the client after the dual write to the local NVMe and remote PMEM is completed through an atomic confirmation protocol. Implement anomaly handling and degradation mechanisms: When the network is interrupted, the master node temporarily stores the unsynchronized WAL logs to the local PMEM buffer and supports breakpoint resume after the network is restored; when the standby node's PMEM is unavailable, the master node switches to asynchronous replication mode and periodically synchronizes the WAL logs to the standby node's NVMeSSD in batches. Step 2: Differential Synchronization of Metadata Snapshots Generate consistent snapshots: The master node starts the snapshot process at fixed intervals, pausing new WAL log writing; it uses the PMDK transactional API to lock the PMEM log area and generate consistent snapshots. Record the snapshot timestamp, start LSN, end LSN and CRC64 check value, and store them in the PMEM header metadata area; Calculate and compress the difference bitmap: Use PMEM's mmap interface to compare the current snapshot with the previous snapshot byte by byte to generate the difference bitmap; Sparse bitmap encoding and LZ4 algorithm are applied to optimize and compress the difference bitmap; the master node transmits the compressed difference bitmap and its corresponding data block to the difference buffer of the standby node PMEM through RDMARead operation; Perform incremental synchronization and merging: The backup node parses the difference bitmap and writes the data to the corresponding area of ​​PMEM according to the address; the pmem_memcpy_persist function of PMDK is used to ensure persistence; the local snapshot metadata is updated to record the latest LSN and checksum. Step 3: Fault Switching Optimization Perform PMEM status monitoring: The master node periodically sends heartbeat packets to the backup node, along with the current maximum LSN value; The standby node polls the primary node's PMEM metadata area in a round-robin fashion to verify log continuity; If heartbeats are lost consecutively, the master node is determined to be faulty, and a failover process is triggered. Perform a fast switchover process: verify log integrity to ensure that the logs are not corrupted; The standby node calls the secondary instance of Rocksdb, mounts the PMEM logs, and applies the logs to generate the latest metadata view; Promote the backup node to the primary node and take over the CephMDS service; Implement a parallel recovery mechanism: when multiple backup nodes exist, each node reads logs from PMEM in parallel; a new master is elected through a distributed consensus protocol to accelerate cluster recovery.

2. The Ceph metadata disaster recovery acceleration method based on persistent memory and RDMA network according to claim 1, characterized in that: During the memory registration and virtual address mapping process of the PMEM-WAL dual-write mechanism, when the PMEM memory pool is initialized, the size and attribute parameters of the persistent memory namespace are precisely configured according to the preset storage capacity and performance requirements. When registering the PMEM virtual address space to the RDMA network card, a specific registration strategy is adopted to ensure that the generation of memory region descriptors meets the high-efficiency transmission requirements of the RDMA network. When the standby node transmits the VA and Key of the MR to the master node, an encryption algorithm is used to encrypt the transmitted data to ensure the confidentiality and integrity of the secure channel.

3. The Ceph metadata disaster recovery acceleration method based on persistent memory and RDMA network according to claim 2, characterized in that: During the generation of consistent snapshots through differential synchronization of metadata snapshots, when the master node initiates the snapshot process, it employs an intelligent scheduling algorithm to dynamically adjust the snapshot cycle based on the current system load and business priorities. When using the PMDK transactional API to lock the PMEM log area, a reasonable lock timeout is set to avoid system performance degradation due to prolonged locking. When recording snapshot metadata, a redundant storage method is adopted, storing the snapshot timestamp, start LSN, end LSN, and CRC64 checksum in multiple locations within the PMEM header metadata area to improve data reliability.

4. The Ceph metadata disaster recovery acceleration method based on persistent memory and RDMA network according to claim 3, characterized in that: During the PMEM status monitoring process optimized for failover, when the master node sends heartbeat packets, an adaptive heartbeat interval adjustment mechanism is adopted to dynamically adjust the heartbeat interval time according to network conditions and system stability. When the standby node polls the master node's PMEM metadata area, an incremental polling method is adopted to obtain only the metadata information that has changed since the last poll, reducing network transmission volume and polling time. When determining the failure of the master node, a comprehensive judgment is made by combining multiple monitoring indicators. In addition to continuous loss of heartbeats, the system resource usage of the master node and network connection quality factors are also considered to improve the accuracy of fault judgment.

5. The Ceph metadata disaster recovery acceleration method based on persistent memory and RDMA network according to claim 4, characterized in that: In the parallel recovery mechanism, when multiple backup nodes read logs from PMEM in parallel, a load balancing algorithm is used to reasonably allocate log reading tasks based on the processing capacity and current load of each node. When electing a new master through the distributed consensus protocol, an optimized election strategy is adopted to reduce communication overhead and time delay during the election process, ensuring that a new master can be quickly elected and take over the service. After the new master takes over the service, a data synchronization and verification mechanism is adopted to fully verify the data between the new master and the backup nodes to ensure data consistency and integrity.

6. A system for accelerating Ceph metadata disaster recovery based on persistent memory and RDMA network as described in claim 5, characterized in that: include: (1) PMEM-WAL dual-write module, including: The memory registration and virtual address mapping submodule is responsible for initializing the PMEM memory pool, formatting the PMEM device into a persistent memory namespace and creating a fixed-size virtual address space; registering the PMEM virtual address space to the RDMA network card and generating memory region descriptors; and arranging for the standby node to transmit the VA and Key of the MR to the master node through a secure channel to establish a remote direct write channel. The atomic dual-write process submodule enables CephMDS to generate WAL logs and write them to the log file of the local NVMeSSD through the Rocksdb interface; the master node asynchronously writes the WAL log content to the specified virtual address of the backup node PMEM using RDMASend / Write operations; an atomic confirmation protocol is used to ensure that a success response is returned to the client only after the dual write between the local NVMe and the remote PMEM is completed. The exception handling and degradation mechanism submodule: When the network is interrupted, the master node temporarily stores the unsynchronized WAL logs to the local PMEM buffer and supports resume transmission after the network is restored; when the standby node's PMEM is unavailable, the master node switches to asynchronous replication mode and periodically synchronizes the WAL logs to the standby node's NVMeSSD in batches. (2) Metadata snapshot differential synchronization module, including: Consistent snapshot generation submodule: The master node starts the snapshot process at fixed intervals, pausing new WAL log writing; it uses the PMDK transactional API to lock the PMEM log area and generate a consistent snapshot; it records the snapshot timestamp, start LSN, end LSN and CRC64 check value and stores them in the PMEM header metadata area. The difference bitmap calculation and compression submodule uses the mmap interface of PMEM to compare the current snapshot with the previous snapshot byte by byte to generate a difference bitmap; it applies sparse bitmap encoding and LZ4 algorithm to optimize and compress the difference bitmap; the master node transmits the compressed difference bitmap and its corresponding data block to the difference buffer area of ​​the standby node PMEM through the RDMARead operation. Incremental synchronization and merging submodule: The backup node parses the difference bitmap and writes the data to the corresponding area of ​​PMEM according to the address; the pmem_memcpy_persist function of PMDK is used to ensure persistence; the local snapshot metadata is updated to record the latest LSN and checksum; (3) Fault switching optimization module, including: PMEM status monitoring submodule: The master node periodically sends heartbeat packets to the standby node, along with the current maximum LSN value; the standby node polls the master node's PMEM metadata area to verify log continuity; if heartbeats are continuously lost, the master node is determined to be faulty, triggering the failover process; Quickly switch to the process submodule: Perform log integrity verification to ensure that the logs are not corrupted; the standby node calls the secondary instance of Rocksdb, mounts the PMEM logs and applies the logs to generate the latest metadata view; promote the standby node to the primary node and take over the CephMDS service; Parallel recovery mechanism submodule: When multiple backup nodes exist, each node reads logs from PMEM in parallel; a new master is elected through a distributed consensus protocol to accelerate cluster recovery.

7. The system according to claim 6, characterized in that: In the memory registration and virtual address mapping submodule of the PMEM-WAL dual-write module, when the PMEM memory pool is initialized, the size, access permissions and caching strategy of the persistent memory namespace are precisely configured according to the system's preset storage capacity and performance indicators. When registering the PMEM virtual address space to the RDMA network card, a specific registration algorithm is used to optimize the generation of memory region descriptors in order to reduce the latency of RDMA operations. When the standby node transmits the VA and Key of the MR to the master node, multiple encryption technologies are used to encrypt the transmitted data to ensure the absolute confidentiality and anti-attack capability of the secure channel.

8. The system according to claim 7, characterized in that: In the consistent snapshot generation submodule of the metadata snapshot differential synchronization module, when the master node starts the snapshot process, an intelligent scheduling algorithm is adopted to dynamically adjust the snapshot cycle based on the current system load, business priority, and historical snapshot generation time. When using the PMDK transactional API to lock the PMEM log area, flexible lock timeout time and conflict resolution strategies are set to avoid system performance degradation due to long-term locking or lock conflicts. When recording snapshot metadata, a distributed storage method is adopted to store the snapshot timestamp, start LSN, end LSN, and CRC64 checksum in the PMEM header metadata area of ​​multiple nodes simultaneously, thereby improving data reliability and availability.

9. A system according to claim 8, characterized in that: In the PMEM status monitoring submodule of the fault switching optimization module, when the master node sends a heartbeat packet, an adaptive heartbeat interval adjustment mechanism is adopted to adjust the heartbeat interval time in real time according to network bandwidth, latency and system stability; when the standby node polls the master node's PMEM metadata area, incremental polling and caching technology is adopted to only obtain metadata information that has changed since the last polling and cache it locally to reduce network transmission volume and polling time. When determining a master node failure, a comprehensive judgment is made by combining multiple monitoring indicators. In addition to continuous loss of heartbeats, the system resource usage, network connection quality, and disk I / O performance of the master node are also considered to improve the accuracy and reliability of fault diagnosis.

10. A system according to claim 9, characterized in that: In the parallel recovery mechanism submodule of the fault switching optimization module, when multiple backup nodes read logs from PMEM in parallel, a load balancing and task allocation algorithm is adopted to reasonably allocate log reading tasks based on the processing capacity, current load, and network bandwidth of each node. When electing a new master through a distributed consensus protocol, an optimized election strategy is adopted, which combines the node's performance indicators, historical election records, and network topology to reduce communication overhead and time delay during the election process, ensuring that a new master can be quickly elected and take over the service. After the new master takes over the service, a data synchronization verification and repair mechanism is adopted to conduct a comprehensive verification of the data between the new master and the backup nodes. If data inconsistencies are found, they are repaired in a timely manner to ensure data consistency and integrity.

Images