Distributed database incremental backup method based on RDMA

Abstract

The invention provides a distributed database incremental backup method based on RDMA, and belongs to the technical field of distributed databases. Incremental data needing to be backed up are accurately positioned through an incremental data detection module of a database main node in combination with a dual recognition mechanism of data block check value comparison and transaction log LSN number marking, and the backup efficiency is improved. Resource waste of total data transmission is avoided; an RDMA reliable connection transmission mode is adopted, so that a backup node can directly access a registered memory area where incremental data of a main node is located, multiple rounds of data copying are omitted, and a database main node CPU does not need to participate in a transmission process; meanwhile, a continuity verification module based on an LSN number is deployed at a backup node, the integrity of received data can be verified in real time, and if data missing exists, rapid supplementary transmission is conducted through RDMA connection, and the integrity of backup data is guaranteed. And the backup task and the user service task have no resource competition, so that an efficient, low-consumption and reliable incremental backup solution is provided for the distributed database.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed databases, and in particular to a distributed database incremental backup method based on RDMA. BACKGROUND

[0002] With the rapid development of digital economy, distributed databases have become the core of data storage in core fields such as finance, the Internet, and government affairs, and the amount of data has shown exponential growth, which has put strict requirements on the efficiency, resource occupation, and reliability of data backup. However, the existing distributed database backup technology has the following bottlenecks:

[0003] 1. High resource consumption caused by redundant data transmission links

[0004] In the traditional backup process, data needs to go through multiple rounds of copying from "database user memory -> operating system kernel buffer -> network card driver buffer -> network card hardware buffer", and CPU is involved in data transfer and protocol processing at each round. Taking TB-level incremental data backup as an example, the CPU occupancy rate of a single master database node can reach 35%-50%, and the continuous occupancy time is more than 2 hours. During this period, it will compete with core business (such as financial transactions and user request processing) for CPU and memory bus resources, causing business response delay to increase by more than 30%, and in severe cases, even causing transaction timeout and request discard.

[0005] 2. TCP / IP protocol stack overhead exacerbates backup delay

[0006] The TCP / IP protocol stack needs to complete complex processes such as three-way handshake, flow control, and retransmission mechanism. In a 10 Gb / s network environment, the transmission delay of a single TCP connection can reach 10-20 ms, and as the data volume increases, the protocol stack processing overhead increases nonlinearly. In the financial industry, the daily backup window period is from 2-4 am, and if the incremental data is more than 5 TB, the traditional TCP / IP solution often cannot complete the backup due to high delay, and is forced to compress the business data retention period, increasing the risk of data loss. In the Internet industry, real-time backup requirements (such as backup every 10 minutes) are difficult to implement due to TCP / IP delay problems.

[0007] 3. Insufficient adaptability of existing RDMA application solutions

[0008] Although some technologies attempt to apply RDMA to database data transmission, there are obvious limitations: first, RDMA is only used as a "transmission tool to replace TCP / IP", without designing adaptive logic for the "data identification-transmission-verification" whole process of database incremental backup, still relying on traditional incremental identification algorithms (such as log scanning), which cancels the low-delay advantage of RDMA; second, there is a lack of end-to-end consistency guarantee mechanism, if data packet loss occurs during RDMA transmission, the upper database protocol needs to be retransmitted, increasing the complexity of backup; third, the problem of multi-node cooperation under distributed architecture is not considered, and when multiple master nodes transmit data to the backup node at the same time, RDMA connection conflict is easy to occur, reducing the backup efficiency. SUMMARY

[0009] In order to solve the above technical problems, the present application provides a distributed database incremental backup method based on RDMA, which simplifies the data transmission path through RDMA network, combines with the incremental data accurate identification mechanism, and realizes low-delay and low-resource-occupied backup.

[0010] The technical scheme of the present application is:

[0011] A distributed database incremental backup method based on RDMA, comprising:

[0012] 1) The database master node identifies the incremental data and generates an incremental data list;

[0013] 2) The master node and the backup node establish a connection through RDMA and transfer the control information of the memory where the incremental data is located;

[0014] 3) The backup node directly reads the incremental data of the master node through RDMA and writes it into the backup memory area;

[0015] 4) The backup node verifies the data consistency and merges the incremental data into the backup database.

[0016] Further,

[0017] The specific steps are as follows:

[0018] 1) The master node reads the historical check value list from the metadata node;

[0019] 2) Compare the current and historical check values, mark the difference block, extract the LSN range of the difference block, and generate an incremental list;

[0020] 3) The incremental list is synchronized to the metadata node, and the state is marked as "identification completed";

[0021] 4) The master node registers the difference block memory information to generate control data, and the master and backup nodes create an RDMA connection to transfer the incremental list;

[0022] 5) Backup node feedback connection success to metadata node;

[0023] 6) Backup node calls RDMA read master node difference block data, every 128KB check, failure then retransmission;

[0024] 7) Master and backup node real-time report transmission progress to metadata node;

[0025] 8) Check if LSN is complete, missing then retransmission;

[0026] 9) Check pass, incremental data write to backup database;

[0027] 10) Backup node synchronizes the latest check value to metadata node, state marker complete.

[0028] Further,

[0029] Incremental data recognition, specifically includes the following steps:

[0030] 1) Master node incremental data detector triggers recognition task according to preset period, first read the last period data block check value list and the last period LSN number maximum value from metadata node;

[0031] 2) Data block level check engine traverses the master node local data storage file, calculates the current check value according to 16KB data block granularity, compares with historical check value one by one, marks the difference data block, generates difference data block list;

[0032] 3) Log analysis engine parses transaction log, extracts the transaction log record corresponding to the difference data block, determines the LSN number range of each difference data block, and associates the difference data block list to generate structured incremental data list;

[0033] 4) Master node synchronizes incremental data list to metadata node, metadata node stores incremental data list copy and updates incremental backup task record.

[0034] RDMA connection establishment, specifically includes the following steps:

[0035] 1) Node RDMA sending module receives incremental data list, memory management submodule scans database buffer pool, locates the memory address range of difference data block, registers the memory area as RDMA readable and writable memory area, generates RDMA control data;

[0036] 2) Connection control submodule requests backup node information from metadata node, metadata node load scheduling module assigns target backup node according to current backup node load state and returns its information;

[0037] 3) Master node sends RDMA connection request to target backup node, backup node RDMA receiving module responds to the request, and both sides complete RDMA connection establishment. Master node transmits incremental data list through the connection;

[0038] 4) After the backup node receives the incremental data list, it is stored in the local cache, and the connection establishment success state is fed back to the metadata node.

[0039] RDMA direct data transmission

[0040] 1) Backup node RDMA receiving module based on RDMA control data, call RDMA read operation access master node memory address range, directly read master node difference data block memory data, master node CPU only participate in connection initialization, do not participate in data transmission operation in the process of data transmission;

[0041] 2) Transmission process adopts pipeline transmission mode, backup node receives the previous data block at the same time, initiates the next data block read request, reduces the transmission waiting time; At the same time, real-time verification sub module carries out CRC32 verification on every 128KB data received, if the verification fails, immediately send retransmission request to master node through RDMA;

[0042] 3) After receiving the retransmission request, the master node RDMA sending module schedules the retransmission data transmission preferentially;

[0043] 4) Metadata node receives the transmission progress report of master and backup nodes in real time, updates the transmission progress metadata, and supports breakpoint resume transmission after abnormal interruption.

[0044] Consistency check and data merging

[0045] 1) Backup node data consistency check module reads LSN number range in incremental data list, starts LSN continuity check engine, traverses LSN number of received data, checks whether there is missing, repetition or sequence error;

[0046] 2) If LSN is missing, generate retransmission request and send it to master node through RDMA. Master node responds to the request and transmits the missing data. After retransmission, recheck;

[0047] 3) If LSN sequence is complete, further verify transaction consistency to ensure that backup data meets the ACID characteristics of database transaction;

[0048] 4) After the check is passed, the backup node writes the incremental data block into the corresponding data storage file of the node, and updates the local LSN number record;

[0049] 5) Backup node synchronizes backup completion state and latest data block check value list to metadata node, and metadata node updates historical check value list to complete this backup process.

[0050] Still further,

[0051] The master node is the source of the incremental data generation and deployment of the incremental data detection module and the RDMA sending module, which are linked through the internal data bus:

[0052] a. The incremental data detection module includes a two-dimensional recognition engine; one is a storage block verification engine based on SHA-256, which traverses the database user data file, calculates the verification value of each data block, and compares it with the historical verification value stored by the metadata node to accurately locate the differential data block; the second is a log analysis engine that extracts the transaction log corresponding to the differential data block, records the LSN number range, and generates a structured incremental data list containing "data block ID, LSN interval, data block size, and storage address";

[0053] b. The RDMA sending module includes a memory management submodule and a connection control submodule; the memory management submodule can dynamically identify the memory address of the differential data block in the database buffer pool, register it as an RDMA accessible area, and generate RDMA control data containing the local key L_Key, the remote key R_Key, and the memory length; the connection control submodule supports the RDMA reliable connection mode, can automatically detect the online state of the backup node, initiate a connection request and complete the transmission of the incremental data list, and has a connection exception reconnection mechanism.

[0054] The backup node is the data receiving and storage end, which deploys the RDMA receiving module and the data consistency verification module;

[0055] a. The RDMA receiving module includes a data receiving submodule and a real-time verification submodule; the data receiving submodule is based on the RDMA control data transmitted by the master node, calls the RDMA read operation to directly read the master node differential data block memory without CPU intervention, and the data receiving rate can match the physical bandwidth of the RDMA network card; the real-time verification submodule uses a block CRC verification mechanism, adds a 4-byte CRC32 checksum to each 128KB received data, calculates the checksum immediately after receiving and compares it with the checksum transmitted by the master node, and if they are inconsistent, triggers an immediate retransmission request;

[0056] b. The data consistency verification module is based on the LSN number range in the incremental data list to construct an LSN sequence integrity detection module to check whether there are missing, repeated or sequence disorder in the received data LSN number; at the same time, it is associated with the database transaction logic to verify the transaction integrity corresponding to the LSN;

[0057] The metadata management node is used for collaborative scheduling and data storage, and is deployed as the system coordination center, deploying the metadata storage module and the load scheduling module:

[0058] a, metadata storage module: store three kinds of core metadata: one is the list of historical data block check value, for the master node incremental identification comparison; Two is the incremental data list copy, for backup node check reference; Three is the system running state data, support the whole system state visualization monitoring;

[0059] b, load scheduling module: design dynamic load balancing algorithm for multi-master multi-backup scene, real-time monitoring of CPU occupancy, memory usage, network bandwidth utilization of each backup node, when multiple master nodes initiate backup request at the same time, automatically allocate backup node resources; At the same time support RDMA connection conflict detection and avoidance, when detecting that several master nodes initiate connection to the same backup node, automatically adjust the connection establishment timing, reduce the transmission delay caused by connection conflict.

[0060] The beneficial effects of the present application are

[0061] 1. Incremental identification: identify incremental data fast through block-level check engine, reduce invalid data transmission volume;

[0062] Transmission resource consumption is reduced: realize zero-intervention transmission of master node CPU, master node CPU only occupies very little resource in connection initialization stage, CPU occupancy rate tends to 0 in data transmission stage, compared with traditional TCP / IP transmission scheme, master node CPU occupancy rate is significantly reduced;

[0063] 2. Consistency guarantee dimension expansion: two-dimensional check of data transmission integrity and transaction logic consistency, backup data consistency pass rate reaches 100%, guarantee the integrity of backup data;

[0064] Multi-node cooperation efficiency optimization: through metadata node load scheduling, backup task allocation efficiency is significantly improved in multi-master multi-backup scene, reduce RDMA connection conflict rate, avoid transmission delay caused by single backup node overload. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 It is the work flow diagram of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical scheme and advantages of the embodiments of the present application more clear, the technical scheme in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application.

[0067] The application provides an RDMA-based distributed database incremental backup method, and constructs a scheme of "incremental data accurate identification-RDMA direct transmission-end-to-end consistency check". Specifically, the application accurately locates the incremental data to be backed up through the incremental data detection module of the database master node, in combination with the dual identification mechanism of "data block checksum comparison + transaction log LSN number marking", to avoid resource waste of full data transmission; the RDMA reliable connection transmission mode is adopted, so that the backup node can directly access the registered memory area of the master node incremental data, and the multi-round data copying of "database memory -> kernel buffer -> network card buffer" is saved, and the CPU of the database master node is not required to participate in the transmission process, so that the system CPU resource occupation is reduced; at the same time, the continuity check module based on the LSN number is deployed on the backup node, so that the integrity of the received data can be verified in real time, and if there is data loss, the backup node can quickly retransmit through the RDMA connection, so that the integrity of the backup data is ensured. The backup task does not compete for resources with the user business task, and the distributed database is provided with an efficient, low-cost and reliable incremental backup solution.

[0068] I. The distributed database incremental backup system designed by the application adopts a three-layer architecture of "master, backup and metadata coordination".

[0069] 1. Database master node (core function: incremental identification and RDMA connection initialization)

[0070] The master node is the generation and source end of the incremental data, and is provided with an incremental data detection module and an RDMA sending module, which are linked through an internal data bus:

[0071] a. The incremental data detection module includes a dual-dimension identification engine. One is a storage block check engine based on SHA-256, which traverses the database user data file, calculates the check value of each data block (default 16 KB), compares the check value with the historical check value stored by the metadata node, and accurately locates the difference data block; the second is a log analysis engine, which extracts the transaction log corresponding to the difference data block, records the LSN (log sequence number) number range, generates a structured incremental data list containing "data block ID, LSN interval, data block size, storage address", and ensures the association of the incremental data and the transaction context.

[0072] b. RDMA sending module: including memory management submodule and connection control submodule. The memory management submodule can dynamically identify the memory address of the differential data block in the database buffer pool, register it as an RDMA accessible area, and generate RDMA control data containing local key (L_Key), remote key (R_Key) and memory length; the connection control submodule supports RDMA reliable connection mode, can automatically detect the online state of the backup node, initiate connection request and complete the transmission of the incremental data list, and has connection exception reconnection mechanism (reconnection interval <100ms) to ensure transmission link stability.

[0073] 2. Database backup node (core function: RDMA receiving and consistency check)

[0074] The backup node is the data receiving and storage end, which deploys RDMA receiving module and data consistency check module.

[0075] a. RDMA receiving module: including data receiving submodule and real-time check submodule. The data receiving submodule reads the memory of the differential data block of the master node directly based on the RDMA control data transmitted by the master node, without the intervention of the CPU of the master node, and the data receiving rate can match the physical bandwidth of the RDMA card. The real-time check submodule uses block CRC check mechanism, adds 4-byte CRC32 checksum to each 128KB received data, calculates the checksum immediately after receiving and compares it with the checksum transmitted by the master node, if they are inconsistent, triggers immediate retransmission request, avoids invalid data accumulation.

[0076] b. Data consistency check module: based on the LSN number range in the incremental data list, construct LSN sequence integrity detection module, traverse the LSN number of received data to check whether there is missing, repetition or sequence disorder; at the same time, associate with database transaction logic, verify the transaction integrity of LSN (such as whether the start LSN and end LSN match), ensure that the backup data is not only physically complete, but also meets the requirements of database transaction consistency.

[0077] 3. Metadata management node (core function: cooperative scheduling and data storage)

[0078] The metadata node is the system coordination center, which deploys metadata storage module and load scheduling module, breaks the bottleneck of traditional distributed backup "no centralized coordination":

[0079] a. Metadata Storage Module: Primarily stores three types of core metadata: First, a list of historical data block checksums (including SHA-256 values, data block IDs, and update timestamps) for incremental identification and comparison by the master node; second, a copy of the incremental data list (synchronized with the master node) for backup node verification reference; and third, system operation status data (such as master / backup node connection status, transmission progress, and verification results), supporting visualized monitoring of the entire system status.

[0080] b. Load Scheduling Module: Designed for multi-master, multi-backup scenarios, a dynamic load balancing algorithm is implemented to monitor the CPU utilization, memory usage, and network bandwidth utilization of each backup node in real time. When multiple master nodes initiate backup requests simultaneously, the module automatically allocates backup node resources (e.g., assigning the backup task of master node A to backup node 1, and the task of master node B to backup node 2) to avoid overloading a single backup node. It also supports RDMA connection conflict detection and avoidance. When multiple master nodes are detected to initiate connections to the same backup node, the module automatically adjusts the connection establishment sequence to reduce transmission delays caused by connection conflicts.

[0081] II. Incremental Backup Process

[0082] Based on the above system architecture, the implementation steps of the distributed database incremental backup method of the present invention are as follows:

[0083] Step 1: Incremental Data Identification

[0084] 1.1 The master node incremental data detector triggers the identification task according to a preset period (configurable, default 5 minutes). First, it reads the list of data block verification values ​​of the previous period and the maximum value of LSN number of the previous period from the metadata node.

[0085] 1.2 The data block-level verification engine traverses the local data storage files of the master node, calculates the current verification value (SHA-256) at a granularity of 16KB data blocks, compares it with the historical verification values ​​one by one, marks the difference data blocks (blocks with different verification values), and generates a list of difference data blocks.

[0086] 1.3 The log parsing engine parses the transaction log, extracts the transaction log records corresponding to the differential data blocks, determines the LSN range of each differential data block (e.g., LSN = 10000-15000 for data block X), and generates a structured incremental data list by associating the differential data block list.

[0087] 1.4 The master node synchronizes the incremental data list to the metadata node, and the metadata node stores a copy of the incremental data list and updates the incremental backup task record.

[0088] Step 2: RDMA Connection Establishment

[0089] 2.1 After the master node RDMA sending module receives the incremental data list, the memory management submodule scans the database buffer pool, locates the memory address range where the difference data block is located, registers the memory region as an RDMA readable and writable memory region, and generates RDMA control data such as L_Key, R_Key and memory length.

[0090] 2.2 The connection control submodule requests backup node information (such as IP address and RDMA port number) from the metadata node. The metadata node load scheduling module allocates the target backup node and returns its information based on the current backup node load status.

[0091] 2.3 The master node sends an RDMA connection request to the target backup node. The backup node's RDMA receiving module responds to the request, and the two parties complete the RDMA connection establishment. The master node then transmits the incremental data list through the connection.

[0092] 2.4 After receiving the incremental data list, the backup node stores it in the local cache and sends a successful connection establishment status back to the metadata node.

[0093] Step 3: RDMA Direct Data Transfer

[0094] 3.1 The backup node RDMA receiving module uses RDMA control data to call RDMA read operations to access the master node's memory address range and directly read the master node's differential data block memory data. During data transmission, the master node CPU only participates in connection initialization and does not participate in data transmission operations.

[0095] 3.2 The transmission process adopts a pipeline transmission mode. While receiving the previous data block, the backup node initiates a read request for the next data block to reduce the transmission waiting time. At the same time, the real-time verification submodule performs CRC32 verification on every 128KB of received data. If the verification fails, it immediately sends a retransmission request (carrying the data block ID and offset) to the master node via RDMA.

[0096] 3.3 After the master node receives the retransmission request, the RDMA sending module prioritizes the transmission of retransmission data to ensure that the retransmission data arrives before the normal data, thus avoiding the backup process being blocked due to retransmission delay.

[0097] 3.4 The metadata node receives real-time transmission progress reports (such as the amount of data transmitted and the amount of data remaining) from the primary and backup nodes, updates the transmission progress metadata, and supports resuming transmission from the breakpoint after abnormal interruption.

[0098] Step 4: Consistency Check and Data Merging

[0099] 4.1 The backup node data consistency verification module reads the LSN number range in the incremental data list, starts the LSN continuity verification engine, traverses the LSN numbers of the received data, and checks whether there are any missing, duplicate or sequential errors.

[0100] 4.2 If a missing LSN is detected (e.g., LSN=12000 not received), a retransmission request (including the missing LSN number and the corresponding data block ID) is generated and sent to the master node via RDMA. The master node responds to the request and transmits the missing data. After retransmission, the data is re-verified.

[0101] 4.3 If the LSN sequence is complete, further verify the transaction consistency (e.g., check whether the transaction start LSN=10000 and end LSN=15000 match, and whether the intermediate LSNs are consecutive) to ensure that the backup data conforms to the ACID properties of database transactions.

[0102] 4.4 After the verification is successful, the backup node writes the incremental data block to the data storage file corresponding to this node, and at the same time updates the local LSN number record (updates the current maximum LSN value to the LSN end value in the incremental data list).

[0103] 4.5 The backup node synchronizes the backup completion status and the latest data block checksum list to the metadata node, and the metadata node updates the historical checksum list to complete the backup process.

[0104] The above description is merely a preferred embodiment of the present invention and is used only to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A distributed database incremental backup method based on RDMA, characterized in that, include: 1) The database master node identifies incremental data and generates an incremental data list; 2) The master node and the backup node establish a connection via RDMA to transfer control information of the memory where the incremental data is located; 3) Backup nodes directly read incremental data from the master node via RDMA and write it to the backup memory area; 4) Verify data consistency at backup nodes and merge incremental data into the backup database.

2. The method according to claim 1, characterized in that, The specific steps are as follows: 1) The master node reads the list of historical verification values ​​from the metadata node; 2) Compare the current and historical check values, mark the difference blocks, extract the LSN range of the difference blocks, and generate an incremental list; 3) The incremental list is synchronized to the metadata node, and the status is marked "Identification Complete"; 4) The master node registers the differential block memory information to generate control data, and the master and standby nodes create an RDMA connection to transfer the incremental list; 5) The backup node reports a successful connection to the metadata node; 6) The backup node calls RDMA to read the differential block data of the master node, and verifies it every 128KB. If it fails, it retransmits the data. 7) The primary and backup nodes report the transmission progress to the metadata node in real time; 8) Check if the LSN is complete; if missing, retransmit it. 9) If the verification passes, the incremental data is written to the backup database; 10) The backup node synchronizes the latest checksum to the metadata node, and the status marking is complete.

3. The method according to claim 2, characterized in that, Incremental data identification specifically includes the following steps: 1) The master node incremental data detector triggers the identification task according to the preset cycle. First, it reads the list of data block verification values ​​of the previous cycle and the maximum value of LSN number of the previous cycle from the metadata node. 2) The data block-level verification engine traverses the local data storage files of the master node, calculates the current verification value at a granularity of 16KB data blocks, compares it with the historical verification values ​​one by one, marks the difference data blocks, and generates a list of difference data blocks. 3) The log parsing engine parses the transaction log, extracts the transaction log records corresponding to the difference data blocks, determines the LSN range of each difference data block, and generates a structured incremental data list by associating the difference data block list. 4) The master node synchronizes the incremental data list to the metadata node, and the metadata node stores a copy of the incremental data list and updates the incremental backup task record.

4. The method according to claim 2, characterized in that, The establishment of an RDMA connection includes the following steps: 1) After the node RDMA sending module receives the incremental data list, the memory management submodule scans the database buffer pool, locates the memory address range where the difference data block is located, registers the memory region as an RDMA readable and writable memory region, and generates RDMA control data. 2) The connection control submodule requests backup node information from the metadata node. The metadata node load scheduling module allocates target backup nodes and returns their information based on the current backup node load status. 3) The master node sends an RDMA connection request to the target backup node, the backup node's RDMA receiving module responds to the request, the two parties complete the RDMA connection establishment, and the master node transmits the incremental data list through the connection; 4) After receiving the incremental data list, the backup node stores it in the local cache and sends a successful connection establishment status back to the metadata node.

5. The method according to claim 2, characterized in that, RDMA direct data transfer 1) The backup node RDMA receiving module uses RDMA control data to call RDMA read operation to access the master node's memory address range and directly read the master node's differential data block memory data. During the data transmission process, the master node CPU only participates in connection initialization and does not participate in data transmission operations. 2) The transmission process adopts a pipeline transmission mode. While receiving the previous data block, the backup node initiates a read request for the next data block, reducing the transmission waiting time. At the same time, the real-time verification submodule performs CRC32 verification on every 128KB of received data. If the verification fails, it immediately sends a retransmission request to the master node through RDMA. 3) After the master node receives the retransmission request, the RDMA transmission module prioritizes the transmission of retransmission data; 4) The metadata node receives the transmission progress report from the primary and backup nodes in real time, updates the transmission progress metadata, and supports resuming transmission from the breakpoint after abnormal interruption.

6. The method according to claim 2, characterized in that, Consistency verification and data merging 1) The backup node data consistency verification module reads the LSN number range in the incremental data list, starts the LSN continuity verification engine, traverses the LSN numbers of the received data, and checks whether there are any missing, duplicate or sequential errors. 2) If an LSN is detected to be missing, a retransmission request is generated and sent to the master node via RDMA. The master node responds to the request and transmits the missing data. After retransmission, the data is re-verified. 3) If the LSN sequence is complete, further verify transaction consistency to ensure that the backup data conforms to the ACID properties of database transactions; 4) After the verification is successful, the backup node writes the incremental data block to the data storage file corresponding to this node, and updates the local LSN record at the same time. 5) The backup node synchronizes the backup completion status and the latest data block checksum list to the metadata node, and the metadata node updates the historical checksum list to complete the backup process.

7. The method according to claim 2, characterized in that, The master node, acting as the generator and source of incremental data, deploys an incremental data detection module and an RDMA transmission module, which work together via an internal data bus. a. Incremental Data Detection Module: Includes a dual-dimensional recognition engine; the first is a storage block verification engine based on SHA-256, which traverses the database user data files, calculates the verification value of each data block, compares it with the historical verification values ​​stored in the metadata node, and locates the discrepancy data blocks; the second is a log parsing engine, which extracts the transaction logs corresponding to the discrepancy data blocks, records the LSN number range, and generates a structured incremental data list containing "data block ID, LSN range, data block size, and storage address"; b. RDMA Transmit Module: Includes a memory management submodule and a connection control submodule; The memory management submodule can dynamically identify the memory addresses of differential data blocks in the database buffer pool, register them as RDMA-accessible areas, and generate RDMA control data containing the local key L_Key, the remote key R_Key, and the memory length. The connection control submodule supports RDMA reliable connection mode, can automatically detect the online status of backup nodes, initiate connection requests and complete the transmission of incremental data lists, and also has a connection reconnection mechanism in case of connection failure.

8. The method according to claim 7, characterized in that, The backup node serves as the data receiving and storage end, and is equipped with an RDMA receiving module and a data consistency verification module. a. RDMA receiver module: includes a data receiving submodule and a real-time verification submodule; The data receiving submodule directly reads the master node's differential data block memory by calling the RDMA read operation based on the RDMA control data transmitted by the master node, and the data receiving rate matches the physical bandwidth of the RDMA network card. The real-time verification submodule adopts a block CRC check mechanism, which adds a 4-byte CRC32 checksum to every 128KB of received data. After receiving the data, it immediately calculates the checksum value and compares it with the checksum transmitted by the master node. If they do not match, it triggers an immediate retransmission request. b. Data Consistency Verification Module: Based on the LSN number range in the incremental data list, an LSN sequence integrity detection module is constructed to traverse the LSN numbers of the received data to check for missing, duplicate, or disordered sequences; at the same time, it is associated with the database transaction logic to verify the transaction integrity corresponding to the LSN.

9. The method according to claim 8, characterized in that, The metadata management node is used to coordinate scheduling and data storage. As the central hub of system coordination, it deploys metadata storage modules and load scheduling modules. a. Metadata storage module: Stores three types of core metadata: First, a list of historical data block check values, for incremental identification and comparison by the master node; Second, an incremental data list copy is provided for backup nodes to verify and reference. Thirdly, it provides system operation status data, supporting visualized monitoring of the entire system status; b. Load scheduling module: Designed for multi-master and multi-standby scenarios, a dynamic load balancing algorithm is used to monitor the CPU utilization, memory utilization, and network bandwidth utilization of each backup node in real time. When multiple master nodes initiate backup requests at the same time, the backup node resources are automatically allocated. It also supports RDMA connection conflict detection and avoidance. When several master nodes are detected to initiate connections to the same backup node, the connection establishment timing is automatically adjusted to reduce transmission delays caused by connection conflicts.

Images