Data backup method, master node, target backup node and medium
By parsing the key-value pair data file on the master node and directly routing data information to the target standby node using the partition table, the problem of low efficiency in reading and writing operation logs in traditional methods is solved, achieving data consistency and efficient backup between master and standby nodes.
Patent Information
- Application Number
- CN202512026854.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-04-10
AI Technical Summary
In traditional distributed database master-slave replication methods, reading and writing operation logs are inefficient, affecting the overall efficiency of master-slave replication. Furthermore, operation logs occupy storage space, leading to the risk of insufficient disk space during backup.
When writing a key-value pair data file to disk from the master node's memory, the data file is parsed to extract the data information of the write request, and the target backup node is determined based on the partition table of the shared storage medium. The data information is directly routed without the need to record operation logs, thus reducing the frequency of disk read/write operations and storage space usage.
It improves the efficiency of cross-node data backup, ensures data consistency between primary and backup nodes, reduces system storage resource overhead, avoids backup interruption, and enhances the stability and efficiency of the backup process.
Smart Images
Figure CN121833349A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of distributed database, and particularly relates to a data backup method, a master node, a target backup node and a medium. BACKGROUND
[0002] The master-slave replication of the distributed database refers to that all write request operations performed by a master node in a distributed system can be recorded and executed by a backup node, so that the write request operations and data saved by the master node and the backup node are consistent, and the data loss or service interruption caused by the failure of the master node can be prevented.
[0003] The traditional master-slave replication method of the distributed database is as follows: after receiving and executing a write request, the master node saves an operation log recording the write request to a local storage medium, and then sends the operation log to the backup node, and the backup node receives and saves the operation log and executes the write request in the operation log to complete data synchronization.
[0004] This method has the problem of low efficiency of read-write operation log, which further affects the overall efficiency of the master-slave replication. SUMMARY
[0005] The present application aims to solve one of the problems in the related art to some extent. To this end, the present application provides a data backup method, a master node for executing the data backup method, a target backup node for executing the data backup method and a computer readable medium, which have the advantages and characteristics of ensuring the consistency of data of the master node and the target backup node and improving the data backup efficiency.
[0006] In order to achieve the above-mentioned purpose, as a first aspect of the present application, a data backup method for a master node of a distributed database is provided, wherein the data backup method comprises: In the case that it is detected that the master node writes the data file of the key-value pair data structure into the disk, the data file is parsed according to the key-value pair data structure to obtain data information carried by a plurality of write requests; The target backup node of the data information is determined according to the parsed data information and a partition table; wherein the partition table comprises a mapping relationship between the hash value of the data information and the target backup node and is located in a shared storage medium; the shared storage medium can be accessed by the master node and the target backup node; The parsed data information is sent to the target backup node.
[0007] Optionally, the file name of the data file carries a write time; In a case where it is detected that the memory of the master node writes a data file of a key-value pair data structure into a disk, the data file is parsed according to the key-value pair data structure, and data information carried by a plurality of write requests is obtained, including: In a case where it is detected that the memory of the master node writes a data file of a key-value pair data structure into a disk, the data file of the earliest write time is determined according to the file name of the data file; The data file of the earliest write time is parsed according to the key-value pair data structure, and data information carried by a write request corresponding to the data file of the earliest write time is obtained.
[0008] Optionally, the data file of the earliest write time is determined according to the file name of the data file, including: A user of a CPU basic computing resource is allocated to each data file; The user and the write time of the corresponding data file are updated to the master node; The write time is sorted, and the data file of the earliest write time and the corresponding user are obtained; The data file of the earliest write time is parsed according to the determined user and the corresponding basic computing resource, and data information carried by a write request corresponding to the data file of the earliest write time is obtained.
[0009] Optionally, the data information includes data identification and data content; and the target standby node of the data information is determined according to the parsed data information and a partition table, including: The data identification is subjected to a hash operation to obtain a hash value; The target standby node of the data identification and the data content is determined according to the hash value and the partition table; The parsed data information is sent to the target standby node, including: The data information and the data content are encapsulated into a backup request and then sent to the target standby node.
[0010] Optionally, the data content includes business data and / or a preset deletion identifier.
[0011] As a second aspect of the present application, a data backup method is provided for a target standby node of a distributed database, wherein the data backup method includes: Data information sent by a master node is received; The data information is checked; In case of passing the verification, the data information is written into a data file according to a key-value pair data structure; The data file is saved into a disk of the target backup node.
[0012] Optionally, the data information received by the master node comprises: receiving a backup request; parsing the backup request to obtain data information, wherein the data information comprises a data identifier and data content; The verification of the data information comprises: performing a hash operation on the data identifier to obtain a hash value; determining a target backup node of the data identifier and the data content according to the hash value and a partition table; in case that the identifier of the target backup node is consistent with the determined identifier of the target backup node, the verification is passed.
[0013] As a third aspect of the present application, a master node is provided, comprising: one or more processors; a memory having one or more computer programs stored thereon, when the one or more computer programs are executed by the one or more processors, the one or more processors implement the data backup method according to the first aspect of the present application.
[0014] As a fourth aspect of the present application, a target backup node is provided, comprising: one or more processors; a memory having one or more computer programs stored thereon, when the one or more computer programs are executed by the one or more processors, the one or more processors implement the data backup method according to the second aspect of the present application.
[0015] In addition, as a fifth aspect of the present application, a computer readable medium having a computer program stored thereon is provided, wherein the computer program is executed by a processor to implement the data backup method according to the first aspect and the second aspect of the present application.
[0016] The data backup method provided by the application takes the data file of the key-value pair data structure written by the main node memory to the disk as the trigger condition of the main node itself persistent data and cross-node backup, parses the data file according to the key-value pair data structure, and extracts the data information corresponding to a plurality of write requests; according to the data information extracted by parsing, in combination with the partition table (which records the mapping relationship between the data information hash value and the target backup node, and can be accessed by each node) in the shared storage medium, the target backup node corresponding to the data information is determined and the data information is routed; after the target backup node writes the received data information into the data file according to the key-value pair data structure, the data file consistent with the main node data structure and content will be formed in the disk. This data backup method is essentially different from the traditional scheme relying on the log of the write operation. Because the data information required for backup can be directly parsed according to the key-value pair data structure, there is no need to record the operation log, the double write operation of the main node write operation and the write log is reduced, the read-write frequency of the disk is reduced, and the backup efficiency is improved; at the same time, since the operation log is omitted, the storage space occupied by the operation log is additionally saved, further releasing the disk resources, avoiding the backup interruption or lag caused by insufficient space or IO competition, and ensuring the stable and efficient backup process. By avoiding the redundant read-write operation caused by the operation log, in combination with the partition table to ensure the accurate matching of the target backup node, and finally routing the data information saved by the main node, the data consistency between the main backup node and the backup node is realized, the data backup efficiency between the cross nodes is improved, and the advantages and characteristics of ensuring the data consistency of the main backup node and the efficient and complete backup of the cross node data are realized.
[0017] The features and advantages of the present application will be described in detail in the following detailed description and accompanying drawings. The best mode or means of the present application will be fully described in conjunction with the accompanying drawings, but it is not a limitation on the technical solutions of the present application. In addition, the features, elements and components appearing in each of the following text and drawings are multiple, and different symbols or numbers are marked for convenience of representation, but all represent the same or similar structure or function parts. BRIEF DESCRIPTION OF DRAWINGS
[0018] The present application will be further described below in conjunction with the accompanying drawings: Figure 1 A flowchart of a data backup method provided by the present application; Figure 2 A flowchart of one embodiment of the data backup method step S110 provided by the present application; Figure 3 A flowchart of the data backup method provided by the present application for determining the data file with the earliest write time and parsing the data file; Figure 4 A flowchart of one embodiment of the data backup method steps S120 and S130 provided by the present application; Figure 5 The detailed flow chart of the data backup method in the main node provided by the present application is realized internally; Figure 6 The event loop flow chart corresponding to the main thread of the main node in the data backup method provided by the present application is realized internally; Figure 7 The event loop flow chart corresponding to the slave thread of the main node in the data backup method provided by the present application is realized internally; Figure 8 The flow chart of the data backup method realized by the target backup node provided by the present application is realized internally; Figure 9 Another step flow chart of the data backup method realized by the target backup node provided by the present application is realized internally; Figure 10 The flow chart of the data backup method realized by the target backup node provided by the present application is realized internally; Figure 11 The module chart of the main node provided by the present application is realized internally; Figure 12 The module chart of the target backup node provided by the present application is realized internally; Figure 13 The schematic diagram of the computer readable medium provided by the present application is realized internally.
[0019] Explanation of reference signs Among them, 101, the processor of the main node; 102, the memory of the main node; 103, the I / O interface of the main node; 104, the bus of the main node; 201, the processor of the target backup node; 202, the memory of the target backup node; 203, the I / O interface of the target backup node; 204, the bus of the target backup node. DETAILED DESCRIPTION
[0020] The embodiments of the present application are described in detail below, and the examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements with the same or similar functions throughout. Based on the embodiments in the embodiments, it is intended to explain the present application, and cannot be understood as a limitation of the present application.
[0021] In this specification, "one embodiment" or "an example" or "an example" means that the specific features, structures or characteristics described in connection with the embodiment itself can be included in at least one embodiment of the present disclosure. The appearance of the phrase "in one embodiment" at various places in the specification does not necessarily mean the same embodiment.
[0022] Traditional master-slave replication is a cross-node data backup method based on operation log. In this traditional data backup process, after the master node receives a write request, it first writes the data information of the write request into the disk to complete local data persistence, then writes the write request into the operation log of the disk to ensure that the operation log is consistent with the executed write request, and finally sends the operation log to the standby node in an asynchronous manner; after the standby node receives the operation log, it first writes the operation log into the disk of the standby node, then reads the operation log in the disk, and finally writes the data information of the write request in the operation log into the disk to complete data backup. This traditional method takes the operation log as the carrier, and the master node needs to complete multiple disk write operations, and the target standby node needs to complete multiple read-write disk operations; multiple interactions with the disk slow down the backup speed of the entire process, which is the main reason for the low efficiency of cross-node data backup. At the same time, an operation log often records multiple write requests, and each write operation log needs to consume disk IO resources and compete with the execution process of writing data information for IO resources, resulting in an increase in the time consumption of executing write request operations for adding, deleting, and modifying data information, which indirectly affects the performance of the master node. In addition, the master and standby nodes need to allocate additional storage space for the operation log, and the file capacity of the operation log increases with the number of write requests. When the operation log excessively occupies the disk space, it may cause insufficient disk space, which poses a risk of data backup failure.
[0023] Therefore, in order to solve the problem of low efficiency of cross-node data backup, as a first aspect of the present application, a data backup method is provided for a master node of a distributed database, as shown in Figure 1 The data backup method comprises: In step S110, when it is detected that the memory of the master node writes a data file of a key-value pair data structure into the disk, the data file is parsed according to the key-value pair data structure to obtain data information carried by multiple write requests; In step S120, the target standby node of the data information is determined according to the parsed data information and a partition table; wherein the partition table comprises a mapping relationship between the hash value of the data information and the target standby node and is located in a shared storage medium; the shared storage medium can be accessed by the master node and the target standby node; In step S130, the parsed data information is sent to the target standby node.
[0024] Unlike traditional cross-node backup methods for distributed databases, this invention does not rely on operation logs. When the master node is detected persisting data files to the local disk, it serves as the trigger condition for cross-node backup. The data file is parsed to obtain data information carrying multiple write requests. Based on the data information and the partition table of the shared storage medium, the target backup node with the data information is determined, and the data information is routed to the target backup node.
[0025] In the actual backup process, when a write request initiated by the client is received by the master node, the master node writes the data information in the write request into the data file in memory according to the key-value pair data structure. When the data file in memory begins to be flushed to disk, it is considered that the master node has begun to persist the data file. At the same time, the disk flushing is used as a trigger condition for cross-node backup, and the data file is sent to the target standby node to ensure efficient backup. As a persistent storage medium, the data files stored on the disk are not easily tampered with or lost. Therefore, the data file flushing to disk, as a trigger condition for master node persistence and cross-node backup, can ensure that the data file to be backed up has been reliably saved to the master node, and can also reduce backup time, ensuring efficient and reliable backup. It should be noted that the data files in different storage media (such as memory and disk) on the master node are consistent (such as all being immutable key-value pair data structures with consistent data information). Data files that store this immutable key-value pair data structure on disk will not change the data content in the data file due to write operations such as adding, deleting, and updating. Compared to traditional B+ multi-way balanced search tree data structures (where write operations directly modify data content), immutable key-value pair data structures further guarantee the consistency of data files (data structure and data content) on the disks of the primary and backup nodes, ensuring backup accuracy. It's important to reiterate that, to guarantee data consistency between the primary and backup nodes, the backup process can be completed only after the primary node's local disk has successfully saved the data file (after a successful disk flush).
[0026] In order to parse the data file in an orderly manner, the filename of the data file carries the write time; as an optional implementation of step S110, such as Figure 2 As shown, when it is detected that the master node is writing a key-value pair data file to the disk, the data file is parsed according to the key-value pair data structure to obtain data information carried by multiple write requests, including: In step S111, if it is detected that the master node's memory is writing a key-value pair data file to the disk, the data file with the earliest writing time is determined according to the file name of the data file; In step S112, the data file with the earliest write time is parsed according to the key-value pair data structure to obtain the data information carried by the write request corresponding to the data file with the earliest write time.
[0027] It should be noted that the write time can be the accurate system time or a data number that is positively correlated with the write time, such as file 1, file 2, and file 3. As an optional implementation of step S111, the specific process for determining the earliest write time is as follows: Figure 3 As shown, determining the data file with the earliest write time based on the filename of the data file includes: In step S111a, users are assigned CPU basic computing resources to each data file; In step S111b, the write time of the user and the corresponding data file is updated to the master node; In step S111c, the data files with the earliest write times and their corresponding users are obtained by sorting them according to the write time. In step S111a above, the user of the CPU basic computing resources can be a thread. Each data file corresponds to one thread, and the thread identifier and the corresponding data file's write time are both stored in the master node's shared hash mapping table (supporting online security). That is, the master node's shared hash mapping table stores all thread identifiers and the corresponding data file's write time; after sorting the write times, the data file corresponding to the earliest write time and the specific thread identifier for parsing this data file can be obtained.
[0028] Once the thread corresponding to the thread identifier has the necessary permissions, it can directly use the CPU's arithmetic units, registers, and other basic computing resources to execute corresponding instructions and complete data processing tasks; for example... Figure 3 As shown, in an optional implementation of step S112, parsing the data structure of the data file with the earliest write time according to the key-value pair to obtain the data information carried by the write request corresponding to the data file with the earliest write time includes: In step S112a, the data file with the earliest write time is parsed based on the determined user and the corresponding basic computing resources to obtain the data information carried by the write request corresponding to the data file with the earliest write time.
[0029] In the process of cross-node data backup, to ensure accurate data transmission to the target backup node, this invention employs a routed, directed transmission scheme instead of the traditional broadcast method, thereby reducing system resource consumption. The target backup node for the data is determined based on the data information and the partition table. For example... Figure 4As shown, the data information carried in the write request includes a data identifier and data content; as an optional implementation of step S120, determining the target backup node of the data information based on the parsed data information and the partition table includes: In step S121, a hash operation is performed on the data identifier to obtain a hash value; In step S122, the target backup node for the data identifier and data content is determined based on the hash value and the partition table; Similarly, Figure 4 As shown, as an optional implementation of step S130, sending the parsed data information to the target backup node includes: In step S131, the data information and data content are encapsulated into a backup request and then sent to the target backup node.
[0030] The data information required for data backup in this invention needs to be described in detail. The data information includes data identifiers and data content; the data content includes business data and / or preset deletion identifiers. The key-value pair data structure is key (data identifier) - value (data content). Specifically, key="user_1001", value="{name:'Zhang San', age:25}" is a set of data information including specific business data; key="user_1002", value="Tombstone" is a set of data information including preset deletion identifiers. By analyzing the data information, the operation type of the write request can be determined. Generally, write requests can be divided into three main categories: add, delete, and update. If the data content is a preset deletion identifier, it is a delete operation; if the data content is business data and there is no corresponding data identifier in the disk data file, it is an add operation; if the data content is business data and there is a corresponding data identifier in the disk data file, it is an update operation.
[0031] The following is in conjunction with the appendix Figure 5A more detailed flowchart of the data backup implementation within the master node is provided. The master node uses a master-slave thread model: it includes one master thread and multiple slave threads, which communicate using a blocking queue. The master thread is responsible for performing the detection work in step S110, i.e., detecting whether a key-value pair data file is being written from memory to disk, and notifying the slave threads if detected. The slave threads are responsible for parsing the data file in step S110 and determining the earliest written data file before parsing. The master thread uses WatchService to monitor the writing of key-value pair data files from memory to disk, encapsulates the path of the data file as an event, and puts it into the master-slave thread communication blocking queue (i.e., the master thread is responsible for listening and notifying). The slave threads listen to the blocking queue; if they find an event in the queue, they retrieve the event (i.e., the slave thread consumes it) and save the slave thread identifier (thread ID) - filename to a shared hash map table. Each slave thread periodically traverses the shared hash map, determining the order of file creation times based on the filename's write time, until it finds the data file with the earliest creation time corresponding to its own thread identifier in the shared hash map. The slave thread then uses the parser to parse the data file with the earliest creation time, obtaining an entry (a list of key-value pairs). Based on the partition table, it encapsulates the entries into an RPC request and sends it to the target standby node in the cluster. Figure 6 and Figure 7 The event loop flowchart is divided into the main thread and the slave thread.
[0032] As a second aspect of the present invention, a data backup method is provided for a target backup node of a distributed database, wherein, as Figure 8 As shown, the data backup method includes: In step S210, data information sent by the master node is received; In step S220, the data information is verified; In step S230, if the verification passes, the data information is written into a data file according to a key-value pair data structure; In step S240, the data file is saved to the disk of the target backup node.
[0033] Figure 9 Another step-by-step flowchart for the target backup node is given. Figure 9 In one implementation of step S210, receiving data information sent by the master node includes: In step S211, a backup request is received; In step S212, the backup request is parsed to obtain data information; wherein, the data information includes data identifier and data content; As one implementation of step S220, the verification of the data information includes: In step S221, a hash operation is performed on the data identifier to obtain a hash value; In step S222, the target backup node for the data identifier and data content is determined based on the hash value and the partition table; In step S223, if the identifier of the target backup node matches the identifier of the determined target backup node, the verification passes. In step S230, if the verification passes, the data information is written into a data file according to a key-value pair data structure; In step S240, the data file is saved to the disk of the target backup node.
[0034] The verification of the target standby node aims to verify the accuracy of the primary node's routing. The target standby node itself stores its own node identifier within the entire cluster; this identifier is calculated based on the received data and the partition table in the shared storage medium. To further clarify, the partition table includes not only the mapping between hash values and nodes, but also the cluster topology (node list, roles) and configuration information such as data sharding strategies. After verification, the process of the target standby node writing the data to disk is as follows: First, the data is categorized according to its content (business data / preset deletion flags). When the data information is a set of data including specific business data, such as key="user_1001", value="{name:'Zhang San', age:25}", the target backup node first checks whether this data identifier exists in the data files saved on the disk, that is, whether key="user_1001" exists. If it does not exist, it is a new addition, and the key-value pair key="user_1001", value="{name:'Zhang San', age:25}" is appended to the data file. If it exists (for example, the previously saved key-value pair is key="user_1001", value="{name:'Zhang San', age:24}"), it is an update. Although the key-value pair key="user_1001", value="{name:'Zhang San', age:25}" is still appended to the data file, after the key-value pairs are classified and merged, the latest key-value pair key="user_1001", value="{name:'Zhang San', age:25}" is retained. When the data information is a set of data information including a preset deletion identifier, such as key="user_1002", value="Tombstone", the target backup node will determine it as a deletion operation. It will first append key="user_1002", value="Tombstone" to the data file. Assuming the corresponding key-value pair in the data file before deletion is key="user_1002", value="{location:'Hangzhou, Zhejiang Province'}", then after classifying and merging the key-value pairs, the previously saved key-value pair key="user_1002", value="{location:'Hangzhou, Zhejiang Province'} is deleted, and the key-value pair key="user_1002", value="Tombstone" is retained. It should be noted again that the process of persisting data information to disk in the primary node is the same as that in the target backup node. With the data backup method of this invention, the primary node and the target backup node can achieve synchronous, efficient, and complete data backup without relying on operation logs. Figure 10 Provide a schematic diagram illustrating the process of a node backing up data to disk.Figure 10 The data structures of the first, second, and third data files are all key-value pair data structures proposed in this invention. Upon receiving a write request, data is written to the first data file in memory according to the key-value pair data structure. When the memory space allocated to the first data file is detected to be full, the content of the first data file is transferred to the second data file (the data in the first data file can be read by the second data file), and the memory space corresponding to the first data file is released to continuously receive external write requests and write new write requests to the first data file. It should be noted that the memory space allocated to the second data file is independent of the memory space of the first data file and does not interfere with each other; the data in the second data file can only be read. The node starts listening through the main thread. When it detects that data in the second data file has been written to the third data file on disk, it uses the method of this invention to parse the third data file according to the key-value pair data structure, determines the target backup node, and sends the parsed data information to the target backup node. For the third data file on disk, a layer-by-layer writing method is adopted: when the current layer of the third data file is full, it is sequentially classified and merged with the third data file of the next layer, ensuring the orderliness and efficiency of disk storage. Testing revealed that the method provided by this invention, compared to traditional master-slave replication, resolves the system performance bottleneck of writing persistent master-slave replication logs, reduces the memory and disk storage overhead of storing persistent master-slave replication logs, and eliminates the problem of reduced master-slave replication efficiency caused by reading persistent master-slave replication logs. The overall system TPS (Transactions Per Second) is increased by 30%, and the storage resource overhead of the storage medium is reduced by 50%.
[0035] As a third aspect of the invention, a master node is provided, such as Figure 11 As shown, it includes: One or more processors 101; The memory 102 stores one or more computer programs that, when executed by the one or more processors 101, cause the one or more processors 101 to implement the data backup method provided according to the first aspect of the invention.
[0036] The tool may also include one or more I / O interfaces 103 connected between the processor 101 and the memory 102, configured to enable information interaction between the processor 101 and the memory 102.
[0037] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit 101 (CPU); the first memory 102 is a device with data storage capabilities, including but not limited to random access memory 102 (RAM, more specifically SDRAM, DDR, etc.), read-only memory 102 (ROM), electrically erasable programmable read-only memory 102 (EEPROM), and flash memory (FLASH); the I / O interface 103 (read-write interface) is connected between the processor 101 and the memory 102, enabling information interaction between the processor 101 and the memory 102, including but not limited to a data bus 104 (Bus).
[0038] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.
[0039] As a fourth aspect of the present invention, a target backup node is provided, such as Figure 12 As shown, it includes: One or more processors 201; The memory 202 stores one or more computer programs that, when executed by the one or more processors 201, cause the one or more processors 201 to implement the data backup method provided according to the second aspect of the present invention.
[0040] The data node may also include one or more I / O interfaces 203 connected between the processor 201 and the memory 202, configured to enable information interaction between the processor 201 and the memory 202.
[0041] Among them, processor 201 is a device with data processing capabilities, including but not limited to central processing unit 201 (CPU); first memory 202 is a device with data storage capabilities, including but not limited to random access memory 202 (RAM, more specifically such as SDRAM, DDR, etc.), read-only memory 202 (ROM), electrically erasable programmable read-only memory 202 (EEPROM), flash memory (FLASH); I / O interface 203 (read-write interface) is connected between processor 201 and memory 202, and can realize information interaction between processor 201 and memory 202, including but not limited to data bus 204 (Bus).
[0042] In some embodiments, the processor 201, memory 202, and I / O interface 203 are interconnected via bus 204, and thus connected to other components of the computing device.
[0043] Furthermore, as a fifth aspect of the present invention, a computer-readable medium is also provided having a computer program stored thereon, such as Figure 13 As shown, when the computer program is executed by the processor, it implements the data backup method provided by the first and second aspects of the present invention.
[0044] The data backup method provided by this invention uses the writing of a key-value pair data file from the master node's memory to the disk as the trigger condition for the master node's own persistent data and cross-node backup. The data file is parsed according to the key-value pair data structure to extract data information corresponding to multiple write requests. Based on the parsed and extracted data information, and combined with the partition table in the shared storage medium (which records the mapping relationship between the hash values of the data information and the target backup nodes, and can be accessed by each node), the target backup node for the corresponding data information is determined and the data information is routed. After the target backup node writes the received data information into the data file using the key-value pair data structure, a data file with the same data structure and content as the master node will be formed on its disk. This data backup method is fundamentally different from traditional log-based solutions that rely on write operations. Because the data information required for backup can be directly parsed from the key-value pair data structure, there is no need to record additional operation logs. By reducing the dual write operations of the master node and the write log, the frequency of disk read and write operations is reduced, improving backup efficiency. At the same time, since the operation log is eliminated, the storage space occupied by the operation log is saved, further freeing up disk resources and avoiding backup interruptions or freezes caused by insufficient space or IO contention, ensuring the stability and efficiency of the backup process. This invention avoids redundant read and write operations caused by operation logs, combines partition tables to ensure accurate matching of target backup nodes, and finally routes the data information stored by the master node, achieving data consistency between master and backup nodes, improving the data backup efficiency across nodes, and has the advantages and characteristics of ensuring data consistency between master and backup nodes and efficient and complete backup of cross-node data.
[0045] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can implement the methods of any of the above embodiments. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
[0046] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes, but is not limited to, the contents described in the accompanying drawings and the specific embodiments above. Any modifications that do not depart from the functional and structural principles of the present invention will be included within the scope of the claims.
Claims
1. A data backup method for the master node of a distributed database, characterized in that, The data backup method includes: When it is detected that the master node is writing a key-value pair data file to the disk, the data file is parsed according to the key-value pair data structure to obtain the data information carried by multiple write requests; The target backup node is determined based on the parsed data information and the partition table; wherein, the partition table includes the mapping relationship between the hash value of the data information and the target backup node and is located in the shared storage medium; the shared storage medium can be accessed by the master node and the target backup node; The parsed data is sent to the target backup node.
2. The data backup method according to claim 1, characterized in that, The filename of the data file carries the write time; When the master node is detected writing a key-value pair data file to the disk, the data file is parsed according to the key-value pair data structure to obtain data information carried by multiple write requests, including: If it is detected that the master node is writing a key-value pair data file to the disk, the data file with the earliest writing time is determined according to the file name of the data file; The data file with the earliest write time is parsed according to the key-value pair data structure to obtain the data information carried by the write request corresponding to the data file with the earliest write time.
3. The data backup method according to claim 2, characterized in that, The step of determining the data file with the earliest write time based on the filename of the data file includes: Users who allocate basic CPU computing resources to each data file; Update the write time of the user and the corresponding data file to the master node; Sort the data files by their earliest write times and obtain the corresponding users; The process of parsing the data file with the earliest write time according to the key-value pair data structure yields the data information carried by the write request corresponding to the data file with the earliest write time, including: Based on the identified user and corresponding basic computing resources, the data file with the earliest write time is parsed to obtain the data information carried by the write request corresponding to the data file with the earliest write time.
4. The data backup method according to any one of claims 1 to 3, characterized in that, The data information includes data identifiers and data content; determining the target backup node of the data information based on the parsed data information and the partition table includes: Perform a hash operation on the data identifier to obtain a hash value; The target backup node for the data identifier and data content is determined based on the hash value and the partition table. Sending the parsed data information to the target backup node includes: The data information and data content are encapsulated into a backup request and then sent to the target backup node.
5. The data backup method according to claim 4, characterized in that, The data content includes business data and / or preset deletion identifiers.
6. A data backup method for a target backup node in a distributed database, characterized in that, The data backup method includes: Receive data information sent by the master node; Verify the data information; If the verification passes, the data information is written to the data file according to the key-value pair data structure; The data file is saved to the disk of the target backup node.
7. The data backup method according to claim 6, characterized in that, The data information received from the master node includes: Receive backup request; The backup request is parsed to obtain data information; wherein, the data information includes a data identifier and data content; The verification of the data information includes: Perform a hash operation on the data identifier to obtain a hash value; The target backup node for the data identifier and data content is determined based on the hash value and the partition table. The verification passes if the identifier of the target backup node matches the identifier of the determined target backup node.
8. A master node, characterized in that, include: One or more processors; A memory having stored one or more computer programs thereon, which, when executed by one or more processors, cause the one or more processors to implement the data backup method according to any one of claims 1 to 5.
9. A target backup node, characterized in that, include: One or more processors; A memory having stored one or more computer programs that, when executed by one or more processors, cause the one or more processors to implement the data backup method according to claim 6 or 7.
10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data backup method according to any one of claims 1 to 5 and 6 to 7.