Data replication methods, systems, devices, media, and software products

Abstract

This invention relates to the field of computer data processing, and particularly to an efficient replication method, system, device, medium, and program product for incremental logs in an LSM storage engine. The method is characterized by: receiving a data replication request from a slave node; determining a first position in the WAL log data and the current write position of the WAL log data, wherein the first position is the position where the last scan of the WAL log data ended; scanning the WAL log data from the first position to the current write position; when it is determined that new WAL log data has been added, scanning the new WAL log data and repeating this process a predetermined number of times until no more new WAL log data is found; storing the data at a second position, which is the position where the scan ends when no more new WAL log data is found; and sending the WAL log data to the slave node. This invention can ensure data consistency while effectively reducing latency.

Description

Technical Field

[0001] This invention relates to the field of computer data processing, and in particular to a data replication method, system, device, medium, and program product based on the LSM storage engine. Background Technology

[0002] In real-world business scenarios, businesses have increasingly stringent requirements for data consistency and are also highly sensitive to latency. Therefore, ensuring data consistency while keeping latency within acceptable limits is a critical challenge in current computer data processing. Current methods primarily employ asynchronous and synchronous replication. Asynchronous replication sends a write success notification to the client as soon as the master node successfully writes the data, while synchronous replication only sends a notification after both the master and slave nodes have successfully written the data. In asynchronous replication, the system offers lower latency and higher throughput. However, considering the possibility of disk failure, if the master node fails, some data may not have been written to the slave nodes, leading to further data loss. In synchronous replication, if the master node fails, the slave nodes have complete backup data, making recovery easier. However, synchronous replication increases data write latency, reduces system throughput, and the higher latency significantly increases the perceived delay for the client. Summary of the Invention

[0003] The purpose of this invention is to provide a data replication method, system, device, medium, and program product based on the LSM storage engine. By controlling the replication log boundary and ensuring the high probability of the WAL log iterator being effective, combined with batch synchronization, the latency of scanning the WAL log is reduced and the network transmission efficiency is improved, thereby solving the problem of high latency in incremental replication.

[0004] An embodiment of the present invention discloses a data copying method, the method comprising:

[0005] Receive data replication requests from slave nodes;

[0006] Determine the first position in the WAL log data and the current write position of the WAL log data. The first position is the position where the last scan of the WAL log data ended.

[0007] Scan the WAL log data from the first position to the current write position;

[0008] If it is determined that there is newly added WAL log data, scan the newly added WAL log data and repeat the process a predetermined number of times until there is no more newly added WAL log data.

[0009] The second location is the position where the scan ends when no more new WAL log data is found.

[0010] The WAL log data is sent to the slave node.

[0011] Optionally, when it is determined that there is newly added WAL log data, scanning the newly added WAL log data includes determining the latest write position of the WAL log data, and determining that there is newly added WAL log data when the current write position and the latest write position are inconsistent.

[0012] Optionally, when it is determined that there is newly added WAL log data, scanning the newly added WAL log data includes scanning from the current write position to the latest write position.

[0013] Optionally, the predetermined number of times is three.

[0014] An embodiment of the present invention discloses a data replication system, the system comprising:

[0015] The receiving module receives data replication requests from slave nodes;

[0016] The determination module determines the first position in the WAL log data and the current writing position of the WAL log data. The first position is the position where the last scan of the WAL log data ended.

[0017] The scanning module scans the WAL log data from the first position to the current write position;

[0018] The repeat module scans the newly added WAL log data and repeats the scan a predetermined number of times when it determines that there is no more newly added WAL log data.

[0019] The storage module stores a second location, which is the location where the scan ends when no more new WAL log data is found.

[0020] The sending module sends the WAL log data to the slave node.

[0021] An embodiment of the present invention discloses a master node device for a storage system, characterized in that the device includes a memory storing computer-executable instructions and a processor, the processor being configured to execute the instructions to implement the data replication method.

[0022] The present invention discloses a computer storage medium using computer program encoding, characterized in that the computer-readable storage medium stores at least one computer instruction, which is loaded and executed by a processor to implement the data copying method.

[0023] This invention provides an efficient synchronous write strategy. By controlling the replication log boundary, it ensures that the WAL log iterator is effective with a high probability. Combined with batch synchronization, it reduces the latency of scanning the WAL log and improves network transmission efficiency. This not only ensures high data availability but also enables incremental operations to be synchronized to slave nodes as soon as possible, effectively reducing the latency of incremental replication.

[0024] The main differences and effects of the embodiments of the present invention compared with the prior art are as follows:

[0025] In existing technologies, each replication requires generating an iterator to read the WAL log and scanning to a specific location. This scanning process involves verifying data in the WAL, consuming CPU resources and time, thus increasing latency. Furthermore, blindly scanning a large chunk of subsequent data at once to increase the amount of data scanned per cycle can easily cause iterators to fail when encountering the end of the WAL log. In short, all these methods contribute to high latency in incremental replication.

[0026] Compared to existing technologies, the distinguishing technical feature of this invention is that it determines a first position in the WAL log data and the current write position of the WAL log data, and scans the WAL log data from the first position to the current write position; when there is newly added WAL log data, it scans the newly added WAL log data and repeats the process a predetermined number of times until there is no more newly added WAL log data.

[0027] The technical advantage of this invention lies in the fact that it only generates an iterator for reading WAL data on the first scan. After reading incremental log data, the iterator moves to the next position to be read, and subsequent scans of WAL logs directly reuse this iterator. This effectively avoids repeated scanning of already scanned data and reduces the latency of each scan. If new WAL log data is found, only the new WAL log data is scanned each time, ensuring that the iterator is valid and reducing the step of recreating the iterator for the next scan. Scanning the new WAL log data and repeating this process a predetermined number of times until no more new WAL log data is found ensures high incremental replication efficiency and effectively reduces latency in high QPS scenarios. Attached Figure Description

[0028] Figure 1 This is a schematic diagram illustrating a scenario application of a data replication method based on an LSM storage engine according to an embodiment of the present invention;

[0029] Figure 2 This is a flowchart of a data replication method based on an LSM storage engine according to an embodiment of the present invention;

[0030] Figure 3 This is a scan timing diagram of a data replication method based on an LSM storage engine according to an embodiment of the present invention;

[0031] Figure 4 This is a structural block diagram of a data replication system based on an LSM storage engine according to an embodiment of the present invention;

[0032] Figure 5 This is a hardware structure block diagram of a master node device of a storage system according to an embodiment of the present invention. Detailed Implementation

[0033] The present invention will be further described below with reference to specific embodiments and accompanying drawings. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, for ease of description, the accompanying drawings show only the parts relevant to the invention, and not all of the structures or processes. It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings.

[0034] It should be understood that although the terms "first," "second," etc., may be used herein to describe various features, these features should not be limited by these terms. The use of these terms is merely for distinction and should not be construed as indicating or implying relative importance. For example, without departing from the scope of the exemplary embodiments, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature.

[0035] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0036] Figure 1 This is a schematic diagram illustrating a scenario application of a data replication method based on an LSM storage engine according to an embodiment of the present invention.

[0037] like Figure 1As shown, the system includes a master node 100 and a slave node 200. The master node 100 and slave node 200 communicate via a wired / wireless network connection, which also allows users to access the system. The master node 100 records the write offset, and the slave node 200 records the read offset. Both the master node 100 and slave node 200, participating in incremental data replication, maintain their own data replication offsets. After processing a write command, the master node 100 increments the command's byte length and records this information in its metrics. The slave node 200 reports its own data replication offset to the master node 100 every second; therefore, the master node 100 also stores the slave node 200's data replication offset. After receiving a command from the master node 100, the slave node 200 also increments its own data replication offset and records this information in its metrics. By comparing the data replication offsets of the master node 100 and slave node 200, it can be determined whether the data between the master and slave nodes is consistent.

[0038] When the client generates new data, the master database first performs the write operation, and then the master database synchronizes these operations to the slave database. That is, when the master node 100 receives the write instruction, the data replication offset will increase. After the slave node 200 executes the synchronous write instruction, the offset of the replicated data will also continue to increase.

[0039] Figure 1 The diagram shows two slave nodes 200. Those skilled in the art will understand that the number of slave nodes is not limited to two. It can be one master node 100 and three slave nodes 200, or it can be several slave nodes 200. The actual number of nodes can be set by those skilled in the art in actual operation.

[0040] Similarly, master node 100 and slave node 200, by setting a cluster name, discover nodes with the same cluster name in the same network and form a cluster. The number of nodes in the cluster composed of master node 100 and slave node 200 should not be considered a limitation of this invention.

[0041] As an example, under normal circumstances, the data replication offset of the master node 100 and the data replication offset of the slave node 200 are basically equal.

[0042] In another embodiment, during a network disconnection, the master node 200 may receive a new write operation command. At this time, the data replication offset of the master node 100 is greater than the data replication offset of the slave node 200. After the disconnection and reconnection, the slave node 200 will send a data replication request command to the master node 100. The master node 100 only needs to synchronize the command with the offset difference between the two to the slave node 200.

[0043] The master node 100 can store data and provide cross-node indexing and search capabilities.

[0044] Current data replication technologies on the market, such as asynchronous replication, only write to the master node (100) and then asynchronously replicate to the slave node (200). Asynchronous replication may fail, which cannot guarantee the high reliability requirements of single-copy data storage. According to consistency strategy requirements, a write operation can be performed by synchronizing the master and slave nodes and then returning a response to the client. This can improve reliability, but it sacrifices write performance and cannot achieve data consistency while ensuring that latency is within an acceptable range.

[0045] To address the aforementioned problems, this invention provides a data replication method based on an LSM storage engine. The method will be described in detail below with reference to the accompanying drawings.

[0046] Figure 2 This is a flowchart of a data replication method based on an LSM storage engine according to an embodiment of the present invention.

[0047] like Figure 2 As shown, an embodiment of the present invention provides a data replication method based on an LSM storage engine, which requires the cooperation of WAL log data 300, first position 301, current write position 302, latest write position 303, second position 304, and slave node 200 in steps S1-S6. The method includes:

[0048] Step S1: Receive a data replication request from slave node 200.

[0049] The master node 100 can send replication commands to all slave nodes 200 using either synchronous or asynchronous replication. Synchronous / asynchronous replication refers to replication relative to write requests. If the slave node data is updated before the write request is returned to the client, it's synchronous replication. Read requests can occur on both the master and slave nodes. The advantage of synchronous replication is that it ensures the data has been updated on the slave nodes when the write request returns. This invention provides an efficient synchronous write strategy that ensures high data availability while enabling incremental data replication operations to be synchronized to slave nodes 200 as quickly as possible, effectively reducing latency.

[0050] After a successful connection between slave node 200 and master node 100, slave node 200 sends a ping command to the master node to check if the master node 100 is currently able to accept and process commands. Generally, after slave node 200 sends the ping command, master node 100 will return a pong command. After receiving the pong reply from master node 100, slave node 200 will proceed to the next step. If slave node 200 does not receive a pong reply from master node 100 or a timeout occurs (e.g., network timeout or the master node is blocked and unable to respond to commands), the slave server will disconnect the replication connection and wait for the next scheduled task. The data replication requests received by master node 100 include psync commands from slave node 200 and write commands from the client. The psync command sent by slave node 200 includes the running ID of the master node being replicated by slave node 200 and the current data offset replicated by slave node 200.

[0051] In one embodiment, the batch synchronous replication method for data replication requests is that the client routes the write request to the master node 100 for execution, and the master node sends the WAL log data to the slave node 200.

[0052] Step S2: Determine the first position in the WAL log data and the current write position of the WAL log data. The first position is the position where the last scan of the WAL log data ended.

[0053] Step S3: Scan the WAL log data from the first position to the current write position.

[0054] The LSM storage engine is characterized by its layered and ordered nature. The Write-Ahead Log (WAL) ensures data integrity and consistency, and WriteBatch allows for batch data writing. Scanning the WAL log data is implemented using an iterator, which accesses the WriteBatch set. Those skilled in the art will understand that determining the first position and current write position of the WAL log data, as well as the specific scanning process, can employ known existing technologies, and will not be elaborated upon here.

[0055] Steps S2 and S3 described above can be referred to as the first scan process of the data replication method according to the present invention. Those skilled in the art will understand that data writing to the master node 100 may be ongoing, resulting in new write requests being generated during the first scan. To improve the efficiency of data replication, embodiments of this application further include a step of repeated scanning.

[0056] In step S4: If it is determined that there is newly added WAL log data, scan the newly added WAL log data and repeat the process a predetermined number of times until there is no more newly added WAL log data.

[0057] As mentioned above, during the first scan, there may be new write requests. Therefore, the current write position of the WAL log determined during the first scan is inconsistent with the latest write position. According to an embodiment of the present invention, after the first scan, the scanning procedure is repeated a predetermined number of times until no new WAL log data is found.

[0058] It should be noted that those skilled in the art will understand that the predetermined number of scans is based on the actual data copying operation performed by those skilled in the art. There is no fixed standard for the predetermined number of scans; it is determined by those skilled in the art based on the actual data copying situation, until no new WAL log data is found during the scan. The predetermined number of scans can be set according to actual needs, such as three, four, etc. The information used to detect the need for repeating the predetermined number of scans can be a heartbeat command, or other signals well known to those skilled in the art, or other detection methods well known to those skilled in the art to achieve the status detection of repeated scans.

[0059] Preferably, the predetermined number of times is three.

[0060] In one implementation, after the first scan, a second scan begins. If a new write request is found (i.e., the current write position and the latest write position are inconsistent), it is determined that new WAL log data has been added, and steps S2 and S3 are repeated. In practice, scanning for new WAL log data includes scanning from the current write position to the latest write position. After the second scan, two more scans are performed. If no new write request is found (i.e., the current write position and the latest write position are consistent), it is determined that no new WAL log data has been added, and the scan ends. If a new write request is found after the second scan, the scan continues, and this scanning logic is repeated three times until no more new WAL log data is found.

[0061] As another implementation method, after the first scan is completed, a second scan is repeated. If there are no new write requests, that is, the current write position is the same as the latest write position, it is determined that there is no new WAL log data, and the scan ends.

[0062] Next, in step S5: store the second position, which is the position where the scan ends when no more new WAL log data is found;

[0063] And in step S6: send the WAL log data to slave node 200.

[0064] When no new WAL log data is found after a predetermined number of logical scans, the scan ends, and the iterator moves to the next position to be scanned. Those skilled in the art will understand that the first position determined for the next scan is the second position stored in step S5 above, i.e., the position where the previous scan ended.

[0065] The master node 100 needs to send the write command to the slave node 200 for execution. After the slave node 200 executes the same write command, the master node 100 and the slave node 200 will be in the same state.

[0066] Next, combined Figure 3 The data replication method based on the LSM storage engine according to the present invention will be described.

[0067] Figure 3 This is a scan timing diagram of a data replication method based on an LSM storage engine according to an embodiment of the present invention.

[0068] Please refer to Figure 3 When the master node 100 receives a data replication request from the slave node 200, it first determines the starting and ending positions of the scan, and then scans the data between the two positions. As one implementation, when a client writes data in batches to a database based on the LSM storage engine, the master node 100 receives the data replication request from the slave node 200 and the client's write commands, such as... Figure 3 As shown, the master node 100 confirms the first position 301 and the current write position 302, and starts the first scan, scanning the WAL log data 300 between the first position 301 and the current write position 302.

[0069] The "blind" scanning data copying technique suffers from iterator invalidation due to improper use. For example, using a large receive buffer to receive WAL log data can cause a new iterator to be opened at the point where the iterator was used when the end of the WAL log is encountered, resulting in a "blind" scan from the beginning to near the end of the WAL log to read newly added WAL log data. In contrast to "blind" scanning, binary search is more efficient. Because the boundaries between records in the WAL log are determined by the length within each written record, specific delimiters can be set between written records. Binary search can then locate the specific written record and begin scanning from that record. However, binary search is a random read, which is not very efficient in terms of I / O performance. The inherent I / O overhead of random reads also limits its use to the offset of the first written record location. Therefore, reusing iterators to maintain the read written records is the most efficient data copying method.

[0070] Preferably, an iterator is generated only during the initial scan of WAL log data 300. After each scan of incremental log data, the iterator moves to the next position to be scanned, and subsequent scans of the WAL log directly reuse the iterator. In implementation, reusing the iterator helps avoid transaction failures caused by repeated data scans, effectively reduces latency, and ensures data consistency during data replication. Preferably, the first position 301 is the position where the last scan of WAL log data 300 ended. When a data replication request is received from slave node 200, the scan continues from the last scan end position, reusing the previous iterator. This avoids the problem of generating an iterator for scanning WAL log data from the next starting position that needs data replication each time, which helps reduce latency.

[0071] like Figure 3 As shown, during the first scan, the WAL log has new writes, and the new write position becomes 303, which is inconsistent with the current write position 302 determined during the first scan. According to an embodiment of the present invention, in order to improve the efficiency of data replication, a repeat scan step needs to be performed after the first scan is completed.

[0072] The second scan begins at position 302 where the previous scan ended and ends at the current write position. Similarly, if new writes occur during the second scan, according to one embodiment of the invention, the scan for newly added WAL log data will continue.

[0073] Figure 3 The embodiment of the invention shown here performs three scans, that is, the current write position is determined three times. Figure 3 In the illustrated embodiment, no new WAL logs were written after the second scan, therefore, the third scan was not actually performed.

[0074] Similarly, those skilled in the art will understand that the number of predetermined attempts is based on the actual data copying operation performed by those skilled in the art. There is no fixed standard for the number of predetermined attempts; it is based on the actual data copying situation performed by those skilled in the art until no new WAL log data is found during the scan. The number of predetermined attempts can be set according to actual needs.

[0075] After three scans are completed, according to an embodiment of the invention, the current write position 304 is stored. Those skilled in the art will understand that... Figure 3 The current write position 304 of the WAL log data shown is also the starting position for scanning the WAL log when the next data replication request is received.

[0076] After scanning the WAL log data is complete, the corresponding data is sent to the slave node. This concludes one data replication process according to an embodiment of the present invention. Through the above... Figure 2 as well as Figure 3 As will be understood by those skilled in the art, the data copying method according to the present invention can effectively avoid repeated scanning of scanned data, reduce the delay of each scan, and thus improve the efficiency of data copying.

[0077] Figure 4 This is a structural block diagram of a data replication system based on an LSM storage engine according to an embodiment of the present invention.

[0078] like Figure 4 As shown, system 400 includes a receiving module 401, a determining module 402, a scanning module 403, a repeating module 404, a storage module 405, and a sending module 406;

[0079] Receiver module 401 receives a data replication request from slave node 200;

[0080] The module 402 determines the first position 301 in the WAL log data 300 and the current write position 302 of the WAL log data 300. The first position 301 is the position where the last scan of the WAL log data 304 ended.

[0081] Scanning module 403 scans WAL log data 300 from the first position 301 to the current write position 302;

[0082] The repeat module 404 scans the newly added WAL log data 300 and repeats the scan a predetermined number of times until no more new WAL log data 300 is added.

[0083] Storage module 405 stores the second location 304, which is the location where the scan ends when no more new WAL log data 300 is found;

[0084] The sending module 406 sends the WAL log data 300 to the slave node 200.

[0085] This embodiment is a corresponding method embodiment to the aforementioned embodiments, and can be implemented in conjunction with the aforementioned embodiments. The relevant technical details mentioned in the aforementioned embodiments remain valid in this embodiment, and will not be repeated here to avoid repetition. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the aforementioned embodiments.

[0086] According to some embodiments of the present invention, a master node device for a storage system is disclosed. The device includes a memory storing computer-executable instructions and a processor, the processor being configured to execute instructions to implement a data replication method based on an LSM storage engine.

[0087] Figure 5 This is a hardware structure block diagram of a master node device implementing a storage system according to an embodiment of the present invention.

[0088] like Figure 5 As shown, the master node device 500 of the storage system may include one or more processors 502, a system motherboard 508 connected to at least one of the processors 502, system memory 505 connected to the system motherboard 508, non-volatile memory (NVM) 506 connected to the system motherboard 508, and a network interface 510 connected to the system motherboard 508.

[0089] Processor 502 may include one or more single-core or multi-core processors. Processor 502 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In embodiments of the invention, processor 502 may be configured to perform operations according to... Figure 2 The method shown.

[0090] In some embodiments, the system motherboard 508 may include any suitable interface controller to provide any suitable interface to at least one of the processors 502 and / or any suitable device or component communicating with the system motherboard 508.

[0091] In some embodiments, system motherboard 508 may include one or more memory controllers to provide an interface to system memory 505. System memory 505 may be used to load and store data and / or instructions. In some embodiments, system memory 505 of master node device 500 of storage system may include any suitable volatile memory, such as suitable dynamic random access memory (DRAM).

[0092] The NVM 506 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, the NVM 506 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of an HDD (Hard Disk Drive), a CD (Compact Disc) drive, or a DVD (Digital Versatile Disc) drive.

[0093] NVM 506 may include a portion of the storage resources on a device installed on the master node device 500 of the storage system, or it may be accessible by the device, but not necessarily as part of the device. For example, NVM 506 may be accessed over a network via network interface 510.

[0094] Specifically, system memory 505 and NVM 506 may each include a temporary copy and a permanent copy of instruction 520, respectively. Instruction 520 may include, when executed by at least one of processors 502, causing the master node device 500 of the storage system to perform actions such as... Figure 2 The instructions for the method shown. In some embodiments, the instructions 520, hardware, firmware and / or their software components may additionally / alternatively be located in the system motherboard 508, network interface 510 and / or processor 502.

[0095] Network interface 510 may include a transceiver for providing a radio interface to the master node device 500 of the storage system, thereby enabling communication with any other suitable device (e.g., a front-end module, antenna, etc.) via one or more networks. In some embodiments, network interface 510 may be integrated into other components of the master node device 500 of the storage system. For example, network interface 510 may be integrated into at least one of processor 502, system memory 505, NVM 506, and firmware device (not shown) with instructions, wherein the master node device 500 of the storage system implements... Figure 2 One or more embodiments of the various embodiments shown.

[0096] The network interface 510 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 510 may be a network adapter, a wireless network adapter, a telephone modem, and / or a wireless modem.

[0097] In one embodiment, at least one of the processors 502 may be packaged together with one or more controllers for the system motherboard 508 to form a system-in-package (SiP). In another embodiment, at least one of the processors 502 may be integrated on the same die with one or more controllers for the system motherboard 508 to form a system-on-a-chip (SoC).

[0098] The master node device 500 of the storage system may further include an input / output (I / O) device 512 connected to the system motherboard 508. The I / O device 512 may include a user interface enabling users to interact with the master node device 500 of the storage system; the peripheral component interface is designed to allow peripheral components to also interact with the master node device 500 of the storage system. In some embodiments, the master node device 500 of the storage system further includes sensors for determining at least one type of environmental condition and location information related to the master node device 500 of the storage system.

[0099] In some embodiments, I / O device 512 may include, but is not limited to, a display (e.g., a liquid crystal display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., a light-emitting diode flash) and a keyboard.

[0100] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0101] It is understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the master node device 500 of the storage system. In other embodiments of this application, the master node device 500 of the storage system may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0102] Program code can be applied to input instructions to perform the functions described in this invention and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a system for processing instructions, including processor 502, includes any system having a processor such as a digital signal processor (DSP), microcontroller, application-specific integrated circuit (ASIC), or microprocessor.

[0103] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this invention are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0104] According to one embodiment of the present invention, a computer-readable storage medium is also provided, wherein at least one computer instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the aforementioned method.

[0105] According to one embodiment of the present invention, a computer program product is also provided, the computer program product comprising computer instructions, which, when executed, implement the aforementioned method.

[0106] The illustrative embodiments of the present invention include, but are not limited to, a data replication method, system, device, medium, and program product based on an LSM storage engine.

[0107] Various aspects of the illustrative embodiments will be described using terminology commonly employed by those skilled in the art to convey the essence of their work to others skilled in the art. However, it will be apparent to those skilled in the art that some alternative embodiments will be practiced using some of the features described herein. Specific figures and configurations are set forth for purposes of explanation in order to provide a more thorough understanding of the illustrative embodiments. However, it will be apparent to those skilled in the art that alternative embodiments may be practiced without specific details. In some other instances, well-known features have been omitted or simplified herein to avoid obscuring the illustrative embodiments of the invention.

[0108] Furthermore, the various operations will be described as multiple separate operations in a manner most conducive to understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations must depend on the order of description, and many of these operations may be performed in parallel, concurrently, or simultaneously. Moreover, the order of the operations may also be rearranged. The process may be terminated when the described operations are completed, but may also include additional steps not included in the figures. The process may correspond to a method, function, procedure, subroutine, subroutine, etc.

[0109] References to "an embodiment," "in practice," "an implementation," etc., in the specification indicate that the described embodiment may include specific features, structures, or properties; however, each embodiment may or may not necessarily include specific features, structures, or properties. Furthermore, these phrases are not necessarily directed at the same embodiment. Moreover, when specific features are described in conjunction with specific embodiments, the knowledge of those skilled in the art can influence the combination of these features with other embodiments, whether or not those embodiments are explicitly described.

[0110] Unless the context otherwise specifies, the terms “comprising,” “having,” and “including” are synonyms. The phrase “A and / or B” means “(A), (B), or (A and B).”

[0111] As used herein, the term "module" may refer to, as part of, or include: a memory (shared, dedicated, or grouped), an application-specific integrated circuit (ASIC), electronic circuitry and / or a processor (shared, dedicated, or grouped), combinational logic circuitry, and / or other suitable components that provide the said functionality for running one or more software or firmware programs.

[0112] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order is not necessary. Rather, in some embodiments, these features may be illustrated in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular drawing does not mean that all embodiments need to include such features; in some embodiments, these features may be omitted or may be combined with other features.

[0113] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0114] Similarly, it should be understood that, in order to simplify the invention and aid in understanding one or more of the various inventive aspects, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of the invention. However, this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in fewer than all features of the single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0115] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature serving the same, equivalent, or similar purpose.

[0116] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

Claims

1. A data replication method for a master node device in a storage system, characterized in that, include: Receive data replication requests from slave nodes; Determine the first position in the WAL log data and the current write position of the WAL log data. The first position is the position where the last scan of the WAL log data ended. Scan the WAL log data from the first position to the current write position; If it is determined that there is newly added WAL log data, scan the newly added WAL log data and repeat the process a predetermined number of times until there is no more newly added WAL log data. Store the data at a second location, which is the location where the scan ends when no more new WAL log data is found; The WAL log data is sent to the slave node.

2. The data copying method according to claim 1, characterized in that, When it is determined that there is newly added WAL log data, scanning the newly added WAL log data includes determining the latest write position of the WAL log data. If the current write position and the latest write position are inconsistent, it is determined that there is newly added WAL log data.

3. A data copying method according to any one of claims 1-2, characterized in that, When it is determined that there is newly added WAL log data, scanning the newly added WAL log data includes scanning from the current write position to the latest write position.

4. The data copying method according to claim 1, characterized in that, The predetermined number of times is three.

5. A data replication system, characterized in that, include: The receiving module receives data replication requests from slave nodes; The determination module determines the first position in the WAL log data and the current writing position of the WAL log data. The first position is the position where the last scan of the WAL log data ended. The scanning module scans the WAL log data from the first position to the current write position; The repeat module, when it is determined that there is newly added WAL log data, scans the newly added WAL log data and repeats it a predetermined number of times until there is no more newly added WAL log data. The storage module stores a second location, which is the location where the scan ends when no more new WAL log data is found. The sending module sends the WAL log data to the slave node.

6. A master node device for a storage system, characterized in that, The device includes a memory storing computer-executable instructions and a processor configured to execute the instructions to implement the data copying method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer instruction, which is loaded and executed by a processor to implement the data copying method as described in any one of claims 1-4.

8. A computer program product, characterized in that, The computer program product includes computer instructions, which, when executed, implement the data copying method as described in any one of claims 1-4.

Images