File snapshot synchronization methods, systems, devices, and media based on differential logs

Abstract

This invention proposes a file snapshot synchronization method, system, device, and medium based on differential logs. The method includes using metadata logs to record file modification operations between a first snapshot and a second snapshot; the first snapshot is a snapshot taken when the master and slave clusters complete synchronization; the second snapshot is a snapshot taken after the first snapshot, during which the master cluster incrementally synchronizes over a period of time; and using differential logs to record the change data of each file modification operation, and achieving synchronization between the master and slave clusters based on the change data of file modification operations. Based on this method, a file snapshot synchronization system, device, and medium based on differential logs are also proposed. This invention designs metadata logs to record file modification operations and differential logs to record the specific content of file modification operations, identifying differences between different snapshots. The differential log distinguishes between using file change ranges and bitmaps according to specific circumstances, significantly improving comparison efficiency and reducing the amount of differences, thus improving the efficiency of incremental replication.

Description

Technical Field

[0001] This invention belongs to the field of distributed file system technology, and specifically relates to a file snapshot synchronization method, system, device, and medium based on differential logs. Background Technology

[0002] With increasing digitalization, data has gradually become the core of business operations for enterprises and institutions. Users also have higher requirements for the stability of storage systems that hold this data. Although many storage vendors can provide highly stable storage devices, they still cannot prevent irreparable damage to production systems caused by various natural disasters. There are various ways to protect data, the most common being to copy our data multiple times and store it in multiple locations. If one location fails and causes data loss, we can recover the data from other locations, thus ensuring data security. To ensure the continuity, recoverability, and high availability of data storage, remote replication technology has emerged, and incremental remote replication is one of its key functions. Incremental remote replication requires transferring as little data as possible to complete the replication function, thus saving bandwidth and improving replication efficiency. Remote replication is asynchronous, and the current mainstream method is to transfer snapshots. A snapshot is defined as a fully usable copy of a specified data set, which includes an image of the corresponding data at a certain point in time (the time the copy begins).

[0003] Each remote replication essentially synchronizes a snapshot of the master cluster to the slave cluster. The difference between the master and slave clusters is simply the difference between these two snapshots. The core of incremental remote replication is synchronizing the differences between the two snapshots. The efficiency of synchronization and the amount of difference directly affect the performance of incremental replication. A common existing snapshot synchronization method is to poll every file in the snapshot, comparing attributes such as modification time and file size to identify differences. This method requires polling the entire snapshot for each synchronization, which significantly reduces efficiency when the data volume is large. Furthermore, differences are processed in file units; when a single file is very large, even small modifications require transmitting the entire file, leading to high bandwidth pressure. Summary of the Invention

[0004] To address the aforementioned technical issues, this invention proposes a file snapshot synchronization method, system, device, and medium based on difference logs. When a file changes, the metadata log records the file modification operations; the difference log records the change data for each file modification operation, which can significantly improve comparison efficiency, reduce the amount of differences, and achieve data synchronization.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] The file snapshot synchronization method based on difference logs includes the following steps:

[0007] The file modification operations between the first snapshot and the second snapshot are recorded using metadata logs; the first snapshot is a snapshot taken when the master cluster and the slave cluster complete synchronization; the second snapshot is a snapshot taken after the master cluster performs incremental synchronization for a period of time after the first snapshot is taken.

[0008] Differential logs are used to record the changes in data for each file modification operation, and the master cluster and slave cluster are synchronized based on the changes in data.

[0009] Furthermore, the process of synchronizing the master cluster and the slave cluster based on the changed data is as follows: read the file change range in the changed data and transmit it to the slave cluster, write the file within the change range into the slave cluster, and realize the synchronization between the master cluster and the slave cluster.

[0010] Furthermore, the file modification operations include, but are not limited to, adding, deleting, and modifying files.

[0011] Furthermore, the difference log uses the filename as the key and the value to record the location of file changes.

[0012] Furthermore, the file change locations recorded in the value content include two types: file change range and bitmap.

[0013] Furthermore, the file variation range consists of the file offset and the file length, and is used during file creation and the first write.

[0014] Furthermore, the bitmap is used to record whether the data has changed; each bit of the bitmap in the difference log represents a block of the file, and when the data changes, the corresponding bit state is 1.

[0015] This invention also proposes a file snapshot synchronization system based on differential logs, including a first recording module and a recording synchronization module;

[0016] The first recording module is used to record file modification operations between the first snapshot and the second snapshot using metadata logs; the first snapshot is a snapshot taken when the master cluster and the slave cluster complete synchronization; the second snapshot is a snapshot taken after the master cluster performs incremental synchronization for a period of time after the first snapshot is taken;

[0017] The recording and synchronization module is used to record the change data of each file modification operation using a difference log, and to realize the synchronization between the master cluster and the slave cluster based on the change data.

[0018] The present invention also proposes an apparatus comprising:

[0019] Memory, used to store computer programs;

[0020] A processor for implementing the steps of the method when executing the computer program.

[0021] The present invention also proposes a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the method described.

[0022] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:

[0023] This invention proposes a file snapshot synchronization method, system, device, and medium based on differential logs. The method includes using metadata logs to record file modification operations between a first snapshot and a second snapshot; the first snapshot is a snapshot taken when the master and slave clusters complete synchronization; the second snapshot is a snapshot taken after the first snapshot, during which the master cluster incrementally synchronizes over a period of time; and using differential logs to record the change data of each file modification operation, and achieving synchronization between the master and slave clusters based on the change data of file modification operations. The invention also proposes a file snapshot synchronization system, device, and medium based on differential logs. This invention designs a reasonable metadata log to record file modification operations, designs a differential log to record the specific content of file modification operations, identifies the differences between different snapshots, and uses file change ranges and bitmap technology in the differential logs according to specific circumstances, which can significantly improve comparison efficiency and reduce the amount of differences, thereby improving the efficiency of incremental replication. Attached Figure Description

[0024] like Figure 1 This is a flowchart of the file snapshot synchronization method based on difference logs in Embodiment 1 of the present invention;

[0025] like Figure 2 This is a schematic diagram of file snapshot synchronization based on difference logs in Embodiment 1 of the present invention;

[0026] like Figure 3 This is a schematic diagram of a file snapshot synchronization system based on difference logs, as described in Embodiment 2 of the present invention. Detailed Implementation

[0027] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings. The following disclosure provides many different embodiments or examples for implementing different structures of the invention. To simplify the disclosure of the invention, components and arrangements of specific examples are described below. Furthermore, reference numerals and / or letters may be repeated in different examples. This repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed. It should be noted that the components illustrated in the drawings are not necessarily drawn to scale. Descriptions of well-known components, processing techniques, and processes are omitted in this invention to avoid unnecessarily limiting the invention.

[0028] Example 1

[0029] Embodiment 1 of this invention proposes a file snapshot synchronization method based on differential logs, which records metadata logs and DCL (Data Changed Log) data change logs during file I / O. When file system data changes, the metadata log records the file's add, delete, and modify operations; the DCL records the specific changes in the data. Each DCL log entry uses the filename as the key, and the file's data changes are recorded in the value as the range of changes or a bitmap.

[0030] DCL stands for Data Changed Log.

[0031] A block is a data block; the smallest logical unit for recording data in a file.

[0032] A snapshot is a backup of data in a file system at a specific point in time.

[0033] like Figure 1 A flowchart of the file snapshot synchronization method based on differential logs according to Embodiment 1 of the present invention is provided;

[0034] In step S101, the file modification operations between the first snapshot and the second snapshot are recorded using metadata logs; the first snapshot is the snapshot taken when the master cluster and the slave cluster complete synchronization; the second snapshot is the snapshot taken after the first snapshot is taken and the master cluster has undergone incremental synchronization over a period of time.

[0035] For example, at a moment immediately after synchronization, both the master and slave clusters have the latest snapshot as snapshot1. Then, the master cluster continues I / O, and after a period of time, incremental synchronization is performed. At this point, a snapshot (snapshot2) is first created on the master cluster. The metadata log and DCL record the changes from snapshot1 to snapshot2; these changes constitute the difference log. Incremental synchronization is the process of comparing the differences between snapshot1 and snapshot2, and then synchronizing these differences to snapshot1 on the slave cluster.

[0036] like Figure 2 This is a schematic diagram of file snapshot synchronization based on difference logs in Embodiment 1 of the present invention. If, after snapshot 1 is taken, changes to the directory are recorded by the metadata log, and the following operations are performed: creating directory dir3 and file4; deleting file2;

[0037] The metadata log needs to be re-executed from the cluster based on snapshot1 to complete the metadata difference synchronization.

[0038] In step S102, a difference log is used to record the change data of each file modification operation, and the synchronization between the master cluster and the slave cluster is achieved based on the change data of the file modification operations.

[0039] DCL (Data Changed Log) is a key technology for distinguishing master-slave differences in incremental replication. It is implemented based on bitmaps and can save memory and quickly deduplicate data.

[0040] The DCL (Data Change Log) and metadata change log record file changes together. Each incremental copy is performed based on these two logs. Each DCL entry uses the filename as the key and the value to record the file's changed location. The content has two types:

[0041] File change range: The file change range consists of offset and length, and is used when the file is created and written for the first time.

[0042] Bitmap: Bitmap is short for bitmap. A bitmap uses each bit to store a certain state, suitable for large-scale data where the number of data states is not large. It's typically used to determine the existence of data; here, it's used to record whether data has changed. Each bit in the DCL bitmap represents a block of the file. When a block changes, the corresponding bit state becomes 1. Using the principle of bitmaps, duplicate operation records can be merged (only the latest record of operations on the same block is taken). The DCL is cleared after each incremental copy.

[0043] exist Figure 2 In the file, changes to file data are recorded by the DCL: file4 writes 4096 bytes of data at offset 0; file3 has a block appended. Since file3 already has two blocks, appending a block is equivalent to a change in block 3 of the file, so the bitmap records it as 001. The number of blocks in the file corresponds to the number of bits in the bitmap.

[0044] Synchronization (metadata synchronization is completed during data synchronization):

[0045] The slave cluster learns from the DCL that file4 has changed within the range of 0 to 4096. At this point, it reads the data within this range from snapshot2 and transmits it to the slave cluster, which then writes it to the corresponding file.

[0046] The slave cluster learns from the DCL that block 3 of file3 has changed. At this point, it reads the data of block 3 of file3 in snapshot2 and transmits it to the slave cluster, which then writes it into the corresponding file's block.

[0047] Embodiment 1 of this invention proposes a file snapshot synchronization method based on a difference log. Snapshot differences are recorded using a metadata log and a DCL (Data Changed Log). The DCL distinguishes between using file change range (offset and length) and bitmap technology depending on the specific situation, which can significantly improve comparison efficiency and reduce the amount of difference. This improves the efficiency of incremental replication.

[0048] Example 2

[0049] Based on the file snapshot synchronization method based on difference logs, Embodiment 2 of this invention also proposes a file snapshot synchronization system based on difference logs, such as... Figure 3 This is a schematic diagram of a file snapshot synchronization system based on difference logs, as described in Embodiment 2 of the present invention.

[0050] The system includes a first recording module and a recording synchronization module;

[0051] The first recording module is used to record file modification operations between the first snapshot and the second snapshot using metadata logs; the first snapshot is a snapshot taken when the master cluster and the slave cluster complete synchronization; the second snapshot is a snapshot taken after the master cluster performs incremental synchronization for a period of time after the first snapshot is taken.

[0052] The synchronization module is used to record the change data of each file modification operation using a difference log, and to synchronize the master cluster and slave cluster based on the change data.

[0053] In the first recording module, for example, at a moment immediately after synchronization, the latest snapshot of both the master and slave clusters is snapshot1. Then, the master cluster continues I / O, and after a period of time, incremental synchronization is performed. At this time, a snapshot (snapshot2) is first created on the master cluster. The metadata log and DCL record the changes from snapshot1 to snapshot2; these changes constitute the difference log. Incremental synchronization is the process of comparing the differences between snapshot1 and snapshot2, and then synchronizing these differences to snapshot1 on the slave cluster.

[0054] In the record synchronization module, DCL (Data Changed Log, a log that records data differences) is a key technology for distinguishing master-slave differences in incremental replication. It is implemented based on bitmaps, which can save memory and quickly deduplicate data.

[0055] The DCL (Data Change Log) and metadata change log record file changes together. Each incremental copy is performed based on these two logs. Each DCL entry uses the filename as the key and the value to record the file's changed location. The content has two types:

[0056] File change range: The file change range consists of offset and length, and is used when the file is created and written for the first time.

[0057] Bitmap: Bitmap is short for bitmap. A bitmap uses each bit to store a certain state, suitable for large-scale data where the number of data states is not large. It's typically used to determine the existence of data; here, it's used to record whether data has changed. Each bit in the DCL bitmap represents a block of the file. When a block changes, the corresponding bit state becomes 1. Using the principle of bitmaps, duplicate operation records can be merged (only the latest record of operations on the same block is taken). The DCL is cleared after each incremental copy.

[0058] Finally, the file change range in the changed data is read and transmitted to the slave cluster. Files within the change range are written into the slave cluster to achieve synchronization between the master and slave clusters.

[0059] Example 3

[0060] The present invention also proposes an apparatus comprising:

[0061] Memory, used to store computer programs;

[0062] When a processor executes the computer program, the method steps are as follows:

[0063] like Figure 1A flowchart of the file snapshot synchronization method based on differential logs according to Embodiment 1 of the present invention is provided;

[0064] In step S101, the file modification operations between the first snapshot and the second snapshot are recorded using metadata logs; the first snapshot is the snapshot taken when the master cluster and the slave cluster complete synchronization; the second snapshot is the snapshot taken after the first snapshot is taken and the master cluster has undergone incremental synchronization over a period of time.

[0065] For example, at a moment immediately after synchronization, both the master and slave clusters have the latest snapshot as snapshot1. Then, the master cluster continues I / O, and after a period of time, incremental synchronization is performed. At this point, a snapshot (snapshot2) is first created on the master cluster. The metadata log and DCL record the changes from snapshot1 to snapshot2; these changes constitute the difference log. Incremental synchronization is the process of comparing the differences between snapshot1 and snapshot2, and then synchronizing these differences to snapshot1 on the slave cluster.

[0066] like Figure 2 This is a schematic diagram of file snapshot synchronization based on difference logs in Embodiment 1 of the present invention. If, after snapshot 1 is taken, changes to the directory are recorded by the metadata log, and the following operations are performed: creating directory dir3 and file4; deleting file2;

[0067] The metadata log needs to be re-executed from the cluster based on snapshot1 to complete the metadata difference synchronization.

[0068] In step S102, a difference log is used to record the change data of each file modification operation, and the synchronization between the master cluster and the slave cluster is achieved based on the change data of the file modification operations.

[0069] DCL (Data Changed Log) is a key technology for distinguishing master-slave differences in incremental replication. It is implemented based on bitmaps and can save memory and quickly deduplicate data.

[0070] The DCL (Data Change Log) and metadata change log record file changes together. Each incremental copy is performed based on these two logs. Each DCL entry uses the filename as the key and the value to record the file's changed location. The content has two types:

[0071] File change range: The file change range consists of offset and length, and is used when the file is created and written for the first time.

[0072] Bitmap: Bitmap is short for bitmap. A bitmap uses each bit to store a certain state, suitable for large-scale data where the number of data states is not large. It's typically used to determine the existence of data; here, it's used to record whether data has changed. Each bit in the DCL bitmap represents a block of the file. When a block changes, the corresponding bit state becomes 1. Using the principle of bitmaps, duplicate operation records can be merged (only the latest record of operations on the same block is taken). The DCL is cleared after each incremental copy.

[0073] exist Figure 2 In the file, changes to file data are recorded by the DCL: file4 writes 4096 bytes of data at offset 0; file3 has a block appended. Since file3 already has two blocks, appending a block is equivalent to a change in block 3 of the file, so the bitmap records it as 001. The number of blocks in the file corresponds to the number of bits in the bitmap.

[0074] Synchronization (metadata synchronization is completed during data synchronization):

[0075] The slave cluster learns from the DCL that file4 has changed within the range of 0 to 4096. At this point, it reads the data within this range from snapshot2 and transmits it to the slave cluster, which then writes it to the corresponding file.

[0076] The slave cluster learns from the DCL that block 3 of file3 has changed. At this point, it reads the data of block 3 of file3 in snapshot2 and transmits it to the slave cluster, which then writes it into the corresponding file's block.

[0077] Embodiment 3 of this invention proposes a file snapshot synchronization device based on a difference log. It records snapshot differences through a metadata log and a DCL (Data Changed Log), where the DCL distinguishes between using file change range (offset and length) and bitmap technology depending on the specific situation, significantly improving comparison efficiency and reducing the amount of difference. This improves the efficiency of incremental replication.

[0078] It should be noted that the present invention also provides an electronic device, including: a communication interface capable of interacting with other devices such as network devices; and a processor connected to the communication interface to enable information interaction with other devices, used to execute the file snapshot synchronization method based on differential logs provided by one or more of the above technical solutions when running a computer program, wherein the computer program is stored in a memory. Of course, in practical applications, the various components in the electronic device are coupled together through a bus system. It is understood that the bus system is used to realize the connection and communication between these components. In addition to a data bus, the bus system also includes a power bus, a control bus, and a status signal bus. The memory in the embodiments of this application is used to store various types of data to support the operation of the electronic device. Examples of this data include any computer program used to operate on the electronic device. It is understood that the memory can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache.By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memory. The methods disclosed in the embodiments of this application can be applied to or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by integrated logic circuits in the processor hardware or by instructions in software. The processor can be a general-purpose processor, a DSP (Digital Signal Processing, i.e., a chip capable of implementing digital signal processing technology), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in a storage medium, which is located in memory. The processor reads the program from the memory and, in conjunction with its hardware, completes the steps of the aforementioned method. When the processor executes the program, it implements the corresponding processes in the various methods of the embodiments of this application; for simplicity, these will not be elaborated further here.

[0079] Example 4

[0080] The present invention also proposes a readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the following method steps:

[0081] like Figure 1 A flowchart of the file snapshot synchronization method based on differential logs according to Embodiment 1 of the present invention is provided;

[0082] In step S101, the file modification operations between the first snapshot and the second snapshot are recorded using metadata logs; the first snapshot is the snapshot taken when the master cluster and the slave cluster complete synchronization; the second snapshot is the snapshot taken after the first snapshot is taken and the master cluster has undergone incremental synchronization over a period of time.

[0083] For example, at a moment immediately after synchronization, both the master and slave clusters have the latest snapshot as snapshot1. Then, the master cluster continues I / O, and after a period of time, incremental synchronization is performed. At this point, a snapshot (snapshot2) is first created on the master cluster. The metadata log and DCL record the changes from snapshot1 to snapshot2; these changes constitute the difference log. Incremental synchronization is the process of comparing the differences between snapshot1 and snapshot2, and then synchronizing these differences to snapshot1 on the slave cluster.

[0084] like Figure 2 This is a schematic diagram of file snapshot synchronization based on difference logs in Embodiment 1 of the present invention. If, after snapshot 1 is taken, changes to the directory are recorded by the metadata log, and the following operations are performed: creating directory dir3 and file4; deleting file2;

[0085] The metadata log needs to be re-executed from the cluster based on snapshot1 to complete the metadata difference synchronization.

[0086] In step S102, a difference log is used to record the change data of each file modification operation, and the synchronization between the master cluster and the slave cluster is achieved based on the change data of the file modification operation.

[0087] DCL (Data Changed Log) is a key technology for distinguishing master-slave differences in incremental replication. It is implemented based on bitmaps and can save memory and quickly deduplicate data.

[0088] The DCL (Data Change Log) and metadata change log record file changes together. Each incremental copy is performed based on these two logs. Each DCL entry uses the filename as the key and the value to record the file's changed location. The content has two types:

[0089] File change range: The file change range consists of offset and length, and is used when the file is created and written for the first time.

[0090] Bitmap: Bitmap is short for bitmap. A bitmap uses each bit to store a certain state, suitable for large-scale data where the number of data states is not large. It's typically used to determine the existence of data; here, it's used to record whether data has changed. Each bit in the DCL bitmap represents a block of the file. When a block changes, the corresponding bit state becomes 1. Using the principle of bitmaps, duplicate operation records can be merged (only the latest record of operations on the same block is taken). The DCL is cleared after each incremental copy.

[0091] exist Figure 2 In the file, changes to file data are recorded by the DCL: file4 writes 4096 bytes of data at offset 0; file3 has a block appended. Since file3 already has two blocks, appending a block is equivalent to a change in block 3 of the file, so the bitmap records it as 001. The number of blocks in the file corresponds to the number of bits in the bitmap.

[0092] Synchronization (metadata synchronization is completed during data synchronization):

[0093] The slave cluster learns from the DCL that file4 has changed within the range of 0 to 4096. At this point, it reads the data within this range from snapshot2 and transmits it to the slave cluster, which then writes it to the corresponding file.

[0094] The slave cluster learns from the DCL that block 3 of file3 has changed. At this point, it reads the data of block 3 of file3 in snapshot2 and transmits it to the slave cluster, which then writes it into the corresponding file's block.

[0095] Embodiment 4 of this invention proposes a file snapshot synchronization medium based on a difference log. Snapshot differences are recorded through a metadata log and a DCL (Data Changed Log). The DCL distinguishes between using file change range (offset and length) and bitmap technology depending on the specific situation, which can significantly improve comparison efficiency and reduce the amount of difference. This improves the efficiency of incremental replication.

[0096] This application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory that stores a computer program, which can be executed by a processor to complete the steps described in the aforementioned method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0097] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks. Alternatively, if the integrated units of this application are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0098] The description of the relevant parts of the file snapshot synchronization device and storage medium based on differential logs provided in the embodiments of this application can be found in the detailed description of the corresponding parts of the file snapshot synchronization method based on differential logs provided in Embodiment 1 of this application, and will not be repeated here.

[0099] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that the elements inherent in a process, method, article, or apparatus that includes a list of elements are included. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Additionally, portions of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of corresponding technical solutions in the prior art have not been described in detail to avoid excessive elaboration.

[0100] While specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art can make other modifications or variations based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A file snapshot synchronization method based on differential logs, characterized in that, When a file changes, the metadata log records the file modification operations; the difference log records the change data for each file modification operation, including the following steps: The file modification operations between the first snapshot and the second snapshot are recorded using metadata logs; the first snapshot is a snapshot taken when the master cluster and the slave cluster complete synchronization; the second snapshot is a snapshot taken after the master cluster performs incremental synchronization for a period of time after the first snapshot is taken. A difference log is used to record the change data of each file modification operation, and the master cluster and slave cluster are synchronized based on the change data. The process of synchronizing the master cluster and slave cluster based on the change data is as follows: read the file change range in the change data and transmit it to the slave cluster, write the file within the change range into the slave cluster, and realize the synchronization between the master cluster and slave cluster. The difference log uses the file name as the key and the value content to record the change position of the file. The file change locations recorded in the value content include two types: file change range and bitmap; The range of file changes consists of the file offset and the file length, and is used when the file is created and written for the first time.

2. The file snapshot synchronization method based on differential logs according to claim 1, characterized in that, The file modification operations include, but are not limited to, adding, deleting, and modifying files.

3. The file snapshot synchronization method based on differential logs according to claim 1, characterized in that, The bitmap is used to record whether the data has changed; each bit of the bitmap in the difference log represents a block of the file, and when the data changes, the corresponding bit state is 1.

4. A file snapshot synchronization system based on differential logs, used to execute the file snapshot synchronization method based on differential logs as described in any one of claims 1 to 3, characterized in that, Includes a first recording module and a recording synchronization module; The first recording module is used to record file modification operations between the first snapshot and the second snapshot using metadata logs; the first snapshot is a snapshot taken when the master cluster and the slave cluster complete synchronization; the second snapshot is a snapshot taken after the master cluster performs incremental synchronization for a period of time after the first snapshot is taken; The recording and synchronization module is used to record the change data of each file modification operation using a difference log, and to realize the synchronization between the master cluster and the slave cluster based on the change data.

5. A device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the method as described in any one of claims 1 to 3.

6. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 3.

Images