A data synchronization method and apparatus

Abstract

Embodiments of the present application disclose a data synchronization method and apparatus. By generating a first snapshot of the primary storage device, copying one or more files included in the first snapshot to the secondary storage device, generating a second snapshot of the primary storage device after receiving processing of one or more files by a user in the primary storage device, comparing the files included in the first snapshot with the files included in the second snapshot, determining the differential files between the first snapshot and the second snapshot, the differential files can be newly added files or modified files, and copying the differential files to the secondary storage device. It can be seen that the embodiments of the present application provide for generating snapshots at two different times, determining differential files in the snapshots at two different times, and copying the differential files to the secondary storage device, thereby achieving accurate copying of files in the primary storage device to the secondary storage device and ensuring data security.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular, to a data synchronization method and apparatus. Background Art

[0002] With the rapid development of computer technology, computers provide more and more services for users, such as data storage services. Facing the increasing data storage requirements of users, the data storage volume is getting larger and larger. How to ensure the security of data storage is the current research focus.

[0003] Currently, the main storage device and the slave storage device are adopted to improve the security of data storage. The main storage device belongs to the main cluster and mainly provides storage services for users. The slave storage device belongs to the slave cluster and mainly serves as a backup device for storing data of the main storage device. The data stored in the main storage device is exactly the same as the data stored in the slave storage device, so as to ensure the security of data storage by the slave storage device when the main storage device fails.

[0004] However, how to accurately copy the data stored in the main storage device to the slave storage device for backup. Currently, a method for synchronizing the data of the main storage device and the slave storage device is urgently needed. Summary of the Invention

[0005] To solve the problems in the prior art, embodiments of this application provide a data synchronization method and apparatus.

[0006] Embodiments of this application provide a data synchronization method, which is applied to a main storage device and a slave storage device. The main storage device belongs to the main cluster, and the slave storage device belongs to the slave cluster. The method includes:

[0007] Generate a first snapshot of the main storage device, where the first snapshot includes one or more files;

[0008] According to the first snapshot, copy the one or more files to the slave storage device;

[0009] In response to a user's processing of the one or more files, generate a second snapshot of the main storage device, where the second snapshot includes one or more files;

[0010] Compare the first snapshot and the second snapshot, and determine the differential files between the one or more files included in the first snapshot and the one or more files included in the second snapshot;

[0011] Copy the differential files to the slave storage device.

[0012] Optionally, the one or more files of the first snapshot include a first file;

[0013] Copying the one or more files to the slave storage device according to the first snapshot includes:

[0014] Creating a blank second file in the slave storage device;

[0015] Opening the first file and the second file, and copying the content of the first file to the second file.

[0016] Optionally, one or more files of the first snapshot include an index file;

[0017] Copying the one or more files to the slave storage device according to the first snapshot includes:

[0018] Obtaining the index file and creating the index file in the slave storage device.

[0019] Optionally, the first snapshot further includes one or more directories;

[0020] Before copying the one or more files to the slave storage device according to the first snapshot, the method further includes:

[0021] Copying the one or more directories to the slave storage device according to the first snapshot;

[0022] Before responding to the user's processing of the one or more files, the method further includes:

[0023] In response to the user's processing of the one or more directories, generating a third snapshot of the main storage device, the third snapshot including one or more directories;

[0024] Comparing the first snapshot and the third snapshot, and determining the different directories between the one or more directories included in the first snapshot and the one or more directories included in the third snapshot;

[0025] Copying the different directories to the slave storage device.

[0026] Optionally, when copying the one or more files to the slave storage device according to the first snapshot, the method further includes:

[0027] Creating metadata information of the one or more files in the slave storage device;

[0028] When copying the different files to the slave storage device, the method further includes:

[0029] Modifying the metadata information of the different files in the slave storage device.

[0030] Optionally, before copying the one or more files to the slave storage device according to the first snapshot, the method further includes:

[0031] Deleting all files on the slave storage device.

[0032] An embodiment of the present application further provides a data synchronization device, the device includes:

[0033] A first generation unit, configured to generate a first snapshot of the master storage device, where the first snapshot includes one or more files;

[0034] A first copy unit, configured to copy the one or more files to the slave storage device according to the first snapshot;

[0035] A second generation unit, configured to generate a second snapshot of the master storage device in response to a user's processing of the one or more files, where the second snapshot includes one or more files;

[0036] A first comparison unit, configured to compare the first snapshot and the second snapshot, and determine differential files between the one or more files included in the first snapshot and the one or more files included in the second snapshot;

[0037] A second copy unit, configured to copy the differential files to the slave storage device.

[0038] Optionally, the one or more files of the first snapshot include a first file;

[0039] The first copy unit is specifically configured to:

[0040] Create a blank second file on the slave storage device;

[0041] Open the first file and the second file, and copy the content of the first file to the second file.

[0042] Optionally, the one or more files of the first snapshot include an index file;

[0043] The first copy unit is specifically configured to:

[0044] Obtain the index file, and create the index file in the slave storage device.

[0045] Optionally, the first snapshot further includes one or more directories;

[0046] The device further includes:

[0047] A third copy unit, configured to copy the one or more directories to the slave storage device according to the first snapshot;

[0048] A third generation unit, configured to generate a third snapshot of the main storage device in response to the user's processing of the one or more directories, where the third snapshot includes one or more directories;

[0049] A second comparison unit, configured to compare the first snapshot and the third snapshot, and determine a difference directory between the one or more directories included in the first snapshot and the one or more directories included in the third snapshot;

[0050] A fourth copying unit, configured to copy the difference directory to the secondary storage device.

[0051] Optionally, the apparatus further includes:

[0052] A creation unit, configured to create metadata information of the one or more files in the secondary storage device;

[0053] A modification unit, configured to modify the metadata information of the difference files in the secondary storage device.

[0054] Optionally, the apparatus further includes:

[0055] A deletion unit, configured to delete all files of the secondary storage device.

[0056] An embodiment of the present application provides a data synchronization method. By generating a first snapshot of a main storage device, copying one or more files included in the first snapshot to a secondary storage device, generating a second snapshot of the main storage device after receiving the user's processing of one or more files in the main storage device, comparing the files included in the first snapshot and the files included in the second snapshot, determining the difference files between the first snapshot and the second snapshot, where the difference files may be newly added files or modified files, and copying the difference files to the secondary storage device. It can be seen that the embodiment of the present application provides generating snapshots at two different times, determining the difference files in the snapshots at two different times, and copying the difference files to the secondary storage device, so as to accurately copy the files in the main storage device to the secondary storage device and ensure data security. Description of the Drawings

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0058] Figure 1 It is a flowchart of a data synchronization method provided by an embodiment of the present application;

[0059] Figure 2 Schematic diagram of a snapshot provided by an embodiment of the present application;

[0060] Figure 3 Block diagram of a data synchronization device provided by an embodiment of the present application. Specific implementation manners

[0061] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0062] As described in the background art, the existing method uses a primary storage device and a secondary storage device to improve the security of data storage. The primary storage device belongs to the primary cluster and mainly provides storage services for users. The secondary storage device belongs to the secondary cluster and mainly serves as a backup device for storing data of the primary storage device. The data stored in the primary storage device is exactly the same as the data stored in the secondary storage device, so as to ensure the security of data storage by the secondary storage device when the primary storage device fails.

[0063] Currently, there are various solutions for copying the data stored in the primary storage device to the secondary storage device for backup. Due to different implementation solutions, the maximum time required from the failure of the primary storage device to its repair (Recovery Time Objective, RTO) and the maximum possible time for losing the stored data (Recovery Point Objective, RPO) are also different. Therefore, how to accurately copy the changed data of the primary storage device to the secondary storage device when the data stored in the primary storage device changes is an urgent problem to be solved.

[0064] Based on this, an embodiment of the present application provides a data synchronization method. By generating a first snapshot of the primary storage device, copying one or more files included in the first snapshot to the secondary storage device, after receiving the processing of one or more files by the user in the primary storage device, generating a second snapshot of the primary storage device, comparing the files included in the first snapshot and the files included in the second snapshot, determining the different files between the first snapshot and the second snapshot, the different files can be newly added files or modified files, and copying the different files to the secondary storage device. It can be seen that the embodiment of the present application provides to generate snapshots at two different times, determine the different files in the snapshots at two different times, and copy the different files to the secondary storage device, so as to accurately copy the files in the primary storage device to the secondary storage device and ensure the security of data.

[0065] The embodiments of the present application are applied to a primary storage device and a secondary storage device. The primary storage device belongs to the primary cluster, and the secondary storage device belongs to the secondary cluster.

[0066] Refer to Figure 1 , which is a flowchart of a data synchronization method provided by an embodiment of the present application.

[0067] The data synchronization method provided in this embodiment includes the following steps:

[0068] S101, generate a first snapshot of the primary storage device, where the first snapshot includes one or more files.

[0069] In the embodiments of the present application, a snapshot is a temporary cache, and the snapshot can carry time information. Generating a first snapshot of the primary storage device means temporarily caching the data that has been stored on the primary storage device at a certain moment.

[0070] In the embodiments of the present application, a snapshot can cache one or more directories, and the directories can include subdirectories and one or more files.

[0071] S102, according to the first snapshot, copy the one or more files to the secondary storage device.

[0072] In the embodiments of the present application, according to the one or more files cached by the snapshot, the one or more files can be copied to the secondary storage device for backup.

[0073] In practical applications, first, a mirror process is required to establish a connection between the primary storage device and the secondary storage device.

[0074] In practical applications, if it is the first time to copy files to the secondary storage device, before copying the one or more files to the secondary storage device according to the first snapshot, if there is data stored in the secondary storage device, all the data in the secondary storage device, such as all files or all directories, will be deleted. This can avoid the confusion between the files copied from the first snapshot to the secondary storage device and the original data in the secondary storage device due to the existence of data in the secondary storage device.

[0075] In the embodiments of the present application, before copying the one or more files to the secondary storage device according to the first snapshot, since the first snapshot also includes one or more directories, one or more directories can be copied to the secondary storage device for backup first according to the first snapshot.

[0076] That is, before copying one or more directories and one or more files in the first snapshot to the slave storage device, it is detected whether the same directories or files exist in the slave storage device. If they exist, the same directories or files in the slave storage device are deleted.

[0077] When actually copying one or more files to the slave storage device according to the first snapshot, there are the following two different copying implementation methods according to different file types:

[0078] The first implementation method is that if the first file, which is a hard link file, is included in one or more files cached in the first snapshot, then first create a blank second file in the slave storage device, and then open the first file and the second file, and copy the content of the first file to the second file, thus completing the process of copying the first file to the slave storage device.

[0079] After copying the content in the first file to the second file, close the first file and the second file.

[0080] The second implementation method is that if the index file, which is a soft link file, is included in one or more files cached in the first snapshot, then directly obtain the soft link file, and then create an index file in the slave storage device.

[0081] S103, in response to the user's processing of the one or more files, generate a second snapshot of the main storage device, where the second snapshot includes one or more files.

[0082] In the embodiments of the present application, when the user processes the data stored in the main storage device, for example, processes one or more files, then it is necessary to copy the processed files to the slave storage device. Processing one or more files can be adding files, modifying files, or deleting files.

[0083] After the user processes the first or more files, a second snapshot of the main storage device can be generated, and one or more files are also cached in the second snapshot.

[0084] In practical applications, if the user processes one or more directories in the main storage device, a third snapshot can also be generated, and one or more directories are cached in the generated third snapshot. Processing one or more directories can be adding directories, modifying directories, or deleting directories.

[0085] In practical applications, the snapshot of the main storage device can be generated not only after the user modifies the data stored in the main storage device, but also at a preset time interval. The preset time interval is determined according to the actual situation, and the embodiments of the present application do not make specific limitations on this.

[0086] S104. Compare the first snapshot and the second snapshot to determine the differential files between the one or more files included in the first snapshot and the one or more files included in the second snapshot.

[0087] In an embodiment of the present application, by comparing the data cached in the first snapshot and the second snapshot, the data changed in the primary storage device can be determined. For example, the differential files between the one or more files included in the first snapshot and the one or more files included in the second snapshot.

[0088] Specifically, first traverse the data cached in the first snapshot and the data cached in the second snapshot. After the traversal is completed, compare the first snapshot and the second snapshot to determine the differential files.

[0089] As an example, as shown in Figure 2 , the first snapshot is Snap1 and the second snapshot is Snap2. The first snapshot includes 3 files: File1, File2, and File3. The second snapshot includes 4 files: File1, File2, File3, and File4. By comparing the first snapshot and the second snapshot, the differential file is determined to be File4.

[0090] In practical applications, the first snapshot and the third snapshot can also be compared to determine the differential directories between the one or more directories included in the first snapshot and the one or more directories included in the third snapshot.

[0091] S105. Copy the differential files to the secondary storage device.

[0092] In an embodiment of the present application, after comparing the first snapshot and the second snapshot, the data changed in the primary storage device, such as the differential files, are determined and copied to the secondary storage device.

[0093] If the differential file is a newly added file, directly copy the newly added file to the secondary storage device.

[0094] If the differential file is a deleted file, directly delete the file in the secondary storage device.

[0095] If the differential file is a modified file, the original file in the secondary storage device can be deleted, and the modified file can be copied to the secondary storage device according to the second snapshot.

[0096] In practical applications, if the first snapshot and the third snapshot are compared to determine the differential directories, the differential directories can be copied to the secondary storage device.

[0097] In an embodiment of the present application, when a file or directory is created, it carries the creation time (ctime), and when a file or directory is modified, it carries the modification time (mtime). If the ctime or mtime of a file or directory is modified, it is considered that the file or directory has changed. Therefore, when comparing the first snapshot and the second snapshot or comparing the first snapshot and the third snapshot, the ctime or mtime of the file or directory can be directly compared.

[0098] In an embodiment of the present application, after copying the differential file or differential directory to the slave storage device, if it is a newly added file or directory, metadata information of one or more files or metadata information of one or more directories is created in the slave storage device. The metadata information may include the ctime and mtime of the file or directory, as well as the owner and group of the file or directory; if it is a modified file or directory, the metadata information of the differential file is modified in the slave storage device to adapt to the creation or modification time of the file or directory by the slave storage device.

[0099] In an embodiment of the present application, after copying the differential file or differential directory to the slave storage device, a fourth snapshot of the slave storage device can also be generated, and the fourth snapshot is compared with the second snapshot or the fourth snapshot is compared with the third snapshot to determine whether the data stored in the slave storage device after the data copying is completed is the same as that of the master storage device, so as to increase the accuracy of data copying.

[0100] The embodiment of the present application provides a data synchronization method. By generating a first snapshot of the master storage device, one or more files included in the first snapshot are copied to the slave storage device. After receiving the processing of one or more files by the user in the master storage device, a second snapshot of the master storage device is generated, and the files included in the first snapshot are compared with the files included in the second snapshot to determine the differential files between the first snapshot and the second snapshot. The differential files may be newly added files or modified files, and the differential files are copied to the slave storage device. It can be seen that the embodiment of the present application provides generating snapshots at two different times, determining the differential files in the snapshots at two different times, and copying the differential files to the slave storage device, so as to accurately copy the files in the master storage device to the slave storage device, realize remote copying of the data in the master storage device to the slave storage device, ensure data security, and use the snapshot to cache the data in the master storage device, reduce data movement, do not affect the normal operation of the master storage device, ensure the storage performance of the master cluster, and improve the stability of the cluster providing storage services to users.

[0101] Based on the data synchronization method provided in the above embodiments, the embodiment of the present application further provides a data synchronization device, and its working principle will be described in detail below with reference to the accompanying drawings.

[0102] See Figure 3, this figure is a structural block diagram of a data synchronization device provided by an embodiment of the present application.

[0103] The data synchronization device 300 provided in this embodiment includes:

[0104] A first generation unit 310, configured to generate a first snapshot of the main storage device, where the first snapshot includes one or more files;

[0105] A first replication unit 320, configured to copy the one or more files to the secondary storage device according to the first snapshot;

[0106] A second generation unit 330, configured to generate a second snapshot of the main storage device in response to a user's processing of the one or more files, where the second snapshot includes one or more files;

[0107] A first comparison unit 340, configured to compare the first snapshot and the second snapshot to determine differential files between the one or more files included in the first snapshot and the one or more files included in the second snapshot;

[0108] A second replication unit 350, configured to copy the differential files to the secondary storage device.

[0109] Optionally, the one or more files of the first snapshot include a first file;

[0110] The first replication unit is specifically configured to:

[0111] Create a blank second file in the secondary storage device;

[0112] Open the first file and the second file, and copy the content of the first file to the second file.

[0113] Optionally, the one or more files of the first snapshot include an index file;

[0114] The first replication unit is specifically configured to:

[0115] Obtain the index file and create the index file in the secondary storage device.

[0116] Optionally, the first snapshot further includes one or more directories;

[0117] The device further includes:

[0118] A third replication unit, configured to copy the one or more directories to the secondary storage device according to the first snapshot;

[0119] A third generation unit, configured to generate a third snapshot of the main storage device in response to the user's processing of the one or more directories, where the third snapshot includes one or more directories;

[0120] A second comparison unit, configured to compare the first snapshot and the third snapshot, and determine a difference directory between the one or more directories included in the first snapshot and the one or more directories included in the third snapshot;

[0121] A fourth copying unit, configured to copy the difference directory to the secondary storage device.

[0122] Optionally, the apparatus further includes:

[0123] A creation unit, configured to create metadata information of the one or more files in the secondary storage device;

[0124] A modification unit, configured to modify the metadata information of the difference files in the secondary storage device.

[0125] Optionally, the apparatus further includes:

[0126] A deletion unit, configured to delete all files of the secondary storage device.

[0127] When introducing elements of various embodiments of the present application, the articles "a", "an", "this" and "the" are all intended to mean that there is one or more elements. The terms "comprising", "including" and "having" are all inclusive and mean that there may be other elements in addition to the listed elements.

[0128] It should be noted that those of ordinary skill in the art can understand that implementing all or part of the processes in the above method embodiments can be completed by instructing relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it may include the processes of the above method embodiments. Among them, the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0129] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the relevant parts of the method embodiments for the relevant parts. The device embodiments described above are merely illustrative, and the units and modules described as separate components may or may not be physically separated. In addition, some or all of the units and modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0130] The above are only specific embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A data synchronization method, characterized in that, Applied to a primary storage device and a secondary storage device. The primary storage device belongs to a primary cluster and is used to provide storage services to users. The secondary storage device belongs to a secondary cluster and serves as a backup device for storing data of the primary storage device. The data stored in the primary storage device is exactly the same as the data stored in the secondary storage device. The method includes: Generate a first snapshot of the primary storage device. The first snapshot caches one or more directories, and the directories include subdirectories and one or more files; If it is the first time to copy a file or a directory to the secondary storage device, and if the secondary storage device has stored data, then delete all the data in the secondary storage device; According to the first snapshot, copy one or more directories to the secondary storage device for backup, and copy one or more directories to the secondary storage device for backup; During the process of copying the one or more files to the secondary storage device, if a first file among the one or more files in the first snapshot is a hard link file, then create a blank second file in the secondary storage device, open the first file and the second file, copy the content of the first file to the second file, and after copying the content in the first file to the second file, close the first file and the second file to complete copying the first file to the secondary storage device; If an index file among the one or more files in the first snapshot is a soft link file, then directly obtain the soft link file and create a corresponding index file in the secondary storage device; In response to the user's processing of the one or more directories, generate a third snapshot of the primary storage device. The third snapshot includes one or more directories; Compare the first snapshot and the third snapshot to determine the difference directories between the one or more directories included in the first snapshot and the one or more directories included in the third snapshot; Copy the difference directories to the secondary storage device; In response to the user's processing of the one or more files, generate a second snapshot of the primary storage device. The second snapshot includes one or more files; Compare the first snapshot and the second snapshot to determine the difference files between the one or more files included in the first snapshot and the one or more files included in the second snapshot; Copy the difference files to the secondary storage device; After copying the difference files or the difference directories to the secondary storage device, generate a fourth snapshot of the secondary storage device, and compare the fourth snapshot with the second snapshot or compare the fourth snapshot with the third snapshot to determine whether the data stored in the secondary storage device after copying the data is the same as that in the primary storage device.

2. The method according to claim 1, wherein When copying the one or more files to the secondary storage device according to the first snapshot, the method further includes: Create metadata information of the one or more files in the secondary storage device; When copying the difference files to the secondary storage device, the method further includes: Modify the metadata information of the difference files in the secondary storage device.

3. A data synchronization device, characterized in that, Applied to a primary storage device and a secondary storage device. The primary storage device belongs to a primary cluster and is used to provide storage services for users. The secondary storage device belongs to a secondary cluster and serves as a backup device for storing data of the primary storage device. The data stored in the primary storage device is exactly the same as the data stored in the secondary storage device. The apparatus includes: A first generating unit, configured to generate a first snapshot of the primary storage device. The first snapshot caches one or more directories, and the directories include subdirectories and one or more files; A deleting unit, configured to, if it is the first time to copy a file or a directory to the secondary storage device, and if the secondary storage device has stored data, delete all the data in the secondary storage device; A first copying unit, configured to copy the one or more files to the secondary storage device according to the first snapshot; A third copying unit, configured to copy the one or more directories to the secondary storage device according to the first snapshot; A second generating unit, configured to generate a second snapshot of the primary storage device in response to a user's processing of the one or more files. The second snapshot includes one or more files; A third generating unit, configured to generate a third snapshot of the primary storage device in response to the user's processing of the one or more directories. The third snapshot includes one or more directories; A first comparing unit, configured to compare the first snapshot and the second snapshot to determine differential files between the one or more files included in the first snapshot and the one or more files included in the second snapshot; A second comparing unit, configured to compare the first snapshot and the third snapshot to determine differential directories between the one or more directories included in the first snapshot and the one or more directories included in the third snapshot; A second copying unit, configured to copy the differential files to the secondary storage device; A fourth copying unit, configured to copy the differential directories to the secondary storage device; Wherein, the first copying unit is further configured to: if a first file among the one or more files of the first snapshot is a hard link file, create a blank second file in the secondary storage device, open the first file and the second file, copy the content of the first file to the second file, and after copying the content in the first file to the second file, close the first file and the second file to complete copying the first file to the secondary storage device; if an index file among the one or more files of the first snapshot is a soft link file, directly obtain the soft link file and create a corresponding index file in the secondary storage device; Wherein, the apparatus is further configured to: after copying the differential files or the differential directories to the secondary storage device, generate a fourth snapshot of the secondary storage device, and compare the fourth snapshot with the second snapshot or compare the fourth snapshot with the third snapshot to determine whether the data stored in the secondary storage device after copying the data is the same as that in the primary storage device.

Images