Disk-rsync-based cross-cluster data migration method

Abstract

The application provides a disk-rsync-based cross-cluster data migration method, which realizes automatic data migration operation by means of a disk-rsync data synchronization tool, cooperation with an ssh file transfer protocol, and host snapshot, cloud disk snapshot, BFV and virtual machine initialization technology based on an openstack cloud platform. The application realizes efficient incremental backup by means of application of a data synchronization migration gateway virtual machine Merkle tree and incremental data synchronization in the disk-rsync data synchronization tool, avoids repeated transmission of existing data, only needs to transmit the changed part on the source disk, reduces the required transmission time and bandwidth, is very useful in long-time backup, can greatly reduce the required resources for backup, brings a new opportunity for data management in a cloud computing environment, has advantages such as data integrity, incremental synchronization and service availability, and can realize more efficient and more reliable data migration.

Description

Technical Field

[0001] This invention relates to the field of cloud computing data migration technology, and more specifically, to a cross-cluster data migration method based on disk-rsync. Background Technology

[0002] With the rapid development of cloud computing technology, data migration has become one of the key challenges for resource optimization and efficient management in cloud environments. Especially in open-source cloud computing platforms like OpenStack, implementing cross-cluster data migration solutions is crucial for achieving goals such as flexible resource scheduling, performance optimization, and disaster recovery.

[0003] However, existing data migration technologies in cloud computing often suffer from the following three drawbacks:

[0004] 1. Lack of data consistency and integrity verification mechanisms: Many current data migration solutions lack mandatory data consistency and integrity verification mechanisms during the migration process. This can lead to data loss, corruption, or inconsistency after migration, thereby threatening business reliability.

[0005] Second, current technologies only support full data synchronization and lack incremental synchronization for large-scale data migrations. This means that the entire dataset needs to be transferred for each migration. This not only consumes a lot of bandwidth and time but also increases the risks during the migration process.

[0006] Third, the availability of current services cannot be guaranteed: In many data migration solutions, related services and applications must be stopped to perform data transfer. This leads to business interruption, impacting user experience and enterprise operations, especially for scenarios requiring 24 / 7 availability. Summary of the Invention

[0007] Therefore, the purpose of this invention is to propose a cross-cluster data migration scheme based on disk-rsync, which combines the latest technologies in cloud computing and data migration to solve the above-mentioned problems and shortcomings of existing technologies. This scheme uses the disk-rsync tool, combined with block-level data synchronization and hash tree verification mechanism, to achieve efficient, secure and reliable data migration operations, and meet the application requirements of flexible resource scheduling, performance optimization and disaster recovery.

[0008] This invention provides a cross-cluster data migration method based on disk-rsync. By using the disk-rsync data synchronization tool in conjunction with the SSH file transfer protocol, and based on OpenStack cloud platform technologies such as host snapshots, cloud disk snapshots, BFV (BootFrom Volume), and virtual machine initialization (Metadata / Config-Drive / Userdata / Cloud-init), automated data migration operations are achieved.

[0009] The method for achieving automated data migration includes the following steps:

[0010] S1. The source platform creates a temporary snapshot of the host to be migrated. If the BFI collection method is used, in addition to creating a temporary snapshot of the host, a temporary snapshot of the cloud disk is also created. A cloud disk volume is created from each created temporary snapshot, and the cloud disk volume is mounted to the data synchronization migration gateway virtual machine (Agent). That is, the gray part on the left, these are all temporary snapshots and volumes, which will be deleted after the data migration is completed.

[0011] Source Platform: This refers to the cloud platform from which resources such as virtual machines and cloud disks are migrated.

[0012] Instance to be migrated: This represents the virtual machine to be migrated. It records the mapping relationship of the virtual machine's disk, network, and other configurations between the source and destination platforms. Multiple migration tasks can be initiated for the same virtual machine.

[0013] S2. After the data synchronization gateway virtual machine completes the data synchronization work through the disk-rsync data synchronization tool, it unmounts the cloud disk volume mounted on the data synchronization migration gateway virtual machine Agent of the source platform, deletes the temporary snapshot and cloud disk volume created by the source platform, and creates a snapshot of the cloud disk volume that has received the data on the target platform; that is, the gray part on the right. If data is synchronized multiple times, a new snapshot will be created after each synchronization is completed.

[0014] Data synchronization migration gateway virtual machine (Agent): A migration agent that performs data synchronization between two platforms. It is actually a virtual machine on the platform, which deploys the migration agent. Both the source platform and the target platform need to build a data synchronization migration gateway virtual machine.

[0015] S3. When starting the host instance on the target platform, create the corresponding cloud disk volume from the cloud disk volume snapshot created in step S2, create a cloud host using BFV, and mount all the cloud disk volumes to the cloud host.

[0016] Target Platform: This refers to the cloud platform where resources such as virtual machines and cloud disks will be connected.

[0017] Host Instance (Server): This refers to the virtual machine created on the target platform using the migrated data after the virtual machine's data has been migrated to the target platform.

[0018] S4. By using the Agent program running in the virtual machine of the data synchronization migration gateway, the block devices that need to be synchronized are transferred to the cloud disk volume of the target platform in an incremental synchronization manner based on the SSH file transfer protocol.

[0019] To improve the agent's adaptability and portability, the agent uses the SSH protocol to copy the block device from the source platform to the block device on the remote target platform. It primarily has the following functions:

[0020] (1) Incremental Data Synchronization: The Agent implements incremental data synchronization by building a Merkle hash tree structure on the block devices of the source and target platforms. Only the data blocks that have changed are transmitted, which greatly reduces data transmission time and bandwidth.

[0021] (2) Compressed transmission and data encryption: For supported file systems such as xfs and ext4, the Agent supports compressed data transmission, reducing the size of transmitted data. At the same time, by utilizing the encryption algorithms supported by the SSH file transfer protocol, the data in transmission is encrypted to ensure data security during the transmission process.

[0022] (3) Adaptive Storage Abstraction: The Agent is designed to adapt to different underlying storage technologies, thereby shielding users from the details of the underlying storage. Using the Agent, users do not need to care about how the underlying storage works, thus simplifying the data migration process.

[0023] (4) Supports sparse writing: When the source block device has an empty disk or a disk that is not full, the Agent can support sparse writing, ensuring that the data size synchronized to the target platform is consistent with that of the source platform, thus avoiding the waste of disk space.

[0024] Strategies can be implemented during the migration process to ensure continued service availability. For example, temporary backups can be set up during the migration.

[0025] Furthermore, the method for transmitting data in incremental synchronization via the Agent program in step S4 includes:

[0026] The data synchronization and migration gateway virtual machine agent constructs a Merkle hash tree for each block device at the underlying level. A Merkle tree is a typical binary tree structure where non-leaf nodes store the hash values ​​of their child nodes. Its advantages include the ability to quickly compare large amounts of data and rapidly locate differences. By comparing the data differences between the source and target platform's disks using the Merkle hash tree, only the changed portions are transmitted, thus achieving efficient incremental backup.

[0027] Agent is a migration proxy service designed to solve the problem of inconsistent underlying storage. By using Agent, the underlying storage can be shielded, eliminating the need to worry about the underlying storage method. At the operating system level, data migration and synchronization are performed by remotely replicating block devices.

[0028] The Agent relies on this feature to implement the following two functions:

[0029] 1. Quickly compare large amounts of data;

[0030] After sorting each set of data, a Merkle tree structure is constructed. If two Merkle trees have the same root, it means the two sets of data they represent are necessarily the same; otherwise, the two sets of data they represent are necessarily different. Since hash calculations are very fast, preprocessing can be completed quickly. Utilizing the Merkle tree structure offers significant performance advantages in comparison.

[0031] 2. Quickly locate and modify.

[0032] Furthermore, the implementation process of the incremental synchronization method in the disk-rsync data synchronization tool includes:

[0033] S41. Initial Backup: The disk on the source platform is divided into fixed-size blocks using the disk-rsync data synchronization tool, and the hash value of each block is calculated. These hash values ​​are used to build the initial Merkle tree, and the root hash value of the entire tree represents the state of the disk on the entire source platform.

[0034] During the initial backup phase, all data needs to be transferred because the target platform data does not yet exist.

[0035] S42, Incremental Backup: When the data on the disk of the source platform changes, the disk-rsync data synchronization tool first detects the changed blocks, and then finds the corresponding blocks in the Merkle tree of the target platform based on the hash values ​​of these blocks.

[0036] After the initial backup is completed, subsequent backups can be performed incrementally using Merkle trees.

[0037] The disk-rsync data synchronization tool can perform incremental synchronization between the source platform and the target platform cluster, transmitting only the changed data blocks, thereby reducing the amount of data transmission.

[0038] S43. Difference Calculation: By comparing the hash values ​​of blocks on the disks of the source platform and the target platform, the disk-rsync data synchronization tool is used to determine which blocks have changed, i.e., the difference blocks; the difference blocks, along with their parent nodes and root nodes, are then transmitted to the target platform.

[0039] S44. Partial data transmission: After receiving the difference blocks, the target platform reconstructs the data by rebuilding these blocks in the Merkle tree;

[0040] This allows the source platform to transmit only the difference blocks instead of the entire file, thereby reducing bandwidth usage and transmission time;

[0041] S45. Verify data migration integrity: After the data is reconstructed on the target platform, the target platform reconstructs the Merkle tree based on the transmitted difference blocks and existing data. The data integrity is verified by comparing the root hash value of the Merkle tree on the target platform with the root hash value of the Merkle tree on the source platform.

[0042] The hash verification mechanism based on Merkle trees ensures the integrity of data in the disk-rsync data synchronization tool. After migration, the correct data transmission can be verified by comparing the Merkle trees on the source and target platform clusters.

[0043] Furthermore, the rules for verifying data integrity in step S45 include:

[0044] If the root hash value at the destination matches the root hash value provided by the sender, then the data integrity is confirmed.

[0045] If the root hash value at the destination does not match the root hash value provided by the sender, it means that the data has been erroneous, corrupted, or maliciously tampered with during transmission. In this case, the sender is notified to take appropriate measures to retransmit the incremental data.

[0046] Furthermore, the automated data migration operation utilizes the Linux file system to complete data reading and writing.

[0047] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the disk-rsync-based cross-cluster data migration method described above.

[0048] The present invention also provides a computer device, the computer device including a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the steps of the disk-rsync-based cross-cluster data migration method described above.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] This invention presents a cross-cluster data migration method based on disk-rsync. By applying the Merkle tree of the data synchronization migration gateway virtual machine and performing incremental data synchronization in the disk-rsync data synchronization tool, it can achieve efficient incremental backup, avoiding the repeated transmission of existing data. Only the changed parts on the source disk need to be transmitted, thereby reducing the time and bandwidth required for transmission. This method is very useful for long-term backups, which can greatly reduce the resources required for backup. This invention brings new opportunities to data management in cloud computing environments. This method has advantages such as data integrity, incremental synchronization, and service availability, and can achieve more efficient and reliable data migration. Attached Figure Description

[0051] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0052] In the attached diagram:

[0053] Figure 1 This is a schematic diagram illustrating the structure of a Merkle tree constructed according to an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram illustrating the resources automatically created by the source platform and the target platform during the migration process, as well as the relationship between them, according to an embodiment of the present invention.

[0055] Figure 3 This is a schematic diagram illustrating how the Agent constructs a Merkle hash tree for each block device in an embodiment of the present invention.

[0056] Figure 4 This is a schematic diagram illustrating the structure of the initial Merkle tree constructed for the data of ABCDEFGH in an embodiment of the present invention.

[0057] Figure 5 This is a schematic diagram of the structure of incrementally transmitted data IJKL and the reconstruction of the Merkle tree structure at the target end in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram of the structure of the target end of the present invention reconstructing the Merkle tree based on the merged data;

[0059] Figure 7 This is a flowchart illustrating a method for implementing automated data migration operations according to an embodiment of the present invention.

[0060] Figure 8 This is a diagram illustrating the implementation process of incremental synchronization in the disk-rsync data synchronization tool according to an embodiment of the present invention.

[0061] Figure 9 This is a schematic diagram of the configuration of a computer device according to an embodiment of the present invention. Detailed Implementation

[0062] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and products consistent with some aspects of this disclosure as detailed in the appended claims.

[0063] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0064] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0065] The embodiments of the present invention will be described in further detail below.

[0066] This invention provides a cross-cluster data migration method based on disk-rsync. By using the disk-rsync data synchronization tool in conjunction with the SSH file transfer protocol, and based on OpenStack cloud platform technologies such as host snapshots, cloud disk snapshots, BFV (Boot From Volume), and virtual machine initialization (Metadata / Config-Drive / Userdata / Cloud-init), automated data migration operations are achieved.

[0067] The method for implementing automated data migration operations is described in [reference needed]. Figure 7 As shown, it includes the following steps:

[0068] S1. The source platform creates a temporary snapshot of the host to be migrated. If the BFI collection method is used, in addition to creating a temporary snapshot of the host, a temporary snapshot of the cloud disk is also created. A cloud disk volume is created from each created temporary snapshot, and the cloud disk volume is mounted to the data synchronization migration gateway virtual machine (Agent). That is, the gray part on the left, these are all temporary snapshots and volumes, which will be deleted after the data migration is completed.

[0069] Source Platform: This refers to the cloud platform from which resources such as virtual machines and cloud disks are migrated.

[0070] Instance to be migrated: This represents the virtual machine to be migrated. It records the mapping relationship of the virtual machine's disk, network, and other configurations between the source and destination platforms. Multiple migration tasks can be initiated for the same virtual machine.

[0071] S2. After the data synchronization gateway virtual machine completes the data synchronization work through the disk-rsync data synchronization tool, it unmounts the cloud disk volume mounted on the data synchronization migration gateway virtual machine Agent of the source platform, deletes the temporary snapshot and cloud disk volume created by the source platform, and creates a snapshot of the cloud disk volume that has received the data on the target platform; that is, the gray part on the right. If data is synchronized multiple times, a new snapshot will be created after each synchronization is completed.

[0072] Data synchronization migration gateway virtual machine (Agent): A migration agent that performs data synchronization between two platforms. It is actually a virtual machine on the platform, which deploys the migration agent. Both the source platform and the target platform need to build a data synchronization migration gateway virtual machine.

[0073] S3. When starting the host instance on the target platform, create the corresponding cloud disk volume from the cloud disk volume snapshot created in step S2, create a cloud host using BFV, and mount all the cloud disk volumes to the cloud host.

[0074] Target Platform: This refers to the cloud platform where resources such as virtual machines and cloud disks will be connected.

[0075] Host Instance (Server): This refers to the virtual machine created on the target platform using the migrated data after the virtual machine's data has been migrated to the target platform.

[0076] S4. By using the Agent program running in the virtual machine of the data synchronization migration gateway, the block devices that need to be synchronized are transferred to the cloud disk volume of the target platform in an incremental synchronization manner based on the SSH file transfer protocol.

[0077] See Figure 2 The diagram shows the resources automatically created by the source and target platforms during the migration process, as well as the relationships between them.

[0078] To improve the agent's adaptability and portability, the agent uses the SSH protocol to copy the block device from the source platform to the block device on the remote target platform. It primarily has the following functions:

[0079] (1) Incremental Data Synchronization: The Agent implements incremental data synchronization by building a Merkle hash tree structure on the block devices of the source and target platforms. Only the data blocks that have changed are transmitted, which greatly reduces data transmission time and bandwidth.

[0080] (2) Compressed transmission and data encryption: For supported file systems such as xfs and ext4, the Agent supports compressed data transmission, reducing the size of transmitted data. At the same time, by utilizing the encryption algorithms supported by the SSH file transfer protocol, the data in transmission is encrypted to ensure data security during the transmission process.

[0081] (3) Adaptive Storage Abstraction: The Agent is designed to adapt to different underlying storage technologies, thereby shielding users from the details of the underlying storage. Using the Agent, users do not need to care about how the underlying storage works, thus simplifying the data migration process.

[0082] (4) Supports sparse writing: When the source block device has an empty disk or a disk that is not full, the Agent can support sparse writing, ensuring that the data size synchronized to the target platform is consistent with that of the source platform, thus avoiding the waste of disk space.

[0083] In this embodiment, the automated data migration operation utilizes the Linux file system to complete data reading and writing.

[0084] Strategies can be implemented during the migration process to ensure continued service availability. For example, temporary backups can be set up during the migration.

[0085] In this embodiment, the method of transmitting data in an incremental synchronization manner via an Agent program includes:

[0086] The data synchronization migration gateway virtual machine Agent constructs a Merkle hash tree for each block device at the underlying level. See [link to relevant documentation] Figure 3As mentioned, a Merkle tree is a typical binary tree structure where non-leaf nodes store the hash values ​​of their child nodes. Its key features are the ability to quickly compare large amounts of data and rapidly locate differences. By comparing data differences between the source and target platform's disks using a Merkle hash tree, only the changed portions are transmitted, thus achieving efficient incremental backups.

[0087] Agent is a migration proxy service designed to solve the problem of inconsistent underlying storage. By using Agent, the underlying storage can be shielded, eliminating the need to worry about the underlying storage method. At the operating system level, data migration and synchronization are performed by remotely replicating block devices.

[0088] The Agent relies on this feature to implement the following two functions:

[0089] 1. Quickly compare large amounts of data;

[0090] After sorting each set of data, a Merkle tree structure is constructed. If two Merkle trees have the same root, it means the two sets of data they represent are necessarily the same; otherwise, the two sets of data they represent are necessarily different. Since hash calculations are very fast, preprocessing can be completed quickly. Utilizing the Merkle tree structure offers significant performance advantages in comparison.

[0091] 2. Quickly locate and modify.

[0092] See Figure 1 As shown, a Merkle tree is constructed based on data D0...D3. If the data in D1 is modified, it will affect N1, N4 and Root.

[0093] Furthermore, for the implementation process of the incremental synchronization method in the disk-rsync data synchronization tool, please refer to [link to documentation]. Figure 8 As shown, it includes the following steps:

[0094] S41. Initial Backup: The disk on the source platform is divided into fixed-size blocks using the disk-rsync data synchronization tool, and the hash value of each block is calculated. These hash values ​​are used to build the initial Merkle tree, and the root hash value of the entire tree represents the state of the disk on the entire source platform.

[0095] During the initial backup phase, all data needs to be transferred because the target platform data does not yet exist.

[0096] S42, Incremental Backup: When the data on the disk of the source platform changes, the disk-rsync data synchronization tool first detects the changed blocks, and then finds the corresponding blocks in the Merkle tree of the target platform based on the hash values ​​of these blocks.

[0097] After the initial backup is completed, subsequent backups can be performed incrementally using Merkle trees.

[0098] The disk-rsync data synchronization tool can perform incremental synchronization between the source platform and the target platform cluster, transmitting only the changed data blocks, thereby reducing the amount of data transmission.

[0099] S43. Difference Calculation: By comparing the hash values ​​of blocks on the disks of the source platform and the target platform, the disk-rsync data synchronization tool is used to determine which blocks have changed, i.e., the difference blocks; the difference blocks, along with their parent nodes and root nodes, are then transmitted to the target platform.

[0100] S44. Partial data transmission: After receiving the difference blocks, the target platform reconstructs the data by rebuilding these blocks in the Merkle tree;

[0101] This allows the source platform to transmit only the difference blocks instead of the entire file, thereby reducing bandwidth usage and transmission time;

[0102] S45. Verify data migration integrity: After the data is reconstructed on the target platform, the target platform reconstructs the Merkle tree based on the transmitted difference blocks and existing data. The data integrity is verified by comparing the root hash value of the Merkle tree on the target platform with the root hash value of the Merkle tree on the source platform.

[0103] If the root hash value at the destination matches the root hash value provided by the sender, then the data integrity is confirmed.

[0104] If the root hash value at the destination does not match the root hash value provided by the sender, it means that the data has been erroneous, corrupted, or maliciously tampered with during transmission. In this case, the sender is notified to take appropriate measures to retransmit the incremental data.

[0105] The hash verification mechanism based on Merkle trees ensures the integrity of data in the disk-rsync data synchronization tool. After migration, the correct data transmission can be verified by comparing the Merkle trees on the source and target platform clusters.

[0106] Suppose there is a file with the following initial state:

[0107] Data:ABCDEFGH

[0108] This data was used to construct the initial Merkle tree, where each data block was treated as a leaf node. Then, a hash value for each leaf node was calculated using a hash algorithm, and the tree structure was constructed. See [link to documentation]. Figure 4 As shown.

[0109] Now we need to incrementally transmit the IJKL data and reconstruct the Merkle tree structure at the target end. See [link / reference] Figure 5 As shown.

[0110] The sending end (source end) transmits the new root hash value (New Root Hash) to the receiving end (destination end).

[0111] Target side operations:

[0112] 1. The target already has an old Merkle tree and data ABCDEFGH, as well as an old root hash value.

[0113] 2. The target end reconstructs the overall data by merging the old data and the new data blocks: ABCDEFGH+IJKL=ABCDEFGHIJKL.

[0114] 3. The target end can reconstruct the Merkle tree based on the merged data. In this example, the new tree is as follows: Figure 6 As shown;

[0115] The target side calculates the new root hash value `New Root Hash`.

[0116] 4. The target end compares the calculated new root hash value with the root hash value provided by the sender. It can be seen that the newroot hash and the root hash are different. Therefore, the agent will further compare the hash values ​​of the next level. When comparing the hash values ​​of the lowest-level data, the ABCD data are existing data and will not differ. Therefore, the agent will skip data with the same hash value to avoid duplicate transmission, and instead select the IJKL data with different hash values ​​for transmission. The advantage of this approach is that it avoids the transmission of duplicate data, and because hash value calculation is an extremely fast process, it can quickly identify all data that needs incremental synchronization, greatly improving efficiency.

[0117] By applying Merkle trees, disk-rsync achieves efficient incremental backups. It avoids repeatedly transferring existing data, only transferring the changed portions of the source disk, thus reducing the time and bandwidth required for transfers. This approach is particularly useful for long-term backups because it significantly reduces the resources needed for backups.

[0118] This invention also provides a computer device. Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention; see the accompanying drawings. Figure 9As shown, the computer device includes: an input device 23, an output device 24, a memory 22, and a processor 21; the memory 22 is used to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the disk-rsync-based cross-cluster data migration method provided in the above embodiments; wherein the input device 23, the output device 24, the memory 22, and the processor 21 can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0119] The memory 22, as a read / write storage medium for computing devices, can be used to store software programs and computer-executable programs, such as the program instructions corresponding to the disk-rsync-based cross-cluster data migration method described in this embodiment of the invention. The memory 22 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device. Furthermore, the memory 22 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 22 may further include memory remotely located relative to the processor 21, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0120] Input device 23 can be used to receive input digital or character information, and generate key signal inputs related to user settings and function control of the device; output device 24 may include display devices such as a display screen.

[0121] The processor 21 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 22, thereby realizing the above-mentioned cross-cluster data migration method based on disk-rsync.

[0122] The computer equipment provided above can be used to execute the disk-rsync-based cross-cluster data migration method provided in the above embodiments, and has corresponding functions and beneficial effects.

[0123] This invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the disk-rsync-based cross-cluster data migration method provided in the above embodiments. The storage medium can be any type of memory device or storage device, including: mounting media such as CD-ROM, floppy disk, or magnetic tape; computer system memory or random access memory such as DRAM, DDRRAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements; the storage medium may also include other types of memory or combinations thereof; furthermore, the storage medium may reside in a first computer system in which the program is executed, or it may reside in a different second computer system connected to the first computer system via a network (such as the Internet); the second computer system can provide program instructions to the first computer for execution. The storage medium includes two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0124] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the cross-cluster data migration method based on disk-rsync as described in the above embodiments, but can also perform related operations in the cross-cluster data migration method based on disk-rsync provided in any embodiment of the present invention.

[0125] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0126] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A cross-cluster data migration method based on disk-rsync, characterized in that, By using the disk-rsync data synchronization tool in conjunction with the SSH file transfer protocol, and based on OpenStack cloud platform's host snapshots, cloud disk snapshots, BFV, and virtual machine initialization technologies, automated data migration operations can be achieved. The method for achieving automated data migration includes the following steps: S1. The source platform creates a temporary snapshot of the host to be migrated. If the BFI collection method is used, in addition to creating a temporary snapshot of the host, a temporary snapshot of the cloud disk is also created. A cloud disk volume is created from each created temporary snapshot, and the cloud disk volume is mounted to the data synchronization migration gateway virtual machine Agent. S2. After the data synchronization gateway virtual machine completes the data synchronization work through the disk-rsync data synchronization tool, the cloud disk volume mounted on the data synchronization migration gateway virtual machine Agent of the source platform is unmounted, the temporary snapshot and cloud disk volume created by the source platform are deleted, and a snapshot is created on the target platform for the cloud disk volume that has received the data. S3. When starting the host instance on the target platform, create the corresponding cloud disk volume from the cloud disk volume snapshot created in step S2, create a cloud host using BFV, and mount all the cloud disk volumes to the cloud host. S4. By using the Agent program running in the virtual machine of the data synchronization migration gateway, the block devices that need to be synchronized are transferred to the cloud disk volume of the target platform in an incremental synchronization manner based on the SSH file transfer protocol. The implementation process of the incremental synchronization method in the disk-rsync data synchronization tool includes: S41. Initial Backup: The disk on the source platform is divided into fixed-size blocks using the disk-rsync data synchronization tool, and the hash value of each block is calculated. These hash values ​​are used to build the initial Merkle tree, and the root hash value of the entire tree represents the state of the disk on the entire source platform. S42, Incremental Backup: When the data on the disk of the source platform changes, the disk-rsync data synchronization tool first detects the changed blocks, and then finds the corresponding blocks in the Merkle tree of the target platform based on the hash values ​​of these blocks; S43. Difference Calculation: By comparing the hash values ​​of blocks on the disks of the source platform and the target platform, the disk-rsync data synchronization tool is used to determine which blocks have changed, i.e., the difference blocks; the difference blocks, along with their parent nodes and root nodes, are then transmitted to the target platform. S44. Partial data transmission: After receiving the difference blocks, the target platform reconstructs the data by rebuilding these blocks in the Merkle tree; S45. Verify data migration integrity: After the data is reconstructed on the target platform, the target platform rebuilds the Merkle tree based on the transmitted difference blocks and existing data. The data integrity is verified by comparing the root hash value of the Merkle tree on the target platform with the root hash value of the Merkle tree on the source platform.

2. The cross-cluster data migration method based on disk-rsync according to claim 1, characterized in that, The method for transmitting data incrementally via the Agent program in step S4 includes: The data synchronization migration gateway virtual machine Agent constructs a Merkle hash tree for each block device at the underlying level. By comparing the differences in data on the disks of the source platform and the target platform through the Merkle hash tree, only the changed parts are transmitted, thereby achieving efficient incremental backup.

3. The cross-cluster data migration method based on disk-rsync according to claim 2, characterized in that, The rules for verifying data integrity in step S45 include: If the root hash value at the destination matches the root hash value provided by the sender, then the data integrity is confirmed. If the root hash value at the destination does not match the root hash value provided by the sender, it means that the data has been erroneous, corrupted, or maliciously tampered with during transmission. In this case, the sender is notified to take appropriate measures to retransmit the incremental data.

4. The cross-cluster data migration method based on disk-rsync according to claim 1, characterized in that, The automated data migration operation utilizes the Linux file system to complete data reading and writing.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the cross-cluster data migration method based on disk-rsync as described in any one of claims 1-4.

6. A computer device, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the cross-cluster data migration method based on disk-rsync as described in any one of claims 1-4.

Images