A method and system for distributed migration of stored data

Abstract

The present application relates to the field of data storage, in particular to a kind of distributed storage data migration method and system.Mainly include: obtaining the data copy of source end needing data migration, using the read IO interface of source end to read data copy, wherein, data copy is LUN or file;Network is opened between source end and destination, and data copy is sent to destination in the form of byte stream;The write IO interface of destination writes the data copy received into storage, and generates other required copies.The present application can improve the migration efficiency, directly open the storage of source end and destination, and use cache as intermediate storage medium to avoid intermediate landing, and use the IO interface of source end and destination to process byte stream data, which shields the problem of heterogeneous storage migration.

Description

Technical Field

[0001] This invention relates to the field of data storage, and in particular to a method and system for migrating distributed storage data. Background Technology

[0002] Distributed storage is characterized by data being scattered across a large number of disks, with all disks participating in data reading and writing simultaneously, greatly increasing read and write performance. Furthermore, due to its decentralized design, it can be horizontally scaled by adding nodes during use. Additionally, to enhance disaster recovery capabilities, distributed storage often incorporates a multi-replica mechanism, ensuring multiple copies of the data to cope with disk or node failures.

[0003] Distributed storage architecture offers numerous benefits, but it also presents significant challenges for data migration.

[0004] 1. Data in distributed storage is often broken down into data blocks distributed across various disks. Therefore, during data migration, it's impossible to pinpoint the exact location of the data, and the migration process cannot be performed in the same way as copying a single file in a typical storage system. Existing solutions generally employ remote storage replication of raw disks to perform a complete copy of all raw disks, achieving the migration of all broken data blocks on the disks. However, this approach cannot migrate individual LUNs or files.

[0005] 2. The distributed data structure of distributed storage offers advantages in data read / write performance and scalability, but it also increases the scope of impact from disk failures. Therefore, to address this, distributed storage introduces a multi-replica mechanism. The principle of multi-replica is to create multiple identical copies of a single piece of data and store them on different disk storage media. When one disk fails, only one copy is damaged, and users can still access the data through the other copies, improving the disaster recovery capabilities of distributed storage. While multi-replica provides data fault tolerance and security, it also significantly increases the amount of data stored. The original data volume of a single file is now multiplied and stored on the distributed storage disks, resulting in a data volume many times larger than before. If data migration were performed using the traditional direct copy method, all copies would be copied indiscriminately, further doubling the amount of data copied.

[0006] 3. In existing data migration methods, migration software installed on the host mounts the data disk to be migrated to the host, and copies the data to the destination storage through point-to-point copying between hosts. During the migration process, the data needs to go through the following path: source distributed storage -> source host -> destination host -> target distributed storage, which spans multiple nodes. The migration speed is limited by the performance of the host's CPU, network card, network, etc., resulting in a slow migration speed that cannot meet the needs of migrating large amounts of data.

[0007] 4. Existing backup software tools back up the volume stored at the source, copy all backup data, and then restore it at the target storage. Both backup and restoration are time-consuming and involve multiple copies of backup data, which cannot meet the needs of large-scale data migration.

[0008] 5. For the migration of heterogeneous distributed storage, data migration cannot be directly achieved because the source and destination remote replication interfaces are inconsistent.

[0009] Therefore, how to overcome the shortcomings of existing technologies and solve the problem that multiple copies of data for multiple backups are required during the data migration process of existing distributed storage is a problem to be solved in this technical field. Summary of the Invention

[0010] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention solves the problem that multiple copies of data from multiple backups are required during the data migration process of distributed storage.

[0011] The embodiments of the present invention adopt the following technical solutions:

[0012] In a first aspect, the present invention provides a method for migrating distributed storage data, specifically as follows:

[0013] Preferably, a copy of the data that needs to be migrated from the source end is obtained, and the data copy is read using the read IO interface of the source end. The data copy is either a LUN or a file. The network between the source end and the destination end is established, and the data copy is sent to the destination end as a byte stream. The write IO interface of the destination end writes the received data copy into the storage and generates other copies as needed.

[0014] Preferably, the data replica is read using the source-side read IO interface, which specifically includes: calling the source-side storage data replica query interface to obtain the list of data replicas in the current storage; and, based on the unique identifier of the data replica to be migrated, calling the distributed storage data replica read IO interface to read the data in the data replica, converting the data into a byte stream and writing it into the source-side cache.

[0015] Preferably, establishing a network connection between the source and destination ends specifically includes: directly establishing a network connection from the source to the destination end in the storage network; and / or deploying a dedicated data migration network to directly connect the source and destination ends.

[0016] Preferably, the network connection from the source to the destination is directly established in the storage network, specifically including: when the source and the destination are in the same Layer 2 network, the connection is established directly in the router through the Layer 2 network; when the source and the destination are not in the same Layer 2 network, the connection is established through the Layer 3 network.

[0017] Preferably, the data copy is sent to the destination in the form of a byte stream, which specifically includes: dividing the byte stream data in the source cache into data blocks of the same size, and sending the data blocks directly to the destination cache through the established network connection.

[0018] Preferably, sending data blocks directly to the destination buffer via the established network connection also includes: marking the positions of the header and tail of each data block during data transmission; recording the position of the tail of the transmitted data block when data transmission is interrupted or data errors occur; and resuming transmission from the position of the tail of the transmitted data block or starting from the header of the erroneous data when data transmission is resumed.

[0019] Preferably, the destination end's write IO interface writes the received data copy into storage, specifically including: the destination end cache converts the byte stream data into a data copy in the destination end's data format by calling the destination end's write IO interface, and writes the data copy to the destination end's target storage location.

[0020] Preferably, the destination end's write IO interface writes the received data copy into storage, and also includes: performing data integrity verification on the received data copy to verify the consistency of data before and after migration.

[0021] Preferably, generating other required copies specifically includes: the destination end copying the data written to the destination end according to its own copying strategy to generate corresponding copies.

[0022] On the other hand, the present invention also provides a distributed storage data migration system, including a source end and a destination end. Specifically, both the source end and the destination end have a distributed storage system and a cache. The source end and the destination end interact with each other through network transmission. The source end and the destination end perform data migration according to the distributed storage data migration method provided in the first aspect.

[0023] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows: migrating only one data copy reduces the amount of data migration and improves migration efficiency; directly connecting the source and destination storage and using cache as an intermediate storage medium avoids intermediate disk writes; and using the respective IO interfaces of the source and destination to process byte stream data shields the problem of heterogeneous storage migration. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0025] Figure 1 A flowchart illustrating a method for distributed storage data migration provided in an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of the architecture of encapsulating the heterogeneous database driver into a storage driver management module in the method provided in the embodiments of the present invention;

[0027] Figure 3 A flowchart illustrating another method for distributed storage data migration provided in this embodiment of the invention;

[0028] Figure 4 This is a schematic diagram of a system architecture for distributed storage data migration provided in an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram illustrating the system operation process of distributed storage data migration provided in an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0031] This invention is an architecture of a specific functional system. Therefore, the specific embodiments mainly describe the functional logic relationship of each structural module, and do not limit the specific software and hardware implementation methods.

[0032] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] Example 1:

[0034] Existing distributed storage data migration solutions typically achieve this through raw disk copying or by generating data snapshots and then restoring them using snapshot technology. These methods do not select the data copies to be migrated and require multiple disk writes, resulting in low data migration efficiency.

[0035] In this embodiment, the read and write I / O interfaces of the source and destination storage are invoked respectively to complete the data read operation at the source storage, the sending and receiving of the data stream, and the writing of data at the destination storage, thereby realizing the data migration process. By utilizing the distributed storage's own read and write interfaces, the problems of scattered data distribution in distributed storage, making indexing difficult, and the differences in data structures between heterogeneous storage systems are cleverly avoided. Furthermore, in this embodiment, only one data copy is migrated; other copies are generated at the destination using the distributed storage's own copy generation strategy, reducing the amount of data migration and further improving data migration efficiency.

[0036] like Figure 1 As shown, the specific steps of the distributed storage data migration method provided in this embodiment of the invention are as follows:

[0037] Step 101: Obtain a copy of the data that needs to be migrated from the source end, and read the data copy using the source end's read IO interface.

[0038] In this embodiment, for multiple distributed data blocks with replicas, only one data replica is obtained for data migration, instead of migrating all data replicas as in the prior art.

[0039] First, the source storage's data replica query interface is called to obtain the list of data replicas currently in the storage. Then, based on the unique identifier of the data replica to be migrated, the distributed storage's data replica read IO interface is called to read the data from the data replica, convert the data into a byte stream, and write it into the source cache.

[0040] In the method provided in this embodiment, the data copy can be a Logical Unit Number (LUN) or a file, rather than the complete data of a storage disk as in the prior art. A LUN is a logical partition used for storage management of physical disks or other storage media, and can be considered a container for data. For distributed storage, a container LUN may contain multiple physical disks. To improve read / write performance and disk lifespan, in general distributed storage strategies, file data is usually not stored on a single disk, but rather broken down into data blocks, with each data block stored on different disks within the same LUN according to a certain pattern. Therefore, migrating data by LUN unit rather than physical disk can ensure file data integrity to a certain extent. Specifically, firstly, the LUN query interface of the source storage is called to obtain the list of LUNs in the current storage, one LUN is selected for data migration, and the LUN ID is used as a unique identifier. Then, the LUN read IO interface of the distributed storage is called to read the data in the LUN into the cache. To further reduce the amount of data being migrated and ensure file integrity, migration can also be performed on a file-by-file basis. For each file, a unique identifier can be obtained by calling the IO read / write interface to read, write, and migrate the file.

[0041] To avoid the efficiency loss caused by disk writes during data migration, this embodiment uses a cache with higher read / write speeds instead of disks with lower read / write speeds, such as snapshot disks, for intermediate storage of data copies. Data is read into the cache through the I / O interface of the source distributed storage system and then transmitted to the destination via the network. In actual implementation, the cache space capacity can be set as needed, allocating a limited cache size to the migration tool application process. In practice, write management can be performed using a message queue-like approach. Data blocks enter the FIFO queue in the order they are read. When the queue is full, later data blocks need to wait. After the read / write request at the head of the queue is completed and written to the cache, the new data block is added to the tail of the queue.

[0042] Existing LUN snapshot export and import data migration solutions, while avoiding multiple copies and reducing the amount of data to be migrated, require the creation of additional snapshot disks. Exporting and importing both involve disk read / write operations, increasing the number of data read / write cycles. This solution directly reads and writes data through the storage interface, eliminating the snapshot export / import process and thus avoiding multiple snapshot read / write operations.

[0043] Step 102: Establish network connectivity between the source and destination ends, and send the data copy to the destination end as a byte stream.

[0044] In the method provided in this embodiment, the source storage and the destination storage are directly connected via the network. The migration path is: source distributed storage → destination distributed storage. This is different from the existing scheme that uses source and destination hosts as intermediaries, and it has less dependence on the performance of the source and destination hosts.

[0045] Data migration requires establishing network connectivity between the source and destination. This can be achieved by directly establishing a network connection between the source and destination within the storage network. Storage networks typically have large capacity and operate on a single business model, thus generally meeting the network performance requirements for data migration without the need for additional networks. Since network bandwidth and latency are determined by the network service provider, a dedicated data migration network can be deployed to directly connect the source and destination when the amount of data to be migrated is large or conditions permit. This can be achieved by adding a network interface card (NIC) to the server and connecting it to a dedicated access switch, setting up a dedicated data migration network plane for data migration. Deploying a leased line can further reduce network interference, increase bandwidth, and lower latency.

[0046] The specific method of network connection establishment depends on the network architecture of the source and destination. When the source and destination are not in the same Layer 2 network, a Layer 3 network connection is established. When the source and destination are in the same Layer 2 network, a Layer 2 network connection is established directly within the router. This embodiment does not have special requirements for the network connection establishment method; it only needs to ensure that one network is established. The bandwidth and latency of the established network determine the migration speed. In practical implementation, to further improve the reliability of network connections during migration or to further improve transmission efficiency, multiple network connections can be established to provide backup network links, or parallel transmission can be performed on multiple network links to increase transmission bandwidth.

[0047] Once the network connection is established, the byte stream data in the source cache is divided into data blocks of uniform size, and these data blocks are sent directly to the destination cache via the established network connection. The entire data migration process is performed as a byte stream, with both the sending and receiving ends using caches for temporary data storage; data is not written to disk during the copying process.

[0048] In heterogeneous distributed storage systems, the source and destination may have different storage architectures or formats, and may also have different storage interface formats. To address these issues, this embodiment converts the data content of the data copy into a byte stream and transmits it to the destination in byte stream format. Byte streams are a universal data format, and various programming languages ​​typically have corresponding processing interfaces, such as Java's InputStream and OutputStream classes, and C++'s std::IOstream library. Regardless of the data format or interface format used by the source and destination, conversion between their own data format and the byte stream format can be achieved, enabling interoperability between different data formats and interface formats.

[0049] Step 103: The destination end's write I / O interface writes the received data copy into storage and generates other required copies.

[0050] After receiving the data copy, the destination buffer converts the byte stream data into a data copy in the destination's data format by calling the destination's write I / O interface, and writes the data copy to the destination's target storage location. Since the data copy received by the destination is in byte stream format, regardless of the interface format and storage format used by the destination, this universal data format can be processed and converted into its own format.

[0051] Similar to the source end, the destination end also uses a cache as a temporary storage medium for data, requiring only one final write to disk, thus avoiding efficiency losses caused by data being written to disk during migration. In practice, the cache space capacity can be set as needed, allocating a limited-size cache to the migration tool application process. When the cache capacity is insufficient, the byte stream needs to wait for a copy of the data in the cache to be written to the destination storage before receiving new data. Once the data in the cache has been read and written, new byte stream data will enter the cache.

[0052] Distributed storage typically employs a replication mechanism to enhance reliability. In this embodiment, only one data copy is migrated. To maintain the multi-replica backup mechanism in the distributed storage system, additional copies need to be generated at the destination. The destination storage replicates the data written to it according to its own replication policy, generating corresponding copies. The destination storage is configured with a replication policy; when data is divided into blocks and stored in the target storage, the target storage replicates the database from the received data copy according to its own replication policy, generating corresponding copies.

[0053] Furthermore, since this embodiment only migrates one data copy and uses byte stream format data blocks for transmission, data loss or errors may occur due to factors such as network instability during data transmission. To ensure that the migrated data is consistent with the source data, data integrity verification is required on the received data copy to verify the consistency of data before and after migration. Specifically, common data integrity verification methods such as parity check, MD5 check, and CRC check can be used. If the verification passes, it indicates that the migrated data is correct; if the verification fails, it indicates that the data contains errors or is missing, and it needs to be corrected or retransmitted.

[0054] After steps 101-103 provided in this embodiment, by migrating only one data copy and generating a copy using the destination's copy strategy, the amount of data migrated is reduced; by using a cache as a temporary storage medium and directly connecting the source and destination networks, efficiency losses caused by data transfer and disk writes are reduced; by using direct IO read / write between the source and destination storage and using byte streams for data migration, inconsistencies in the interfaces and formats of the source and destination are avoided.

[0055] Furthermore, in steps 101 and 103, source and destination I / O interfaces were used for data reading and writing, respectively. I / O interfaces are fundamental functional interfaces that every distributed storage system must expose. Data read and write I / O are inherent interfaces of distributed storage, and external devices can read and write data in the distributed storage system through its I / O interfaces. However, the names and parameters of different distributed storage interfaces are not uniform, and the data formats used are also inconsistent, depending on the device and system platform. For example, Ceph's read interface requires importing the librbd tool and using the rbd_aIO_read interface to read data, while Huawei FusIOnStorage requires Huawei's driver for reading, and the read interface name may be different. When performing I / O read and write operations, if the method of calling the Ceph interface is used without modification to call the FusIOnStorage read interface, it will be impossible to read data. In practical applications, to improve the code portability of the method provided in this embodiment, design patterns such as the facade pattern can be used to encapsulate the IO interfaces of different distributed storage systems into IO interface driver modules. For drivers of different storage systems, the corresponding IO read / write interfaces are encapsulated, and a unified IO read / write module is used to achieve read / write operations on different distributed storage systems, further resolving the storage interface differences at the software level. For example... Figure 2As shown, the drivers for heterogeneous databases such as Ceph, FusIOnStorage, and Netapp can be encapsulated into a storage driver management module. The storage driver management module provides a unified source and destination I / O interface for users to call, making the differences in underlying storage structure and interface transparent to users.

[0056] In practical use, a failure in the data copying link can cause byte stream blocking; on the other hand, data retransmission is also required in cases of data loss or data transmission errors that fail verification. To avoid reduced migration efficiency due to retransmission, this embodiment also provides a breakpoint resume function. The tool marks the location where the byte stream is interrupted or erroneous, and when the link is restored, it will resume the interrupted byte stream from the point of interruption.

[0057] Specifically, such as Figure 3 As shown, you can complete the resume download by following these steps.

[0058] Step 201: During data transmission, mark the positions of the header and tail of each data block.

[0059] In this embodiment, the data copy is transmitted sequentially in the form of byte stream data blocks. Recording the header and tail positions of the data blocks allows us to determine the current data transmission progress and the location of breakpoints. In practical applications, data offsets based on the LUN start address or the file header can be used as position markers.

[0060] Step 202: When data transmission is interrupted or data errors occur, record the position of the end of the transmitted data block.

[0061] When data transmission is interrupted or errors occur, the already transmitted data does not need to be retransmitted; only the untransmitted data needs to be transmitted, or the erroneous data needs to be retransmitted. Therefore, it is necessary to record the position of the end of the transmitted data block to determine the data that has been transmitted.

[0062] Step 203: After data transmission is restored, continue transmission from the end of the already transmitted data block or start retransmission from the beginning of the erroneous data.

[0063] In cases of interrupted data transmission, already transmitted data does not need to be retransmitted; only subsequent data needs to be transmitted. Therefore, it is necessary to locate the end of the transmitted data block and retransmit from the next data block. In cases of data errors, already transmitted data needs to be retransmitted. Therefore, it is necessary to locate the header of the erroneous data block and retransmit from the header of the erroneous data block. For consecutive data blocks in a LUN or a file, the end of the previous data block and the header of the next data block are usually in the same position; the end position of the previous data block can be used as the header position of the next data block. For non-consecutive data, the positions of the data blocks need to be recorded separately to avoid transmission order errors.

[0064] Steps 201-203 enable the function of resuming interrupted downloads, improving the efficiency and stability of data migration.

[0065] The distributed storage data migration method provided in this embodiment has the following advantages compared with existing distributed storage system data migration methods.

[0066] (1) Shielding Heterogeneity: Data read and write operations are performed using the local distributed storage system's own read and write I / O interfaces at both the source and destination ends, effectively avoiding data structure differences between heterogeneous storage systems and enabling data migration across heterogeneous storage systems. Simultaneously, byte streams are used for transmission, allowing different storage architectures and I / O interfaces to be directly processed.

[0067] (2) No disk write: The copied data stream is not written to disk, but is directly copied using the storage cache, which saves the time-consuming problem of multiple data read and write caused by data writing to disk and greatly improves the efficiency of data copying.

[0068] (3) Shorten the path: Data copying is performed directly between the two storage devices at the source and destination ends. It only requires connecting the storage networks of the two storage devices, without having to read the data to the host device first. This eliminates the process of forwarding through the host device and shortens the data copying path.

[0069] (4) Reduced performance loss: Since there is no snapshot export and import process, the performance loss of storage devices is reduced.

[0070] (5) Reduce data copying volume: Distributed storage generally has multiple replicas. This solution only needs to copy one replica. The replicas will be automatically generated according to the configuration of the destination storage.

[0071] Example 2:

[0072] Based on the distributed storage data migration method provided in Embodiment 1 above, the present invention also provides a system for implementing the distributed storage data migration method described above, such as... Figure 4The diagram shown is a schematic representation of the system architecture of an embodiment of the present invention.

[0073] The system provided in this embodiment includes a source end and a destination end, and data needs to be migrated from the source end to the destination end. In actual implementation scenarios, the source end and the destination end can be two physically independent devices, or they can be two independent distributed storage systems on the same physical device.

[0074] Both the source and destination ends have distributed storage systems and caches. For example... Figure 4 As shown, both the source and destination ends manage at least one distributed storage node server through a storage gateway server. The distributed storage node server is connected to the storage gateway server via a network or other data communication lines. The distributed storage node server contains non-volatile memory for distributed data storage, such as at least one disk storage device, tape storage device, or other non-volatile solid-state storage device. In this embodiment, the cache used by the source and destination ends can use various commonly used cache devices, such as RAM, DRAM, SRAM, SDRAM, FLASH, etc., to provide higher read / write speeds compared to the disk devices of the distributed storage system itself, thereby improving the efficiency of data migration.

[0075] During data migration, data exchange occurs between the source and destination endpoints via network transmission. Specific network connection methods include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0076] Data migration is performed at the source and destination ends according to the distributed storage data migration method provided in Example 1. In specific implementation, a source migration tool capable of performing the data copy reading and sending function in step 101 can be deployed on the source distributed system, and a destination migration tool capable of performing the data receiving and copy generation function in step 103 can be deployed on the destination distributed storage system.

[0077] like Figure 5 As shown, the data migration process in Example 1 can be completed through the following steps.

[0078] Step 301: According to the method provided in step 102, establish the network connection used for data migration between the source and destination ends, and establish a pass-through network connection between the source storage and the destination storage.

[0079] Step 302: The source migration tool reads a copy of the data from the source distributed storage system and puts it into the source cache, as provided in step 101.

[0080] Step 303: The source migration tool converts the data copy into a byte stream format as provided in step 102, and sends the byte stream format data copy directly to the destination through the network connection established in step 301.

[0081] Step 304: The destination migration tool receives a data copy in byte stream format according to the method provided in step 103, converts it into the data format used by the local end, writes it into the destination storage, and generates a corresponding number of copies according to the local copy strategy.

[0082] As shown in steps 301-304, the data migration system provided in this embodiment can complete the data migration method provided in embodiment 1, realizing rapid data migration between distributed storage systems with different architectures. In practical use, where there is no conflict, the above steps can also be combined with other methods provided in embodiment 1, or optimized using existing technologies.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for migrating distributed storage data, characterized in that, Specifically, it includes: Obtain a copy of the data that needs to be migrated from the source end, use the read IO interface of the source end to read the data in the data copy, convert the data into a byte stream and write it into the source end buffer. The data copy is a logical unit number (LUN) or a file. Connect the source and destination networks and send a data copy to the destination as a byte stream; when the source and destination are in the same Layer 2 network, connect directly within the router to establish a Layer 2 network connection; when the source and destination are not in the same Layer 2 network, connect through a Layer 3 network. The destination end's write IO interface writes the received data copy into the storage. The destination end then replicates the data copy written to it according to its own replication strategy, generating the corresponding copy. The destination also uses a cache as a temporary storage medium for data, which only needs to be written to the disk once.

2. The method for distributed storage data migration according to claim 1, characterized in that, The process of reading a data copy using the source's read I / O interface specifically includes: Call the data replica query interface of the source storage to obtain the list of data replicas in the current storage; Based on the unique identifier of the data copy to be migrated, the data copy read IO interface of the distributed storage is called to read the data in the data copy, and the data is converted into a byte stream and written into the source cache.

3. The method for distributed storage data migration according to claim 1, characterized in that, Sending the data copy to the destination as a byte stream specifically includes: The byte stream data in the source cache is divided into data blocks of the same size, and the data blocks are sent directly to the destination cache through the established network connection.

4. The method for distributed storage data migration according to claim 3, characterized in that, The step of sending the data block directly to the destination cache via the established network connection also includes: During data transmission, the positions of the header and tail of each data block are marked; When data transmission is interrupted or data errors occur, record the position of the end of the transmitted data block; Once data transmission is restored, it resumes transmission from the end of the already transmitted data block or starts retransmission from the beginning of the erroneous data block.

5. The method for distributed storage data migration according to claim 1, characterized in that, The destination end's write I / O interface writes a copy of the received data into storage, specifically including: The destination cache converts the byte stream data into a data copy in the destination data format by calling the destination's write IO interface, and writes the data copy to the destination's target storage location.

6. The method for distributed storage data migration according to claim 1, characterized in that, The destination end's write I / O interface writes the received data copy into storage, and also includes: Perform data integrity verification on the received data copy to verify the consistency of data before and after the migration.

7. A system for distributed storage data migration, characterized in that, Including the source and destination, specifically: Both the source and destination ends have distributed storage systems and high-speed caches. The source and destination ends exchange data through network transmission. The source and destination ends perform data migration according to the distributed storage data migration method provided in any one of claims 1-6.

Images