A data online migration method and device, electronic equipment and storage medium

Abstract

This application discloses a data online migration method, apparatus, electronic device, and storage medium, relating to the technical field of data storage. The method includes: determining a block device to be migrated in a first storage system (old storage system) and a target block device in a second storage system (new storage system); pre-connecting a host device to the block device to be migrated; disconnecting the data transmission channel between the host device and the block device to be migrated after the host device is connected to the target block device, leaving only the connection between the host device and the block device to be migrated; migrating the device data of the block device to be migrated to the data storage space corresponding to the target block device; thereby enabling the second storage system to respond to and complete business requests (e.g., write requests or read requests) from the host device. This solves the problem of business interruption during data migration of heterogeneous storage systems in related technologies, achieving the technical effect of improving business continuity and data migration reliability.

Description

Technical Field

[0001] This application relates to the technical field of data storage, and in particular to a method, apparatus, electronic device, and storage medium for online data migration. Background Technology

[0002] With the explosive growth of data, there is a need to expand the capacity of storage systems or replace old storage systems with new ones with larger capacities to meet data storage demands. These new and old storage systems are also known as heterogeneous storage systems, meaning the two systems are from different vendors, different versions of the same vendor, or different architectures of the same vendor. Heterogeneous storage systems generally do not support online data synchronization.

[0003] Currently, data migration between heterogeneous storage systems uses an offline migration method. First, the new storage system is linked to the business host. Then, business access is stopped. Data from the old storage system is copied to the new storage system on the business host. Next, the metadata on the business host is updated, and the data and business requests are re-linked to the new storage system. Finally, the system is restarted to restore business access. This offline migration method requires shutting down the devices on the business host. Since the recovery process after restarting is very time-consuming, it causes a prolonged interruption of business operations on the client side.

[0004] Therefore, in related technologies, there is a problem of business interruption when migrating data in heterogeneous storage systems.

[0005] There is currently no effective solution to the problem of business interruption during data migration in heterogeneous storage systems.

[0006] Therefore, it is necessary to improve the relevant technology to overcome the aforementioned defects. Summary of the Invention

[0007] This application provides a data online migration method, apparatus, electronic device, and storage medium to at least solve the problem of service interruption during data migration in heterogeneous storage systems in related technologies.

[0008] This application provides an online data migration method, comprising: determining a target block device after a block device to be migrated in a first storage system is mapped to a second storage system based on a block device mapping relationship, wherein the block device mapping relationship represents the path mapping relationship between the block device to be migrated and the target block device, and the block device information of the block device to be migrated is the same as that of the target block device; disconnecting the data transmission channel between the host device and the block device to be migrated after the host device connects to the target block device; and migrating the device data of the block device to be migrated to the data storage space corresponding to the target block device, wherein the host device has been pre-connected to the block device to be migrated.

[0009] In an exemplary embodiment, before determining the target block device after mapping the block device to be migrated from the first storage system to the second storage system based on the block device mapping relationship, the method further includes: adding an access control list corresponding to the system node of the second storage system to the device data of the block device to be migrated, and mounting the device node of the block device to be migrated to the system node according to the access control list; creating a blank block device in the second storage system according to the block device information of the block device to be migrated, setting the block device information of the block device to be migrated as the block device information of the blank block device to obtain the target block device; setting a migration tag for the target block device, and binding the migration tag and the access path of the block device to be migrated to obtain the block device mapping relationship.

[0010] In one exemplary embodiment, the host device is connected to the target block device by: determining a first device node path of the block device to be migrated from the block device information of the block device to be migrated, and determining a second device node path of the target block device from the block device information of the target block device, wherein the first device node path and the second device node path have the same device number; storing the first device node path and the second device node path in the host device to obtain aggregated path information; and attaching the device node of the target block device to the device node of the host device according to the aggregated path information to connect the host device to the target block device.

[0011] In an exemplary embodiment, migrating device data of the block device to be migrated to the data storage space corresponding to the target block device includes: establishing a data migration thread between the first storage system and the second storage system; using the data migration thread to migrate all data fragments corresponding to the device data of the block device to be migrated to the target block device; and generating a migration result after all data fragments have been migrated, and sending the migration result to the first storage system and the second storage system, wherein the migration result includes at least one of the following: migration completed, migration interrupted, migration failed, or migration in progress.

[0012] In one exemplary embodiment, the method further includes: during the process of migrating device data of the block device to be migrated to the data storage space corresponding to the target block device, determining the data fragment to be written on the target block device based on the first offset value and the first data length of the write request of the host device; and if it is determined that the data fragment to be written already exists in the second storage system, writing the data to be written in the write request to the data fragment to be written.

[0013] In one exemplary embodiment, the method further includes: when it is determined that the data fragment to be written does not exist in the second storage system, determining the original data fragment corresponding to the first offset value from all data fragments of the block device to be migrated, wherein the original data fragment has a start offset value and an end offset value, the first offset value is greater than the start offset value, and the first offset value is less than the end offset value; determining the difference between the first offset value and the start offset value as the relative offset value of the data to be written; determining the data sub-fragment to be updated in the original data fragment, wherein the start offset value of the data sub-fragment to be updated is the same as the relative offset value, and the length between the start offset value and the end offset value of the data sub-fragment to be updated is the same as the length of the first data; updating the data in the data sub-fragment to be updated with the data to be written to obtain a target data fragment; and writing the data of the target data fragment into the target block device.

[0014] In one exemplary embodiment, the method further includes: during the process of migrating device data of the block device to be migrated to the data storage space corresponding to the target block device, determining a first data fragment to be read on the target block device based on a second offset value and a second data length of the read request of the host device; and if it is determined that the first data fragment to be read already exists in the second storage system, acquiring data read from the first data fragment to be read.

[0015] This application also provides an online data migration apparatus, comprising: a determining module, configured to determine, based on a block device mapping relationship, a target block device after a block device to be migrated in a first storage system is mapped to a second storage system, wherein the block device mapping relationship represents a path mapping relationship between the block device to be migrated and the target block device, and the block device information of the block device to be migrated is the same as the block device information of the target block device; a disconnecting module, configured to disconnect the data transmission channel between the host device and the block device to be migrated after the host device is connected to the target block device; and a migration module, configured to migrate the device data of the block device to be migrated to the data storage space corresponding to the target block device, wherein the host device has been pre-connected to the block device to be migrated.

[0016] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described online data migration methods.

[0017] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described online data migration methods.

[0018] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described online data migration methods.

[0019] This application addresses the issue of service interruption during data migration in heterogeneous storage systems by identifying the block device to be migrated in a first storage system (the old storage system) and the target block device in a second storage system (the new storage system). A host device is pre-connected to the block device to be migrated. After the host device connects to the target block device, the data transmission channel between the host device and the block device to be migrated is disconnected, leaving only the connection between the host device and the block device to be migrated. The device data of the block device to be migrated is then migrated to the data storage space corresponding to the target block device. The second storage system then responds to and completes service requests (e.g., write or read requests) from the host device. This solves the problem of service interruption during data migration in related technologies, reduces data loss due to interruptions during data migration, achieves seamless data migration between storage systems, supports online takeover of online data migration between the host system and heterogeneous storage systems, enables seamless service switching on the host side, improves the continuity of services on the host side, and further enhances the reliability of data migration. Attached Figure Description

[0020] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A hardware structure block diagram of a computer terminal for an online data migration method provided in this application embodiment;

[0022] Figure 2 A flowchart illustrating an online data migration method provided in this application embodiment;

[0023] Figure 3 This application provides a schematic diagram illustrating the principle of online data migration.

[0024] Figure 4 This is a schematic diagram of a write request provided in an embodiment of this application;

[0025] Figure 5 This is a schematic diagram of an online data migration device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0027] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0028] The methods and embodiments provided in this application can be executed on a computer terminal, a cloud server corresponding to the computer terminal, or a similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for an online data migration method according to an embodiment of this application. Figure 1 As shown, a computer terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor or programmable logic device) and a memory 104 for storing data are also shown. The computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0029] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the online data migration method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby realizing the aforementioned online data migration method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the card in the visible area via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0030] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0031] This embodiment provides an online data migration method applied to the aforementioned computer terminal. The following execution flow provides a detailed description. Figure 2 This is a flowchart of an online data migration method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0032] Step S202: Determine the target block device after the block device to be migrated in the first storage system is mapped to the second storage system according to the block device mapping relationship. The block device mapping relationship represents the path mapping relationship between the block device to be migrated and the target block device. The block device information of the block device to be migrated is the same as the block device information of the target block device.

[0033] Step S204: After the host device connects to the target block device, disconnect the data transmission channel between the host device and the block device to be migrated;

[0034] Step S206: Migrate the device data of the block device to be migrated to the data storage space corresponding to the target block device. The host device has been pre-connected to the block device to be migrated.

[0035] Through the above steps, by identifying the block device to be migrated in the first storage system (the old storage system) and the target block device in the second storage system (the new storage system), the host device is pre-connected to the block device to be migrated. After the host device connects to the target block device, the data transmission channel between the host device and the block device to be migrated is disconnected, leaving only the connection between the host device and the block device to be migrated. The device data of the block device to be migrated is then migrated to the data storage space corresponding to the target block device. Thus, the second storage system responds to and completes the business requests (e.g., write requests or read requests) from the host device. This solves the problem of business interruption during data migration of heterogeneous storage systems in related technologies, reduces data loss due to interruptions during data migration, achieves seamless data migration between storage systems, supports online takeover of online data migration between the host system and heterogeneous storage systems, realizes seamless business switching on the host side, improves the continuity of services on the host side, and further enhances the reliability of data migration.

[0036] It should be noted that the aforementioned block device can refer to physical storage devices such as hard disk drives (HDDs) and solid-state drives (SSDs), or logical storage devices in a virtualized environment. In the operating system, a block device is represented as a specific file system node, such as / dev / sda. Block device information includes attributes such as the device type, size, and identifier; this information is crucial for the correct reading and writing of data. In this embodiment, by creating a target block device in the second storage system with the same block device information as the block device to be migrated in the first storage system, the consistency of block device attributes before and after data migration can be ensured, thereby avoiding data incompatibility or read / write errors caused by differences in device attributes.

[0037] In an exemplary embodiment, before determining the target block device after mapping the block device to be migrated from the first storage system to the second storage system based on the block device mapping relationship, an access control list corresponding to the system node of the second storage system can be added to the device data of the block device to be migrated. The device node of the block device to be migrated is then mounted to the system node according to the access control list. A blank block device is created in the second storage system based on the block device information of the block device to be migrated, and the block device information of the block device to be migrated is set as the block device information of the blank block device to obtain the target block device. A migration tag is set for the target block device, and the migration tag is bound to the access path of the block device to be migrated to obtain the block device mapping relationship. By pre-creating a target block device with the same block device information as the first storage system on the second storage system and performing path mapping on the host device, normal access of the host device during the data migration process is achieved, improving the user experience.

[0038] In storage systems, ACLs (Access Control Lists) are used to control which hosts can access specific storage resources, such as LUNs (Logical Unit Numbers).

[0039] The ACL information of the system nodes of the storage system specifically includes a series of rules. These rules determine whether to allow the host to perform read and write operations on the storage resource based on the host's IQN (Initiator Qualified Name, which is the host's unique identifier for iSCSI) or WWN (World Wide Name, which is the unique identifier for the FC protocol).

[0040] In heterogeneous storage migration scenarios, adding the ACL information of the new storage system node to the block device to be migrated in the old storage system allows the new storage system to legally access data on the old storage system, facilitating subsequent data transfer. Therefore, by adding the ACL corresponding to the system node of the second storage system to the device data of the block device to be migrated, access permissions of the target block device can be controlled, ensuring that only authorized host devices can access it, thus increasing the security of data migration. Simultaneously, by copying the block device information of the block device to be migrated to a blank block device in the second storage system, the target block device created has the same attributes as the block device to be migrated, guaranteeing data consistency during migration.

[0041] Simultaneously, setting migration tags and binding them to access paths identifies the data migration status, helping subsequent operations (such as data read / write) determine whether special handling is needed, such as prioritizing the use of migrated data. This embodiment solves the preparatory work before data migration, ensures that no access conflicts occur during the data migration process, and improves the security of data migration.

[0042] Among them, you can use query commands such as "REPORT LUN" and "INQUIRY" to query the block device information of the block device to be migrated.

[0043] In one exemplary embodiment, the host device is connected to the target block device by: determining a first device node path of the block device to be migrated from the block device information of the block device to be migrated, and determining a second device node path of the target block device from the block device information of the target block device, wherein the first device node path and the second device node path have the same device number; storing the first device node path and the second device node path in the host device to obtain aggregated path information; and attaching the device node of the target block device to the device node of the host device according to the aggregated path information to connect the host device to the target block device.

[0044] The device number is a unique identifier for each block device. By ensuring that the first device node path and the second device node path have the same device number, the host device can access the block device through the same path before and after data migration. This avoids access interruption caused by path changes and ensures a smooth transition of data migration.

[0045] The above aggregated path information represents the merged path information of two device nodes, so that the host device can intelligently switch access paths during data migration, giving priority to accessing the target block device. When the target block device is unavailable or the data migration is not completed, it can fall back to the block device to be migrated, which not only improves the reliability of data migration, but also reduces system downtime.

[0046] In an exemplary embodiment, the process of migrating device data of the block device to be migrated to the data storage space corresponding to the target block device includes: establishing a data migration thread between the first storage system and the second storage system; using the data migration thread to migrate all data fragments corresponding to the device data of the block device to be migrated to the target block device; and generating a migration result after all data fragments have been migrated, and sending the migration result to the first storage system and the second storage system, wherein the migration result includes at least one of the following: migration completed, migration interrupted, migration failed, or migration in progress.

[0047] It should be noted that the data migration thread is the specific execution unit responsible for data copying during the data migration process. It can be a separate process or thread used to migrate the data fragments of the block device to be migrated to the target block device one by one.

[0048] Data sharding involves dividing large amounts of data into smaller blocks. This partitioning facilitates parallel processing and improves migration speed. Once all data shards have been migrated, the resulting migration information can be promptly fed back to both storage systems, enabling them to monitor the data migration process.

[0049] This embodiment significantly improves the speed of data migration through thread-level concurrent processing. At the same time, the real-time feedback mechanism of migration results makes the entire migration process more transparent and controllable, improving the efficiency and security of data migration.

[0050] Optionally, after all data shard migrations are completed, disconnect the block device to be migrated from the second storage system and destroy the thread resources of the data migration thread.

[0051] In an exemplary embodiment, further, during the process of migrating the device data of the block device to be migrated to the data storage space corresponding to the target block device, the write request is used to determine the write data fragment corresponding to the write request on the target block device based on the first offset value and the first data length of the write request of the host device; if it is determined that the write data fragment already exists in the second storage system, the write data in the write request is written to the write data fragment.

[0052] Here, the offset value refers to the data position relative to the starting position of the block device, while the data length refers to the size of the data being written. In this embodiment, during the data migration process, for write requests from the host device to the target block device, if the corresponding data fragment to be written already exists on the target block device, the data is written directly, simplifying unnecessary data copying operations and improving the efficiency of data migration.

[0053] This allows for the processing of write operations on the host device during the data migration process, improving data consistency during the migration process.

[0054] Optionally, after writing the data to be written in the write request to the data fragment to be written, a write success notification message can be sent to the host device.

[0055] In an exemplary embodiment, optionally, if it is determined that the data shard to be written does not exist in the second storage system, the original data shard corresponding to the first offset value is determined from all data shards of the block device to be migrated, wherein the original data shard has a start offset value and an end offset value, the first offset value is greater than the start offset value, and the first offset value is less than the end offset value; the difference between the first offset value and the start offset value is determined as the relative offset value of the data to be written; the data sub-shard to be updated in the original data shard is determined, wherein the start offset value of the data sub-shard to be updated is the same as the relative offset value, and the length between the start offset value and the end offset value of the data sub-shard to be updated is the same as the first data length; the data in the data sub-shard to be updated is updated using the data to be written to obtain the target data shard; and the data of the target data shard is written to the target block device. In this embodiment, during data migration, if a write request from the host device does not exist on the target block device, the original data fragment containing the write location will be found in the target block device. Then, the relative offset value will be calculated to locate the specific sub-fragment of data to be updated, and the data will be updated before being written to the target block device. This embodiment ensures that even if data migration is incomplete, write operations on the host device will not affect the progress of data migration, while also guaranteeing data consistency and integrity.

[0056] Optionally, in this embodiment, writing the data of the target data fragment into the target block device means writing a data fragment corresponding to the target data fragment into the target block device.

[0057] In an exemplary embodiment, during the migration of device data from the block device to be migrated to the data storage space corresponding to the target block device, a first data fragment to be read on the target block device is determined based on a second offset value and a second data length of the read request from the host device. If it is determined that the first data fragment to be read already exists in the second storage system, data is retrieved from the first data fragment to be read. In this embodiment, during data migration, for read requests from the host device to the second storage system, when a corresponding data fragment to be read exists on the target block device, data is directly read from it, avoiding repeated readings from the block device to be migrated and improving the efficiency of data migration.

[0058] Optionally, if it is determined that the first data fragment to be read does not exist in the second storage system, a second data fragment to be read is determined from all data fragments of the block device to be migrated. The starting offset of the second data fragment to be read is the same as the second offset, and the length between the starting offset and the ending offset of the second data fragment to be read is the same as the second data length. Data is then obtained from the second data fragment to be read. In this embodiment, during data migration, if a corresponding data fragment to be read does not exist on the target block device for a read request from the host device, the corresponding data fragment to be read is determined from the first storage system and read. During data migration, accurate data is obtained based on the read operation of the host device, ensuring the continuity of services on the host side while enhancing the reliability of data migration.

[0059] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] In this embodiment, combined with Figure 3 The specific application environment on which the online data migration method depends is provided, and the architecture of the specific application environment is described here.

[0061] like Figure 3 As shown, the system includes an old storage system (first storage system), a new storage system (second storage system), and client hosts (corresponding host devices). The new storage system includes functional modules such as a heterogeneous storage volume management module, a data migration module, and a business request processing module. The client host's functional module is the host-side path control module. All these functional modules are developed using C++. The data migration process is explained below with reference to the implementation steps.

[0062] Step 1: The heterogeneous storage volume management module adds the ACL information of the new storage system node to the volume (which can be represented as LUN1, corresponding to the block device to be migrated) of the old storage system. This mounts the LUN1 of the old storage system to the node of the new storage system, enabling the new storage system to access, control, and store the data in LUN1.

[0063] Among them, you can use commands such as "REPORT LUN, INQUIRY, READ_CAPACITY" to query block device information such as LUN_ID (logical unit number), capacity, and WWN (World Wide Name) of storage system LUN1.

[0064] In this context, LUN_ID serves as an identifier in the storage system, uniquely identifying a storage device or logical partition. Each LUN is assigned a LUN_ID to help the host determine which specific storage device to communicate with.

[0065] Capacity refers to the storage capacity of a LUN1, that is, the size of data that can be stored, measured in bytes, and can be expressed in GB (gigabytes), TB (terabytes), etc.

[0066] WWN serves as a globally unique identifier for storage devices, used in network storage environments such as FC (Fibre Channel). During heterogeneous storage system migration, WWN helps ensure that newly created LUN2 and the original LUN1 are recognized as different paths of the same device on the host, thus enabling multi-path management.

[0067] Step 2: The heterogeneous storage volume management module creates a new volume (which can be represented as LUN2, corresponding to the target block device) in the new storage system based on the block device information obtained in Step 1. The WWN, capacity, and LUN_ID of LUN2 are consistent with those of LUN1. In the new storage system, a migration target mark is added to LUN2 to indicate that the volume is a target volume for heterogeneous storage migration.

[0068] Obtain the file descriptor (fd) of LUN1 mounted in step 1, denoted as "fd_src", which is used to access the new storage system and read data from LUN1 in the old storage system. Pass fd_src to LUN2 as its data source.

[0069] Step 3: The service redirection process is completed by the host-side path control module.

[0070] Mount the new storage system's LUN2 to the host side, disconnect the host side from the old storage system (i.e., disconnect LUN1 from the host side), and notify the data migration module to start the data migration process.

[0071] Only the path to LUN2 is retained on the host side, and business requests on the host side will be automatically sent to the new storage system through the LUN2 link.

[0072] In this step, LUN2 and LUN1 share the same LUN_ID and other block device information. The host side identifies LUN2 and LUN1 as multiple different paths to the same block device using the block device ID, thus obtaining the aggregated path for LUN1 and LUN2. When LUN1 is disconnected and the connection is switched to the LUN2 path, service requests are sent to the new storage system through the LUN2 path. This allows both the block device to be migrated and the target block device to be connected to the host side simultaneously. The host side aggregates the two paths to the same block device, and subsequent path disconnection enables a seamless service redirection process at the service level.

[0073] Step 4: After disconnecting LUN1 from the host in Step 3, the data migration module performs data migration. Based on the target volume tag and fd_src of LUN2, LUN1 is linked to the new storage system, and the data pointed to by fd_src is migrated to LUN2. For example, the data migration thread reads data from LUN1 one by one in the form of data shards and writes it to the new storage system LUN2. Once all data shards have completed the migration process, the data migration is reported as complete. The link between LUN1 and the new storage system is then disconnected, and the thread resources of the data migration thread are destroyed.

[0074] Step 5: The business request processing module processes the read and write requests on the host side.

[0075] For write request W1 sent from the host side to LUN2, based on the offset of W1 (i.e. Figure 4 (off1) and data length (i.e.) Figure 4 The corresponding data shard is calculated using the length (len). The system then checks if the corresponding data shard exists in the new storage system.

[0076] If the data shard already exists (generated by writing LUN1 data read by the data migration module into LUN2 or generated by step ③), then write the data of W1 into the corresponding data shard.

[0077] like Figure 4 As shown in steps ① to ④, if it is determined that the data fragment does not exist, the corresponding data fragment in LUN1 is read through fd_src to obtain data DATA1. Then, W1 and DATA1 are merged according to the offset to be written by W1 to form new data DATA2.

[0078] Specifically, the relative offset of the data on the corresponding data fragment in LUN1 is calculated by using the offset off2 of LUN1 and the offset off1 of W1. Then, based on len, the data to be written by W1 overwrites the data in DATA1, forming data DATA2, which is then written to the new storage system. A write request for W1 can then be successfully returned to the host.

[0079] For read request R1 sent from the host to LUN2, similarly, calculate the corresponding data shard based on the offset and length of R1. Then, query whether the corresponding data shard exists in the new storage system.

[0080] If the data shard already exists, the data is read directly from the new storage system; if the data shard does not exist, the data with the corresponding offset and length in LUN1 is read using fd_src based on the offset and length of the read request R1. Finally, a success message for read request R1 is sent to the host, along with the data read.

[0081] In this embodiment, taking the iSCSI application scenario as an example, four modules work together to link the volume of the old storage system to the new storage system. The new storage system generates a new volume based on the original block device of the old storage system, and then maps the new volume to the host side through iSCSI and other methods. After that, the host-side path control module, data migration module and business request processing module jointly perform data migration and business request processing between heterogeneous storage systems, thereby completing the online data migration between different storage systems.

[0082] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0083] Embodiments of this application also provide an online data migration device, such as... Figure 5 As shown, it includes:

[0084] The determination module 52 is used to determine the target block device after the block device to be migrated in the first storage system is mapped to the second storage system according to the block device mapping relationship. The block device mapping relationship represents the path mapping relationship between the block device to be migrated and the target block device. The block device information of the block device to be migrated is the same as the block device information of the target block device.

[0085] Disconnect module 54 is used to disconnect the data transmission channel between the host device and the block device to be migrated after the host device is connected to the target block device;

[0086] Migration module 56 is used to migrate the device data of the block device to be migrated to the data storage space corresponding to the target block device, wherein the host device has been pre-connected to the block device to be migrated.

[0087] The aforementioned apparatus identifies the block device to be migrated in the first storage system (the old storage system) and the target block device in the second storage system (the new storage system). The host device is pre-connected to the block device to be migrated. After the host device connects to the target block device, the data transmission channel between the host device and the block device to be migrated is disconnected, leaving only the connection between the host device and the block device to be migrated. The device data of the block device to be migrated is then migrated to the data storage space corresponding to the target block device. The second storage system then responds to and completes business requests (e.g., write or read requests) from the host device. This solves the problem of business interruption during data migration in heterogeneous storage systems, reduces data loss due to interruptions during migration, achieves seamless data migration between storage systems, supports online takeover of online data migration between the host system and heterogeneous storage systems, enables seamless business switching on the host side, improves the continuity of host-side services, and further enhances the reliability of data migration.

[0088] In an exemplary embodiment, before determining the target block device after mapping the block device to be migrated from the first storage system to the second storage system based on the block device mapping relationship, the determining module is further configured to: add an access control list corresponding to the system node of the second storage system to the device data of the block device to be migrated, and mount the device node of the block device to be migrated to the system node according to the access control list; create a new blank block device in the second storage system according to the block device information of the block device to be migrated, set the block device information of the block device to be migrated as the block device information of the blank block device, and obtain the target block device; set a migration tag for the target block device, and bind the migration tag and the access path of the block device to be migrated to obtain the block device mapping relationship.

[0089] In one exemplary embodiment, the disconnect module is further configured to connect the host device to the target block device by: determining a first device node path of the block device to be migrated from the block device information of the block device to be migrated, and determining a second device node path of the target block device from the block device information of the target block device, wherein the first device node path and the second device node path have the same device number; storing the first device node path and the second device node path in the host device to obtain aggregated path information; and attaching the device node of the target block device to the device node of the host device according to the aggregated path information to connect the host device to the target block device.

[0090] In an exemplary embodiment, the migration module is further configured to: establish a data migration thread between the first storage system and the second storage system; use the data migration thread to migrate all data fragments corresponding to the device data of the block device to be migrated to the target block device; and generate a migration result after all data fragments have been migrated, and send the migration result to the first storage system and the second storage system, wherein the migration result includes at least one of the following: migration completed, migration interrupted, migration failed, or migration in progress.

[0091] In an exemplary embodiment, the migration module is further configured to, during the process of migrating device data of the block device to be migrated to the data storage space corresponding to the target block device, determine the data fragment to be written on the target block device based on the first offset value and the first data length of the write request of the host device; and, if it is determined that the data fragment to be written already exists in the second storage system, write the data to be written in the write request to the data fragment to be written.

[0092] In an exemplary embodiment, the migration module is further configured to: if it is determined that the data shard to be written does not exist in the second storage system, determine the original data shard corresponding to the first offset value from all data shards of the block device to be migrated, wherein the original data shard has a start offset value and an end offset value, the first offset value is greater than the start offset value and less than the end offset value; determine the difference between the first offset value and the start offset value as the relative offset value of the data to be written; determine the data sub-shard to be updated in the original data shard, wherein the start offset value of the data sub-shard to be updated is the same as the relative offset value, and the length between the start offset value and the end offset value of the data sub-shard to be updated is the same as the length of the first data; update the data in the data sub-shard to be updated using the data to be written to obtain a target data shard; and write the data of the target data shard to the target block device.

[0093] In an exemplary embodiment, the migration module is further configured to, during the process of migrating device data of the block device to be migrated to the data storage space corresponding to the target block device, determine the first data fragment to be read on the target block device based on the second offset value and the second data length of the read request of the host device; and, if it is determined that the first data fragment to be read already exists in the second storage system, obtain the data read from the first data fragment to be read.

[0094] For a description of the features in the embodiments of the above-described online data migration device, please refer to the relevant descriptions in the embodiments of the above-described online data migration method, which will not be repeated here.

[0095] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described online data migration method embodiments.

[0096] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described online data migration method embodiments at runtime.

[0097] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0098] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described online data migration method embodiments.

[0099] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described online data migration method embodiments.

[0100] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0101] The above provides a detailed description of the online data migration method, apparatus, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for online data migration, characterized in that, include: Based on the block device mapping relationship, the target block device after the block device to be migrated in the first storage system is determined to be mapped to the second storage system. The block device mapping relationship represents the path mapping relationship between the block device to be migrated and the target block device. The block device information of the block device to be migrated is the same as the block device information of the target block device. After the host device connects to the target block device, the data transmission channel between the host device and the block device to be migrated is disconnected; The device data of the block device to be migrated is migrated to the data storage space corresponding to the target block device, and the host device has been pre-connected to the block device to be migrated; Before determining the target block device to be migrated from the first storage system to the second storage system based on the block device mapping relationship, the method further includes: A new access control list corresponding to the system node of the second storage system is added to the device data of the block device to be migrated, and the device node of the block device to be migrated is mounted to the system node according to the access control list; a blank block device is created in the second storage system according to the block device information of the block device to be migrated, and the block device information of the block device to be migrated is set as the block device information of the blank block device to obtain the target block device; a migration tag is set for the target block device, and the migration tag is bound to the access path of the block device to be migrated to obtain the block device mapping relationship; The host device is connected to the target block device in the following manner: A first device node path of the block device to be migrated is determined from the block device information of the block device to be migrated, and a second device node path of the target block device is determined from the block device information of the target block device, wherein the first device node path and the second device node path have the same device number; the first device node path and the second device node path are stored in the host device to obtain aggregated path information; the device node of the target block device is mounted to the device node of the host device according to the aggregated path information to connect the host device to the target block device; In the process of migrating the device data of the block device to be migrated to the data storage space corresponding to the target block device, the data fragment to be written on the target block device is determined according to the first offset value and the first data length of the write request of the host device. If it is determined that the data fragment to be written does not exist in the second storage system, the original data fragment corresponding to the first offset value is determined from all data fragments of the block device to be migrated. The original data fragment has a start offset value and an end offset value, the first offset value is greater than the start offset value, and the first offset value is less than the end offset value. The difference between the first offset value and the start offset value is determined as the relative offset value of the data to be written. A data sub-fragment to be updated is determined from the original data fragment. The start offset value of the data sub-fragment to be updated is the same as the relative offset value, and the length between the start offset value and the end offset value of the data sub-fragment to be updated is the same as the length of the first data fragment. The data in the data sub-fragment to be updated is updated using the data to be written to obtain the target data fragment. The data of the target data fragment is written to the target block device.

2. The online data migration method according to claim 1, characterized in that, Migrating the device data of the block device to be migrated to the data storage space corresponding to the target block device includes: Establish a data migration thread between the first storage system and the second storage system; The data migration thread is used to migrate all data fragments corresponding to the device data of the block device to be migrated to the target block device. After all data shards have been migrated, a migration result is generated and sent to the first storage system and the second storage system. The migration result includes at least one of the following: migration completed, migration interrupted, migration failed, or migration in progress.

3. The online data migration method according to claim 1, characterized in that, The method further includes: If it is determined that the data fragment to be written already exists in the second storage system, the data to be written in the write request is written to the data fragment to be written.

4. The online data migration method according to claim 1, characterized in that, The method further includes: During the process of migrating the device data of the block device to be migrated to the data storage space corresponding to the target block device, the first data fragment to be read on the target block device is determined according to the second offset value and the second data length of the read request of the host device. If it is determined that the first data fragment to be read already exists in the second storage system, the data read from the first data fragment to be read is obtained.

5. A data online migration device, characterized in that, include: The determination module is used to determine the target block device after a block device to be migrated in the first storage system is mapped to the second storage system based on the block device mapping relationship. The block device mapping relationship represents the path mapping relationship between the block device to be migrated and the target block device. The block device information of the block device to be migrated is the same as the block device information of the target block device. The disconnect module is used to disconnect the data transmission channel between the host device and the block device to be migrated after the host device is connected to the target block device; The migration module is used to migrate the device data of the block device to be migrated to the data storage space corresponding to the target block device, wherein the host device has been pre-connected to the block device to be migrated; The determining module is further configured to: add an access control list corresponding to the system node of the second storage system to the device data of the block device to be migrated; mount the device node of the block device to be migrated to the system node according to the access control list; create a blank block device in the second storage system according to the block device information of the block device to be migrated; set the block device information of the block device to be migrated to the block device information of the blank block device to obtain the target block device; set a migration tag for the target block device; and bind the migration tag and the access path of the block device to be migrated to obtain the block device mapping relationship. The disconnect module is further configured to connect the host device to the target block device in the following manner: A first device node path of the block device to be migrated is determined from the block device information of the block device to be migrated, and a second device node path of the target block device is determined from the block device information of the target block device, wherein the first device node path and the second device node path have the same device number; the first device node path and the second device node path are stored in the host device to obtain aggregated path information; the device node of the target block device is mounted to the device node of the host device according to the aggregated path information to connect the host device to the target block device; The migration module is further configured to determine the data fragment to be written on the target block device based on the first offset value and the first data length of the write request of the host device during the process of migrating the device data of the block device to be migrated to the data storage space corresponding to the target block device. If it is determined that the data fragment to be written does not exist in the second storage system, the original data fragment corresponding to the first offset value is determined from all data fragments of the block device to be migrated. The original data fragment has a start offset value and an end offset value, the first offset value is greater than the start offset value, and the first offset value is less than the end offset value. The difference between the first offset value and the start offset value is determined as the relative offset value of the data to be written. A data sub-fragment to be updated is determined from the original data fragment. The start offset value of the data sub-fragment to be updated is the same as the relative offset value, and the length between the start offset value and the end offset value of the data sub-fragment to be updated is the same as the length of the first data fragment. The data in the data sub-fragment to be updated is updated using the data to be written to obtain the target data fragment. The data of the target data fragment is written to the target block device.

6. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the online data migration method as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the online data migration method as described in any one of claims 1 to 4.

Images