Online data migration method, apparatus and device

Abstract

Embodiments of the present application provide an online data migration method, device and equipment. The method comprises: receiving a data migration instruction sent by a user end; creating a migration pair composed of a source volume and a target volume according to the data migration instruction; performing background migration of data in the source volume to the target volume, and monitoring the migration progress, wherein during the background migration process, continuously receiving data write requests sent by a host through a logical volume; when the migration progress is less than a first threshold, issuing the data write requests to the source volume; when the migration progress is greater than or equal to the first threshold, issuing the data write requests to the source volume and the target volume simultaneously; after the data migration is completed, switching the mapping relationship of the logical volume from the source volume to the target volume, and keeping the globally unique identifier presented by the logical volume unchanged, so that the target volume takes over the source volume to provide storage services to the host as the back-end storage of the logical volume. The method provided by the present application effectively reduces the load of the storage system and improves the resource utilization.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to an online data migration method, apparatus, and device. Background Technology

[0002] In storage systems, data migration is a critical operation for responding to business growth, equipment upgrades, and resource consolidation. As business grows, the capacity or performance of existing storage volumes may no longer meet demand, necessitating the migration of the logical volume backend storage to a new target volume. To ensure business continuity at the upper layers, the migration process must support seamless online access from the host.

[0003] In related technologies, to achieve uninterrupted business migration, storage systems typically create a target volume at the logical layer, initiate full data replication in the background, and employ a dual-write mechanism to handle data write requests sent by the host: each data write request is simultaneously sent to both the source and target volumes, and a response is only returned to the host after both the source and target volumes confirm successful writes. After migration, the target volume takes over the service by switching mapping relationships. However, existing online data migration methods result in severe storage backend bandwidth consumption and a significant increase in host write latency, leading to low storage system resource utilization. Summary of the Invention

[0004] This application provides an online data migration method, apparatus, and device. By introducing a dynamic write strategy based on migration progress, when the background migration progress is less than a first threshold, data write requests sent by the host are only sent to the source volume to ensure write performance in the initial stage of migration. After the migration progress reaches or exceeds the first threshold, data write requests are sent to both the source and target volumes simultaneously to ensure that subsequent writes are synchronized on both volumes, laying the foundation for eventual consistency. This avoids redundant writes to data blocks that have not yet been copied by the background migration task in the early stage of migration, effectively reducing the storage system load and improving resource utilization.

[0005] In a first aspect, embodiments of this application provide an online data migration method, the method comprising:

[0006] Receive data migration instructions sent by the client, wherein the data migration instructions indicate that the backend storage of a logical volume is migrated from the source volume to the target volume, and the logical volume is currently mapped to the source volume;

[0007] Create a migration pair consisting of a source volume and a target volume according to the data migration instructions;

[0008] The data in the source volume is migrated to the target volume in the background, and the migration progress is monitored. During the background migration, data write requests sent by the host through the logical volume are continuously received, and the host's access to the logical volume remains available.

[0009] When the migration progress is less than the first threshold, the data write request is sent to the source volume, and a success response is returned to the host after the source volume is successfully written.

[0010] When the migration progress is greater than or equal to the first threshold, the data write request is sent to both the source volume and the target volume at the same time, and a success response is returned to the host after both the source volume and the target volume are successfully written.

[0011] After the data migration is complete, the mapping relationship of the logical volume is switched from the source volume to the target volume, while keeping the globally unique identifier of the logical volume unchanged, so that the target volume can take over as the backend storage of the logical volume to provide storage services to the host.

[0012] By dynamically switching between single-write and dual-write strategies based on the migration progress, and seamlessly switching logical volume mapping relationships while maintaining their globally unique identifiers after the migration is complete, the storage backend migration is achieved without the host business being aware of or interrupted, while ensuring strong data consistency. This significantly improves the availability, reliability, and operational flexibility of the storage system.

[0013] In one possible implementation, the process of migrating data from the source volume to the target volume in the background also includes:

[0014] Monitor the host's read / write load on the logical volume;

[0015] When the read / write load exceeds the preset load threshold, the data transfer rate of the background migration will be limited to no more than the first rate threshold.

[0016] When the read / write load is less than or equal to a preset load threshold, the data transfer rate of the background migration is set to a second rate threshold, where the second rate threshold is greater than the first rate threshold.

[0017] Through the aforementioned load-aware dynamic rate adjustment mechanism, the storage system can adaptively optimize the execution efficiency of background data migration while ensuring that the performance of host services is not affected. During the data migration process, it effectively balances low interference to upper-layer applications with the high throughput of the storage system.

[0018] In one possible implementation, it also includes:

[0019] Before starting the background migration, perform a snapshot operation on the source volume to generate a migration start point snapshot;

[0020] During the migration process, all data blocks modified by newly generated data write requests after the migration start point snapshot are recorded as difference data blocks;

[0021] After the background migration is completed and before the mapping relationship is switched, copy the difference data blocks to the target volume to ensure that the target volume and the source volume are completely consistent.

[0022] All new data written to the host after the snapshot from the migration start point is created is accurately recorded and completely written to the target volume through final synchronization, ensuring no user data loss. In scenarios where host services continue to run and source volume data is constantly modified, online data migration with zero downtime, zero data loss, and strong consistency is achieved, effectively guaranteeing host service continuity and data integrity.

[0023] In one possible implementation, after sending the data write request to both the source and target volumes simultaneously, the method further includes:

[0024] When writing to the source volume fails, a write error is returned and the original mapping relationship is maintained;

[0025] If the source volume is successfully written but the target volume fails to be written, the background migration is paused and a fault alarm message is generated.

[0026] During a target volume failure, data write requests received during the target volume failure will only be sent to the source volume to maintain service availability;

[0027] After the target volume is recovered from the failure, the differential data blocks are copied back to the target volume and the migration process continues.

[0028] Through fault tolerance and recovery mechanisms, even in the event of a temporary failure of the target volume, the storage system can still ensure the continuous availability of host services, prevent data loss, and automatically connect the migration process after the fault is recovered, thereby improving the overall reliability and stability of online data migration.

[0029] In one possible implementation, after creating a migration pair consisting of a source volume and a target volume according to the data migration instructions, the process further includes:

[0030] Verify that the capacity of the target volume is not less than the sum of the used capacity of the source volume and the preset safety margin;

[0031] Set the block size, access permissions, and cache configuration of the target volume to be consistent with those of the source volume;

[0032] Reserve metadata storage space in the target volume for recording migration progress and verification information.

[0033] By performing initialization verification and configuration steps, the storage system can establish a basic environment with sufficient capacity, compatible attributes, and traceable status before the migration officially begins, effectively reducing the possibility of migration failure and providing necessary support for subsequent incremental synchronization, consistency switching, and fault recovery.

[0034] In one possible implementation, the method further includes:

[0035] When switching mapping relationships, the target volume is marked as the inherited volume of the source volume, and the association information between the inherited volume and the source volume is persistently stored. The association information includes the binding relationship between the globally unique identifier of the source volume and the internal identifier of the target volume. The inherited volume refers to the target volume that logically inherits the identity information of the source volume.

[0036] Update the path mapping table used to maintain the mapping relationship, and change the backend storage path corresponding to the logical volume from the source volume to the target volume;

[0037] After the switch is complete, disconnect the source volume from the logical volume, and retain the data and metadata of the source volume;

[0038] After the storage controller that maintains the mapping relationship fails and recovers, the target volume is identified as the inherited volume of the source volume based on the association information of the persistent storage, and the mapping relationship between the logical volume and the target volume is rebuilt to maintain the globally unique identifier presented to the outside world of the logical volume unchanged.

[0039] It ensures the continuity of logical volume services and the stability of data access even in abnormal scenarios such as storage controller failure, path interruption, or storage node failure, thereby improving the self-healing and fault recovery capabilities of the storage system under partial failures and meeting the stringent requirements of upper-layer applications for business continuity and data persistence.

[0040] Secondly, embodiments of this application provide an online data migration apparatus, the apparatus comprising:

[0041] The receiving module is used to receive data migration instructions sent by the user terminal, wherein the data migration instructions indicate that the back-end storage of a logical volume is migrated from the source volume to the target volume, and the logical volume is currently mapped to the source volume;

[0042] The creation module is used to create migration pairs consisting of a source volume and a target volume based on data migration instructions;

[0043] The migration module is used to migrate data from the source volume to the target volume in the background and monitor the migration progress. During the background migration process, it continuously receives data write requests sent by the host through the logical volume, and the host's access to the logical volume remains available.

[0044] The single-write module is used to send a data write request to the source volume when the migration progress is less than the first threshold, and return a success response to the host after the source volume is successfully written.

[0045] The dual-write module is used to simultaneously send data write requests to both the source volume and the target volume when the migration progress is greater than or equal to the first threshold, and return a success response to the host after both the source volume and the target volume have been successfully written.

[0046] The switching module is used to switch the mapping relationship of logical volumes from the source volume to the target volume after the data migration is completed, while keeping the globally unique identifier of the logical volume unchanged, so that the target volume can take over as the back-end storage of the logical volume to provide storage services to the host.

[0047] In one possible implementation, the migration module is specifically used for:

[0048] Monitor the host's read / write load on the logical volume;

[0049] When the read / write load exceeds the preset load threshold, the data transfer rate of the background migration will be limited to no more than the first rate threshold.

[0050] When the read / write load is less than or equal to a preset load threshold, the data transfer rate of the background migration is set to a second rate threshold, where the second rate threshold is greater than the first rate threshold.

[0051] In one possible implementation, the device further includes a consistency module, which is specifically used for:

[0052] Before starting the background migration, perform a snapshot operation on the source volume to generate a migration start point snapshot;

[0053] During the migration process, all data blocks modified by newly generated data write requests after the migration start point snapshot are recorded as difference data blocks;

[0054] After the background migration is completed and before the mapping relationship is switched, copy the difference data blocks to the target volume to ensure that the target volume and the source volume are completely consistent.

[0055] In one possible implementation, the dual-write module is also used for:

[0056] When writing to the source volume fails, a write error is returned and the original mapping relationship is maintained;

[0057] If the source volume is successfully written but the target volume fails to be written, the background migration is paused and a fault alarm message is generated.

[0058] During a target volume failure, data write requests received during the target volume failure will only be sent to the source volume to maintain service availability;

[0059] After the target volume is recovered from the failure, the differential data blocks are copied back to the target volume and the migration process continues.

[0060] In one possible implementation, the apparatus further includes a migration preparation module, which is specifically used for:

[0061] Verify that the capacity of the target volume is not less than the sum of the used capacity of the source volume and the preset safety margin;

[0062] Set the block size, access permissions, and cache configuration of the target volume to be consistent with those of the source volume;

[0063] Reserve metadata storage space in the target volume for recording migration progress and verification information.

[0064] In one possible implementation, the switching module is also used for:

[0065] When switching mapping relationships, the target volume is marked as the inherited volume of the source volume, and the association information between the inherited volume and the source volume is persistently stored. The association information includes the binding relationship between the globally unique identifier of the source volume and the internal identifier of the target volume. The inherited volume refers to the target volume that logically inherits the identity information of the source volume.

[0066] Update the path mapping table used to maintain the mapping relationship, and change the backend storage path corresponding to the logical volume from the source volume to the target volume;

[0067] After the switch is complete, disconnect the source volume from the logical volume, and retain the data and metadata of the source volume;

[0068] After the storage controller that maintains the mapping relationship fails and recovers, the target volume is identified as the inherited volume of the source volume based on the association information of the persistent storage, and the mapping relationship between the logical volume and the target volume is rebuilt to maintain the globally unique identifier presented to the outside world of the logical volume unchanged.

[0069] Thirdly, embodiments of this application provide an online data migration device, comprising:

[0070] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method shown in any of the first aspects.

[0071] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the method shown in any of the first aspects.

[0072] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method shown in any of the first aspects.

[0073] The online data migration method, apparatus, and device provided in this application divide the migration process into two stages: single-write and dual-write. The write strategy is dynamically adjusted based on a first threshold of the migration progress, maximizing write performance in the early stages and ensuring strong data consistency in the later stages. Simultaneously, by maintaining the globally unique identifier of the logical volume and seamlessly switching mapping relationships, the method completely eliminates the business interruption or reconnection issues caused by storage switching in traditional migration solutions. This method effectively solves the long-standing performance-consistency contradiction in online data migration, improving the resource utilization of the storage system and the user experience. Attached Figure Description

[0074] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0075] Figure 1 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0076] Figure 2 A flowchart illustrating an online data migration method provided in an embodiment of this application;

[0077] Figure 3 A flowchart illustrating a data transmission rate adjustment method provided in an embodiment of this application;

[0078] Figure 4 A flowchart illustrating a background data migration method based on snapshot and differential data synchronization provided in this application embodiment;

[0079] Figure 5 A flowchart illustrating another online data migration method provided in this application embodiment;

[0080] Figure 6 This is a schematic diagram of the structure of an online data migration device provided in an embodiment of this application;

[0081] Figure 7 This is a schematic diagram of the structure of an online data migration device provided in an embodiment of this application.

[0082] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0083] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0084] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0085] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0086] To facilitate understanding, the following will be combined with... Figure 1 The application scenarios applicable to the embodiments of this application will be described.

[0087] Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. Please refer to [link / reference]. Figure 1 The application scenario includes the user terminal 101, the storage system 102, and the host 103.

[0088] Specifically, the user terminal 101 is used to issue data migration instructions (such as creating migration pairs, starting / pausing migration, etc.) to the storage system 102. Specifically, it can be a management terminal, a World Wide Web (Web) interface, a command-line interface (CLI) tool, or an application programming interface (API) call client.

[0089] Host 103 serves as the runtime environment for upper-layer business applications. It continuously initiates input / output (I / O) requests (including read and write) to the logical volume through the block device driver, and maintains seamless access to the logical volume throughout the migration process.

[0090] Storage system 102, as the executing entity of the method of this invention, can integrate functional units such as management module, migration module, protocol module, and I / O processing engine. It is responsible for receiving data migration instructions, creating migration pairs, performing background data replication, dynamically adjusting write strategies (single-write / dual-write phase switching), maintaining mapping relationships, and seamlessly switching the backend storage of the logical volume to the target volume after the migration is completed. The entire migration process ensures that the host 103's access to the logical volume is continuously available, and that the globally unique identifier (World Wide Name, WWN) presented to the outside world of the logical volume remains unchanged, thereby achieving data migration with zero business interruption.

[0091] In storage systems, data migration is a critical operation for addressing business growth, equipment upgrades, and resource consolidation. With the rapid development of technologies such as cloud computing, big data, and artificial intelligence, enterprise business data volumes are growing exponentially, placing unprecedented capacity pressure and performance challenges on storage systems. When the capacity of an existing storage volume reaches its limit, performance bottlenecks occur, or hardware upgrades are necessary, the backend storage of the logical volume it hosts must be migrated to a new target volume. Simultaneously, modern enterprises have increasingly stringent requirements for business continuity. Storage migration operations must be completed without affecting the normal operation of upper-layer business processes; that is, the migration process must support seamless online access from the host, ensuring high availability of data services and business continuity.

[0092] In related technologies, to achieve uninterrupted business migration, storage systems typically employ the following technical solutions: First, a target volume is created at the logical layer as the target storage entity for migration. This target volume has the same logical volume attributes and access interfaces as the source volume. Then, a full data replication task is initiated in the background, copying all data blocks from the source volume sequentially or by priority to the target volume. During this process, to ensure data consistency, the storage system typically uses a dual-write mechanism to handle data write requests sent by the host: each data write request is simultaneously sent to both the source and target volumes, and a response is only returned to the host after both the source and target volumes confirm successful writes, while simultaneously marking the corresponding data blocks as synchronized. Once the background full replication is complete, the storage system uses an atomic mapping switch operation, allowing the target volume to take over the services of the source volume, completing the entire migration process. However, existing online data migration methods have significant technical drawbacks: the use of a dual-write mechanism throughout the migration process leads to severe consumption of network bandwidth at the storage backend, as each write operation needs to be executed on two storage entities, resulting in a significant increase in I / O load; at the same time, host write latency also increases significantly, because the storage system must wait for both storage entities to complete the write operation before returning a response, which is particularly evident in high-concurrency write scenarios; in addition, the dual-write mechanism also increases the processing burden on the storage controller, resulting in low overall resource utilization of the storage system and affecting the normal service capability of the storage system during the migration process.

[0093] This application proposes an online data migration method. By introducing a dynamic write strategy based on migration progress, an effective balance between migration performance and data consistency is achieved. Specifically, during the background migration task execution, the method monitors migration progress indicators in real time. When the background migration progress is less than a preset first threshold (e.g., 70%), data write requests sent by the host are only sent to the source volume, avoiding redundant write operations on the target volume. This maximizes the host's write performance in the early stages of migration and reduces network bandwidth consumption and I / O load on the storage backend. When the migration progress reaches or exceeds the first threshold, the storage system automatically switches the data write strategy to dual-write mode, that is, subsequent data write requests are sent to both the source and target volumes simultaneously. This ensures that all newly generated write data remains synchronized on the source and target volumes in the final stage of migration, laying a solid foundation for final data consistency. By employing this phased dynamic write strategy, this application effectively avoids redundant writes to data blocks that have not yet been copied by the background migration task during the initial migration phase, significantly reducing the overall load on the storage system and improving resource utilization. Simultaneously, by switching to dual-write mode in a timely manner at the end of the migration, it ensures complete data consistency between the source and target volumes upon completion of the migration, guaranteeing the reliability and data security of the migration. Furthermore, this method exhibits good scalability, allowing for flexible adjustment of the first threshold based on different business scenarios and performance requirements, achieving a balance between migration performance and data consistency.

[0094] The method described in this application will now be illustrated through specific embodiments. It should be noted that the following embodiments may exist independently or in combination with each other; identical or similar content will not be repeated in different embodiments.

[0095] Figure 2 This is a flowchart illustrating an online data migration method provided in an embodiment of this application. Please refer to... Figure 2 The method may include:

[0096] S201, Receive data migration instructions sent by the user terminal.

[0097] The execution entity in this application embodiment can be a storage controller in a storage system (such as the storage controller in the aforementioned storage system 102), or it can be an online data migration device installed in the storage controller. The online data migration device can be implemented in software or in a combination of software and hardware, and is used to execute the various steps of the online data migration method.

[0098] The data migration instruction specifies that the backend storage of a logical volume be migrated from the source volume to the target volume, and the logical volume is currently mapped to the source volume.

[0099] This embodiment provides an online data migration method for seamlessly migrating the backend storage of a logical volume from one physical volume to another without affecting the continuity of host services.

[0100] Specifically, the user terminal can be a management terminal, a web interface, a command-line tool, or an API client, which can initiate a data migration request to the storage system. This data migration instruction explicitly indicates that the backend storage of a logical volume needs to be migrated from its currently associated source volume to a specified target volume.

[0101] Among them, a logical volume is a virtual storage unit accessed by the host through a block device driver, presenting a unified WWN to the outside world; while the source volume and the target volume are both actual physical or logical storage entities within the storage system, serving as the data carrying medium before and after the migration, respectively.

[0102] S202. Create a migration pair consisting of a source volume and a target volume according to the data migration instruction.

[0103] In this embodiment, a migration pair is a basic logical unit within the storage system used to manage online data migration tasks. Each migration pair uniquely associates with a pair of volumes: the source volume currently serving as the logical volume backend and the target volume used to receive the data. The migration pair records the correspondence between the source and target volumes, the current migration status, progress information, configuration parameters, and metadata information. During the creation of a migration pair, the storage system verifies the status and basic compatibility of the source and target volumes (e.g., whether the target volume exists and is available) to ensure that the migration task can be started safely.

[0104] In this embodiment of the application, the storage system divides the state of the migration pair into five states: Normal, Syncing, Sync_Pause, Divided, and Sync_Interrupted.

[0105] Specifically, the normal state indicates that the initial and incremental synchronization has been completed and the pair is ready to perform the split operation; the synchronizing state indicates that the migration pair is currently performing background data synchronization; the synchronization paused state indicates that the user has actively paused the synchronization operation for the migration pair; the split state indicates that the mapping switch has been completed and the target volume has officially taken over the source volume to provide services to the outside world; the synchronization interrupted state indicates that the synchronization of the migration pair was forced to be interrupted due to an anomaly (such as link failure, volume unavailability, etc.) during the synchronization process.

[0106] It is important to note that before performing any operation (such as pause, resume, split, or delete) on a migration pair, the storage system must first verify its current state to ensure the legality of the operation and data consistency. For example: splitting is only allowed when the migration pair is in a normal state; that is, when the migration pair is in a normal state, pausing synchronization, modifying, splitting, and deleting operations are permitted. When in a synchronization state, modification, pausing, and deleting are permitted, but splitting is not. When synchronization is paused, modification, resuming synchronization, and deleting operations are permitted. Migration pairs in a split state typically only support deletion operations. When synchronization is interrupted, modification and pausing synchronization are permitted. Although further checks on volume health status, data protection policies, and other constraints are performed during actual execution, this pre-judgment based on the migration pair's state is the fundamental mechanism for ensuring data consistency between the source and target volumes throughout the entire migration cycle.

[0107] During the creation of a migration pair, upon receiving a data migration instruction from the user, the storage system first enters the migration pair creation phase. This creation phase specifically includes three steps: target volume preparation, constraint verification, and migration pair initialization and persistence.

[0108] Target volume preparation refers to the storage system first determining whether a new target volume needs to be created based on data migration instructions. If a new target volume needs to be created, a new volume with a compatible structure is created as the target volume, using the source volume's capacity, block size, access permissions, caching strategy, and other key attributes as templates. If the target volume already exists, its original data is cleared before creating the migration pair to prevent residual data from interfering with the migration process, thereby ensuring the integrity and consistency of logical volume data before and after the migration.

[0109] Constraint verification refers to the comprehensive verification of the current status of the source and target volumes by the storage system after the target volume is determined. This includes, but is not limited to, whether the volume is available, whether there are active snapshots or copy task conflicts, and whether the target volume capacity meets the requirements.

[0110] Migration pair initialization and persistence refers to the formal creation of the migration pair after successful verification, and setting its initial state to "synchronizing". Subsequently, the complete information of the migration pair, including the source volume identifier (ID), target volume ID, initial state, WWN binding relationship, and metadata reserved space location, can be persistently saved to non-volatile storage media to ensure that the migration task can still be accurately identified and recovered even after the storage controller restarts or the storage system fails.

[0111] Through the above mechanism, the migration pair not only serves as the running context of the migration task, but also as the core object driven by the state machine, providing reliable basic support for subsequent background migration, write strategy switching, splitting operations and fault self-healing.

[0112] S203. Perform background migration of data from the source volume to the target volume and monitor the migration progress.

[0113] During the background migration process, the system continuously receives data write requests from the host through the logical volume, and the host's access to the logical volume remains available.

[0114] After creating the migration pair consisting of the source and target volumes, a background data migration process can be initiated, with continuous monitoring of the migration progress. During this stage, the storage system can copy data block by block from the source volume to the target volume in the background, while continuously receiving and processing I / O requests from the host. Specifically, the host's read and write access to the logical volume remains available throughout the entire migration process without being aware of changes to the underlying storage, thus achieving true online migration.

[0115] S204. When the migration progress is less than the first threshold, send the data write request to the source volume and return a success response to the host after the source volume is successfully written.

[0116] To balance performance and data consistency during migration, this embodiment introduces a dynamic write strategy based on migration progress thresholds.

[0117] Specifically, when the migration progress is less than a first threshold (e.g., 70% of the total data volume), the storage system is in the single-write phase. At this time, for each data write request from the host, the storage system's I / O processing engine can only send it to the source volume. Once the source volume confirms a successful write, it can return a success response to the host. Simultaneously, the background migration thread can asynchronously copy the data blocks involved in the write (or subsequently scanned unsynchronized data) to the target volume. This single-write strategy in the early stages of migration can significantly reduce write operation latency, bringing write performance in the early stages of migration close to the level of the native volume.

[0118] S205. When the migration progress is greater than or equal to the first threshold, the data write request is sent to both the source volume and the target volume simultaneously, and a success response is returned to the host after both the source volume and the target volume are successfully written.

[0119] Specifically, when the migration progress reaches or exceeds a first threshold, the storage system can automatically switch to the dual-write phase. During the dual-write phase, all new data write requests are simultaneously sent to both the source and target volumes. The storage system only returns a success response to the host after both the source and target volumes have successfully completed the write operation. This mechanism ensures that the data content of the source and target volumes is completely consistent at the end of the migration, laying the foundation for a secure switchover.

[0120] S206. After the data migration is completed, switch the mapping relationship of the logical volume from the source volume to the target volume, and keep the globally unique identifier of the logical volume unchanged, so that the target volume can take over the source volume as the back-end storage of the logical volume to provide storage services to the host.

[0121] In this embodiment, after the data migration is complete, a mapping relationship switch can be performed. Specifically, the storage system can update the mapping path of the logical volume from pointing to the source volume to pointing to the target volume.

[0122] It's important to note that during this switchover process, the WWN presented to the outside world by the logical volume remains unchanged. In other words, from the host's perspective, the logical volume it accesses remains unchanged, thus achieving zero service interruption and seamless storage backend replacement for applications. From this point on, the target volume officially replaces the source volume as the backend storage for that logical volume, continuously providing storage services to the host.

[0123] The online data migration method provided in this application divides the migration process into two stages: single-write and dual-write. It dynamically adjusts the write strategy based on a first threshold of the migration progress, maximizing write performance in the early stages and ensuring strong data consistency in the later stages. Simultaneously, by maintaining the logical volume WWN unchanged and seamlessly switching the mapping relationship, it completely eliminates the business interruption or reconnection issues caused by storage switching in traditional migration solutions. This method effectively solves the long-standing performance and consistency contradiction in online data migration, improving the resource utilization of the storage system and the user experience.

[0124] Figure 3 This is a flowchart illustrating a data transmission rate adjustment method provided in an embodiment of this application. Please refer to... Figure 3 The process of migrating data from the source volume to the target volume in the background also includes:

[0125] S301. Monitor the host's read / write load on the logical volume.

[0126] Specifically, the read and write load of the host on the logical volume can be monitored in real time. The read and write load reflects the host's current I / O access activity on the logical volume, and its specific indicators may include: the total number of read and write requests received per unit time, the amount of data transferred per unit time (i.e., data throughput), any one of the average or peak I / O response latency, or a combination of the above indicators.

[0127] S302. When the read / write load exceeds the preset load threshold, the data transfer rate of the background migration is limited to no more than the first rate threshold.

[0128] Specifically, when the read / write load on a logical volume by the host exceeds a preset load threshold, it can be determined that the current host service is operating under high load. In this situation, if the background migration task continues to execute at a high rate, it may compete with the host's I / O operations for shared storage bandwidth, central processing unit (CPU) resources, memory bandwidth, or the processing power of the storage controller, thereby leading to increased host I / O latency and decreased throughput, affecting the response performance and service quality of upper-layer applications.

[0129] To avoid such interference, the background migration strategy can be dynamically adjusted to limit the data transfer rate of the background migration to no more than a first rate threshold. This first rate threshold is a relatively low upper limit on the transfer rate. Its value can be pre-configured by the storage system administrator via CLI, graphical user interface (GUI), or automated policy engine, or it can be adaptively determined based on current hardware performance, network conditions, or Quality of Service (QoS) objectives. The purpose of setting this first rate threshold is to ensure that host I / O operations receive priority resource guarantees while allowing background migration to continue with the minimum necessary bandwidth, thereby ensuring that the performance of host services is not affected and maintaining the continuity of the background data migration process.

[0130] S303. When the read / write load is less than or equal to the preset load threshold, the data transfer rate of the background migration is set to the second rate threshold.

[0131] In this embodiment of the application, the second rate threshold is greater than the first rate threshold.

[0132] Specifically, when the read / write load of the host on the logical volume is less than or equal to a preset load threshold, it can be determined that the current host service is in a light-load state, indicating that storage system resources such as CPU, storage bandwidth, memory resources, and storage controller are relatively idle. In this case, to improve the execution efficiency of background tasks, the data transfer rate of background migration can be set to a second rate threshold to fully utilize the currently available I / O bandwidth and storage system resources, thereby accelerating the data migration process. The second rate threshold is higher than the first rate threshold, and its specific value can be statically configured by the storage system administrator through CLI or GUI based on factors such as the storage system's configuration parameters, network bandwidth capacity, and storage device performance indicators (such as maximum throughput or input / output operations per second (IOPS) capability), or dynamically adjusted by the storage system based on real-time load and resource status.

[0133] Through the aforementioned load-aware dynamic rate adjustment mechanism, the storage system can adaptively optimize the execution efficiency of background data migration while ensuring that the performance of host services is not affected. During the data migration process, it effectively balances low interference to upper-layer applications with the high throughput of the storage system.

[0134] By monitoring the host's read / write load on logical volumes, the storage system limits the background migration rate (first rate threshold) when the load is high and increases the migration rate (second rate threshold) when the load is low. By dynamically adjusting the migration bandwidth allocation, the system avoids migration tasks competing for storage resources with host I / O operations, ensuring that host service performance is not affected, while maximizing migration efficiency when the load is low, thus achieving intelligent resource scheduling and quality of service assurance.

[0135] Figure 4 This is a flowchart illustrating a background data migration method based on snapshot and differential data synchronization, provided as an embodiment of this application. Please refer to... Figure 4 The method may include:

[0136] S401. Before starting the background migration, perform a snapshot operation on the source volume to generate a migration start point snapshot.

[0137] Specifically, before initiating the background data migration task, the storage system can first perform a snapshot operation on the source volume (i.e., the original logical volume currently carrying host I / O services), generating a snapshot of the migration start point that is consistent with the time of the snapshot. This migration start point snapshot is used to record the data state of the source volume at the start of the migration, serving as a consistency benchmark for subsequent full data replication. Through this mechanism, even if the host continuously initiates new data write requests to the source volume during the migration process, the migrated data can still maintain consistency with the content at the snapshot time, thereby avoiding logical inconsistencies or data corruption between the target volume and the source volume due to dynamic updates of the source volume data.

[0138] S402. During the migration process, record all newly generated data write requests that modify the data blocks after the migration start point snapshot as difference data blocks.

[0139] In one possible implementation, the storage system can synchronously enable a differential data tracking mechanism during the background migration process. Specifically, the storage system can monitor and record in real time the logical block addresses (LBAs) modified by all new data write requests generated by host services after the snapshot is created at the migration start point.

[0140] In this embodiment, a difference data block refers to a data unit in the source volume that has been modified by host write operations since the snapshot at the migration start point was created, and its location is uniquely identified by the corresponding logical block address. Therefore, by tracing the modified logical block addresses, the set of difference data blocks that need to be synchronized in the later stages of the migration can be determined.

[0141] Tracking differential data blocks can be achieved through various techniques. For example, a storage system can maintain a bitmap, where each bit corresponds to a logical block address. When a logical block address is written to by the host, the corresponding bit in the bitmap is marked as changed (e.g., set to 1), thus identifying the data block corresponding to that logical block address as a differential data block. Alternatively, a redirect-on-write mechanism can be used to write new data to a new physical location and update the metadata mapping when a write operation occurs on the source volume. Simultaneously, the modified logical block address is recorded in the difference log, thereby indirectly achieving difference tracking. Another approach is to use the storage controller's built-in I / O interception module to capture the target logical block address of a write request before it is sent to the storage medium and add it to the difference record set.

[0142] S403. After the background migration is completed and before switching the mapping relationship, copy the difference data blocks to the target volume to ensure that the target volume and the source volume are completely consistent.

[0143] Specifically, after the background migration completes the copying of all data corresponding to the migration start point snapshot and writes this data completely to the target volume, before officially switching the host I / O mapping relationship (that is, redirecting subsequent read and write requests from the source volume to the target volume), the storage system can perform a final incremental synchronization operation: copying all the difference data blocks recorded during the migration process from the source volume or its associated cache / log area to the corresponding logical block address location in the target volume.

[0144] Because the aforementioned tracking mechanism accurately records the addresses of all logical blocks modified by host write operations, the storage system can read the latest data content at the corresponding address from the source volume and write it to the same logical block address of the target volume. This synchronization operation ensures that at the time of mapping switchover, the data content of the target volume at every logical block address is consistent with that of the source volume, fully covering all new data written by the host service from the creation of the snapshot at the start of the migration until the switchover.

[0145] Through the above mechanism, even in scenarios where host services are continuously running and source volume data is concurrently modified, the storage system can complete the entire migration process without interrupting host I / O services. Specifically, initial data replication, real-time tracking of differential data blocks, and final incremental synchronization are all performed transparently in the background, ensuring that host read and write requests are always processed normally without interrupting services, thus achieving a zero-downtime migration. All new data written by the host after the snapshot is created at the migration start point is accurately recorded and completely written to the target volume through final synchronization, ensuring no user data loss. Furthermore, at the instant the host I / O mapping relationship is switched, the data content of the target volume at every logical block address is strictly consistent with the source volume, and the data state presented externally is completely equivalent to the actual state of the source volume at that moment, without any inconsistency windows of partial updates or intermediate states, thereby ensuring strong consistency. In summary, the embodiments of this application effectively guarantee business continuity and data integrity during online data migration.

[0146] After the migration start point snapshot is created, all data blocks modified by write requests on all hosts are recorded as difference blocks. After the migration is complete, the difference blocks are synchronized to the target volume to ensure complete data consistency between the target and source volumes. This resolves the issue of data inconsistency between the target and source volumes caused by continuous writes during the migration process, achieving strong consistency migration and guaranteeing data integrity and accuracy after the switchover even with a long migration time.

[0147] In one possible implementation, after sending the data write request to both the source volume and the target volume simultaneously, the process further includes: returning a write error and maintaining the original mapping relationship when the source volume write fails; pausing the background migration and generating a fault alarm message when the source volume write succeeds but the target volume write fails; sending data write requests received during the target volume failure period only to the source volume to maintain service availability; and re-copying the differential data blocks to the target volume and continuing the migration process after the target volume failure is resolved.

[0148] In this embodiment, the storage system is further configured with a write fault tolerance and fault recovery mechanism. Specifically, when a data write request initiated by the host is received, the storage controller can concurrently send the data write request to the source volume and the target volume, and monitor the write execution results of both respectively.

[0149] If the source volume write fails (e.g., due to source storage media failure, I / O timeout, or checksum error), the storage system immediately returns a write error response to the host, regardless of whether the target volume write is successful, and maintains the current I / O mapping relationship (i.e., continues to direct subsequent read and write requests to the source volume). This ensures data persistence is based on the source volume, avoiding data inconsistency or silent data corruption caused by source volume write failures.

[0150] If the source volume is successfully written but the target volume fails to be written (e.g., the target storage device is offline, network is interrupted, or there is insufficient space), it can be determined that the target volume has experienced a temporary failure. At this time, the storage system can pause the background migration task (including initial data replication and differential data block synchronization) and generate a fault alarm message containing the fault type, timestamp, and target volume identifier to the management platform or operation and maintenance system for timely intervention and troubleshooting.

[0151] To ensure business continuity during a target volume failure, the storage system can automatically revert to single-write mode: all newly received data write requests are only sent to the source volume for execution, and no further attempts are made to write to the target volume. Simultaneously, the differential data tracking mechanism remains enabled; that is, from the moment the target volume failure begins, all logical block addresses modified by host write operations are still recorded as differential data blocks (e.g., through bitmap marking or appending to the differential log), ensuring accurate identification of the incremental range to be synchronized upon subsequent target volume recovery.

[0152] Once the target volume recovers from the failure (e.g., device re-brought back online, link restored, or resources released), the storage system can first verify the availability and consistency of the target volume. If everything is confirmed to be correct, the background migration process can be reactivated. Based on the set of differential data blocks accumulated during the failure period, the latest data content at the corresponding logical block address in the source volume is copied to the same location in the target volume. After the differential data is fully synchronized and the storage system is stable, the original migration steps (such as eventual consistency verification and mapping switch preparation) are executed.

[0153] In other words, during the dual-write phase, if the source volume write fails, a write error is returned directly while maintaining the original mapping relationship to avoid data loss due to source volume failure. If the target volume write fails, the migration is paused and an alarm is generated, temporarily reverting to single-write mode (writing only to the source volume) to ensure host service availability. After the target volume recovers, the migration task can be resumed automatically and interrupted by resynchronizing the differential data blocks, reducing operational complexity.

[0154] Through the above fault tolerance and recovery mechanisms, even if the target volume experiences a temporary failure, the storage system can still ensure the continuous availability of host services, prevent data loss, and automatically connect the migration process after the fault is recovered, thereby improving the overall reliability and stability of online data migration.

[0155] In one possible implementation, after creating a migration pair consisting of a source volume and a target volume according to the data migration instruction, the method further includes: verifying that the capacity of the target volume is not less than the sum of the used capacity of the source volume and the preset security margin; setting the block size, access permissions, and cache configuration of the target volume to be consistent with the source volume; and reserving metadata storage space in the target volume for recording migration progress and verification information.

[0156] In this embodiment, after receiving a data migration instruction, the storage system first creates a migration pair consisting of a source volume and a target volume to identify the source and target of this migration task. Based on this, to ensure the reliability, compatibility, and recoverability of the migration process, the storage system can further perform initialization configuration and verification operations.

[0157] Specifically, firstly, the storage system can obtain the current used capacity of the source volume (i.e., the space occupied by the actual stored valid user data, rather than the total logical capacity of the source volume), and calculate the sum of the used capacity of the source volume and the preset safety margin in combination with the preset safety margin.

[0158] This safety margin is used to handle temporary data scalability that may occur during the migration process (such as write amplification due to snapshot metadata, log buffers, or concurrent writes). The storage system can further verify to ensure that the available physical capacity of the target volume is not less than the sum of the used capacity of the source volume and the preset safety margin. If the verification fails (i.e., the target volume capacity is insufficient), the migration task can be refused, and an insufficient capacity error can be returned to the management interface, prompting the user to expand the capacity or select another target volume.

[0159] Secondly, to avoid performance anomalies or application compatibility issues caused by configuration differences, the storage system can proactively set the target volume's key runtime attributes to be consistent with the source volume, including but not limited to block size, access permissions, and cache configuration.

[0160] Block size refers to the basic data unit size used by the storage system when performing read and write operations. To ensure consistent I / O behavior between the source and target volumes, the storage system can configure the block size of the target volume to be exactly the same as that of the source volume. This alignment mechanism is crucial. If the block sizes of the target and source volumes are inconsistent, host-initiated I / O requests may cross multiple physical block boundaries on the target volume, triggering the storage controller to perform a Read-Modify-Write (RMW) operation. This involves reading the original block content, merging the new data, and then writing the entire block back. Such operations not only significantly increase I / O latency but also exacerbate write amplification, reduce throughput performance, and may introduce additional data consistency issues. By forcing consistent block sizes, all I / O requests are aligned with the same granularity on both the source and target volumes, avoiding unnecessary RMW overhead and maintaining performance stability and data path efficiency during migration.

[0161] Access permissions refer to a set of security policies used by a storage system to control a host's access capabilities and operational scope to a volume. These policies include the volume's read / write attributes (e.g., setting it to read-only or read-write mode), the authorized host identifiers or initiators in the host's Access Control List (ACL), and Logical Unit Number (LUN) mapping rules. When creating a migration pair, the storage system can configure the target volume's access permissions to be completely consistent with the source volume. This ensures that after the migration is complete and the I / O path is switched, each host's visibility, connection permissions, and data operation permissions to the target volume remain consistent with their behavior towards the source volume before the switch. This avoids problems such as application connection failures, I / O request rejections, or violations of security compliance policies due to permission mismatches.

[0162] Cache policy refers to the caching management strategy adopted by a storage system when handling I / O requests, used to optimize the balance between performance and data persistence. This cache policy includes whether read caching is enabled and the operating mode of write caching, such as write-back mode, where data is written to the cache and an acknowledgment is returned to the host before being asynchronously flushed to persistent storage; or write-through mode, where data must be synchronously written to persistent storage before an acknowledgment is returned. It also includes cache allocation ratios and cache partitioning strategies, such as reserving dedicated cache areas for specific types of I / O operations such as sequential reads, random writes, or metadata I / O.

[0163] During the migration initialization phase, the storage system can configure the cache of the target volume to be completely consistent with that of the source volume to maintain similar I / O response latency, throughput bandwidth, and burst load handling capabilities. This avoids performance fluctuations, timeouts, or service degradation caused by sudden changes in cache behavior, ensuring that the business maintains continuity and a stable user experience during the migration process.

[0164] Finally, metadata space reservation refers to the storage system pre-allocating a fixed-size or dynamically expandable storage space in the starting area or dedicated metadata area of ​​the target volume for persistently recording critical control information during the migration process. This metadata includes at least: migration task identifier (such as a Universally Unique Identifier (UUID)), current migration progress (such as the address range of logical blocks that have been synchronized or a bitmap index)), differential data block tracking status (such as a delta bitmap or logpointer), verification information (such as cyclic redundancy check (CRC) or secure hash algorithm (SHA) digests for each data segment, used for subsequent consistency verification), and fault recovery context (such as the last synchronization timestamp, double-write status flags, etc.).

[0165] This metadata space is initialized before the migration begins and continuously updated throughout the migration lifecycle. Even if the storage system experiences an unexpected power outage or process crash, the migration state can be quickly reconstructed by reading this metadata after restarting, avoiding a full copy from scratch, thus significantly improving migration efficiency and fault tolerance.

[0166] Through the above initialization verification and configuration steps, the storage system can establish a basic environment with sufficient capacity, compatible attributes, and traceable status before the migration officially begins, effectively reducing the possibility of migration failure and providing necessary support for subsequent incremental synchronization, consistency switching, and fault recovery.

[0167] Before migration, the target volume's capacity, attribute alignment (block size, permissions, cache configuration), and reserved metadata space are verified to ensure compatibility and sufficient capacity. This verification step prevents migration failure due to target volume incompatibility or insufficient capacity, and provides reliable metadata support for the migration process, improving overall controllability.

[0168] In one possible implementation, the method further includes: when switching mapping relationships, marking the target volume as a successor volume of the source volume, and persistently storing the association information between the successor volume and the source volume, wherein the association information includes the binding relationship between the globally unique identifier of the source volume and the internal identifier of the target volume, and the successor volume refers to the target volume that logically inherits the identity information of the source volume; updating the path mapping table used to maintain the mapping relationship, and pointing the backend storage path corresponding to the logical volume from the source volume to the target volume; after the switch is completed, removing the association between the source volume and the logical volume, and retaining the data and metadata of the source volume; after the storage controller maintaining the mapping relationship fails and recovers, identifying the target volume as a successor volume of the source volume according to the persistently stored association information, and reconstructing the mapping relationship between the logical volume and the target volume, so as to maintain the globally unique identifier presented to the outside world of the logical volume unchanged.

[0169] Specifically, when switching the mapping relationship between a logical volume and a backend storage entity, the storage system can mark the target volume as a successor volume of the source volume and persistently store the association information between the successor volume and the source volume in a non-volatile storage medium. This association information includes at least the binding relationship between the source volume's WWN and the target volume's internal identifier, thereby ensuring that the target volume can logically inherit the identity characteristics of the source volume.

[0170] Simultaneously, the storage system can update the path mapping table used to maintain the mapping relationship, seamlessly switching the backend storage path corresponding to the logical volume from pointing to the source volume to pointing to the target volume. This switching process is transparent to upper-layer applications, and the device attributes exposed by the logical volume (such as device name, serial number, and globally unique identifier) ​​remain unchanged, thereby avoiding problems such as host-side driver rescanning, multi-path failure, or application connection interruption caused by changes in the underlying storage entity.

[0171] After the mapping switch is successfully completed, the storage system can disconnect the source volume from the logical volume, but retain the user data and its metadata in the source volume to support possible subsequent rollback, auditing or data comparison operations.

[0172] Furthermore, if the storage controller responsible for maintaining the above mapping relationship fails after the switchover is completed and then recovers, the storage system can read the association information of the previously persistent storage during the restart initialization phase, identify the target volume as the inherited volume of the source volume, and automatically rebuild the mapping relationship between the logical volume and the target volume accordingly. Through this mechanism, even in the event of a storage controller failure, the globally unique identifier presented to the outside world by the logical volume remains constant, ensuring the continuity of the host-side I / O path and the stability of business access. Thus, in abnormal scenarios such as storage controller failure, path interruption, or storage node failure, the continuity of logical volume services and the stability of data access can still be guaranteed, improving the self-healing and fault recovery capabilities of the storage system under partial failures, and meeting the stringent requirements of upper-layer applications for business continuity and data persistence.

[0173] By marking the target volume as a descendant of the source volume and persistently storing the association information between the descendant volume and the source volume (such as the WWN binding relationship), the storage system can automatically rebuild the mapping based on this binding relationship after fault recovery, maintaining the logical volume's external identity unchanged. This mechanism ensures that the continuity of logical volume services is not affected in the event of storage controller failure, resolving the service interruption problem caused by metadata loss.

[0174] In some embodiments, to ensure the integrity and consistency of migrated data, the storage system can perform consistency checks on the migrated data blocks in the target volume during the background migration process or before the mapping relationship is switched after migration is complete. Specifically, hash values ​​can be calculated based on the corresponding data blocks of the source and target volumes and compared. If an inconsistency is found, the data block is marked as abnormal, and a retransmission operation is automatically triggered. Only after all migrated data has passed the check is the mapping switch phase allowed. This consistency check mechanism can effectively prevent silent data corruption caused by network transmission errors, storage media failures, or firmware defects, thereby significantly improving the reliability and durability of data migration.

[0175] In some embodiments, this method can also support migration scenarios across storage pools, storage media types, and even storage systems from different vendors. The source and target volumes can reside in storage pools of different performance tiers, such as a high-performance solid-state drive (SSD) pool and a high-capacity hard disk drive (HDD) pool, or be provided by heterogeneous storage devices. When creating a migration pair, the storage system can automatically identify the I / O characteristics of the target volume (such as block size, alignment requirements, access protocols, etc.) and dynamically insert format conversion or metadata adaptation modules into the migration path to resolve compatibility issues caused by differences in block structure, metadata layout, or protocol semantics. Therefore, this embodiment can be widely applied to complex environments such as hybrid cloud architectures, storage tier optimization, and legacy system replacement, significantly expanding the applicability and deployment flexibility of online data migration methods.

[0176] In some embodiments, the data transfer rate during background migration can be dynamically adjusted based on host read / write load, and can also be finely controlled in conjunction with quality of service (QoS) policies. For example, when a logical volume is associated with high-priority services (such as core databases or real-time transaction systems), the storage system can further limit migration bandwidth to minimize the impact on front-end services; conversely, when the overall storage system resources are idle and there are no critical tasks running, the migration rate can be temporarily increased to accelerate completion. Furthermore, users can set the maximum bandwidth limit for migration tasks through management interfaces or APIs. This achieves an intelligent balance between migration efficiency and service level agreements (SLAs), taking into account both storage system resource utilization and application performance stability.

[0177] In some embodiments, the storage system can support full lifecycle management of migration tasks, including pause, resume, and cancellation operations. During the migration process, if a user instruction is received or a preset system event is detected (such as a planned maintenance window, a critical alarm, etc.), the migration process can be immediately paused, and the current migration progress and difference data status can be persistently saved; the migration can then be seamlessly resumed from the breakpoint. If it is decided to terminate the migration, the mapping relationship of the logical volume is rolled back to the source volume, and the resources occupied by the target volume are released. This enhances the flexibility and controllability of operations and maintenance, meeting the stringent requirements for operational security and emergency response capabilities in production environments.

[0178] In some embodiments, for enterprise-level multi-path access scenarios, when switching logical volume backend mappings, the storage system can broadcast path state change notifications to all active paths and force a refresh of LUN metadata in the cache of each path, ensuring that the visibility and state of all paths on the host side to the target volume remain consistent. This effectively avoids I / O failures, path oscillations, or path splits caused by asynchronous path states, ensuring service continuity and smooth switching in a multi-path environment.

[0179] In some embodiments, to ensure compatibility with advanced storage features, the storage system can perform special handling of operations such as snapshots or cloning during migration. If a snapshot request for a logical volume is received during migration, the storage system can transparently redirect the operation to the source volume (because the target volume is not yet fully active); or, after differential data synchronization is complete, correctly associate the newly generated snapshot metadata with the target volume, thereby maintaining the logical continuity of the snapshot chain. This ensures that critical data management functions such as backup and Continuous Data Protection (CDP) remain operational during migration and do not affect existing data protection strategies.

[0180] In some embodiments, after migration is complete, the storage system can perform automated lifecycle management of the source volume based on preset policies. For example, after a successful mapping switch and a period of observation (e.g., 24 hours) confirming no anomalies, the source volume data can be automatically deleted to free up storage space; or it can be converted into a read-only archive copy for auditing or compliance retention; or a space reclamation process can be triggered according to a resource reclamation policy. This effectively avoids redundant data occupying valuable storage resources for extended periods, improving the resource utilization efficiency and automated operation and maintenance level of the storage system.

[0181] In one possible implementation, the single-write / dual-write switching threshold for the migration progress is not a fixed value, but is dynamically adjusted based on the real-time load of the storage system. Specifically, the storage system dynamically calculates the current load status by monitoring the host's read / write load on the logical volume (e.g., the number of I / O requests per unit time, data throughput, or latency metrics). Under high load scenarios (such as CPU utilization exceeding a preset threshold or I / O throughput approaching the storage backend bandwidth limit), the system automatically lowers the switching threshold for the single-write phase (e.g., from 70% to 50%) to enter the dual-write phase earlier, avoiding the risk of data inconsistency due to background synchronization competition. Under low load scenarios, the system raises the switching threshold (e.g., to 85%) to extend the single-write phase and maximize performance gains. Through this dynamic adjustment mechanism, the migration strategy can adapt to different load scenarios, achieving a dynamic balance between performance and consistency.

[0182] In one possible implementation, the background data replication task during the incremental synchronization phase is broken down into multiple subtasks executed in parallel to improve synchronization efficiency. Specifically, the storage system divides the data block range of the target volume into several sub-ranges (e.g., divided contiguously by data block address), with each sub-range corresponding to an independent subtask. Each subtask is handled by an independent thread or process, responsible for reading the corresponding data blocks from the source volume and writing them to the target volume. Simultaneously, the system maintains an independent bitmap and synchronization progress record for each subtask to avoid resource contention between subtasks. Through parallel processing, the background synchronization efficiency of the dual-write phase is significantly improved, the duration of the dual-write phase is shortened to 20%-30% of the original duration, while reducing write latency fluctuations and improving host write performance.

[0183] In one possible implementation, a dedicated hardware module, such as a Field-Programmable Gate Array (FPGA) or Application-Specific Integrated Circuit (ASIC), is embedded in the storage controller to accelerate inherited volume switching. Specifically, this hardware module directly handles host I / O requests without relying on mapping updates from software layer protocol modules. When a migration pair splits, the hardware module directly redirects host I / O requests to the physical storage path of the target volume based on pre-stored inheritance identifiers (the binding relationship between the source volume WWN and the target volume ID). In hardware-accelerated mode, inherited volume switching latency is reduced from milliseconds to microseconds, achieving sub-second seamless service switching. Furthermore, the hardware module supports dynamically loading inheritance identifier metadata, ensuring that the mapping relationship can be quickly rebuilt without software layer intervention after system failure recovery.

[0184] In one possible implementation, the storage system supports multi-level inheritance identifier management to adapt to multi-stage migration scenarios (such as from a source volume to an intermediate inherited volume, and then to a target volume). Specifically, the storage system records the mapping relationships between the source volume and intermediate inherited volumes, and between intermediate inherited volumes and the target volume, by storing inheritance identifier metadata in a hierarchical manner. During the migration process, the storage system prioritizes reading the highest-level inheritance identifier (such as the target volume). If the target volume is unavailable, it falls back to the mapping relationship of the intermediate inherited volumes. Through multi-level inheritance identifier management, the storage system can flexibly support complex migration topologies and improve the migration flexibility of large-scale distributed storage systems.

[0185] Figure 5 This is a flowchart illustrating another online data migration method provided in an embodiment of this application. Figure 5As shown, client 101 can send a data migration instruction to storage system 102 to instruct the backend storage of a logical volume to be migrated from the source volume to the target volume. Storage system 102 can receive the data migration instruction sent by client 101 and create a migration pair consisting of the source volume and the target volume according to the data migration instruction. Then, storage system 102 can perform background migration of data from the source volume to the target volume and monitor the migration progress. During the background migration process, host 103's access to the logical volume remains available, and host 103 can still send data write requests to storage system 102. Storage system 102 can continuously receive data write requests sent by host 103 through the logical volume. When the migration progress is less than a first threshold, storage system 102 can send the data write request to the source volume and return a success response to host 103 after the source volume is successfully written; when the migration progress is greater than or equal to the first threshold, storage system 102 can send the data write request to both the source volume and the target volume simultaneously, and return a success response to host 103 after both the source volume and the target volume are successfully written. After the data migration is completed, the storage system 102 can switch the mapping relationship of the logical volume from the source volume to the target volume, while keeping the globally unique identifier of the logical volume unchanged, so that the target volume can take over the source volume as the back-end storage of the logical volume to provide storage services to the host 103.

[0186] Figure 6 This is a schematic diagram of an online data migration device provided in an embodiment of this application. Please refer to... Figure 6 The device includes:

[0187] The receiving module 61 is used to receive a data migration instruction sent by the user terminal, wherein the data migration instruction indicates that the back-end storage of a logical volume is migrated from the source volume to the target volume, and the logical volume is currently mapped to the source volume.

[0188] Create module 62 to create migration pairs consisting of a source volume and a target volume according to data migration instructions.

[0189] The migration module 63 is used to migrate data from the source volume to the target volume in the background and monitor the migration progress. During the background migration process, it continuously receives data write requests sent by the host through the logical volume, and the host's access to the logical volume remains available.

[0190] The single-write module 64 is used to send a data write request to the source volume when the migration progress is less than the first threshold, and return a success response to the host after the source volume is successfully written.

[0191] The dual-write module 65 is used to simultaneously send data write requests to the source volume and the target volume when the migration progress is greater than or equal to the first threshold, and return a success response to the host after both the source volume and the target volume have been successfully written.

[0192] Switching module 66 is used to switch the mapping relationship of the logical volume from the source volume to the target volume after the data migration is completed, while keeping the globally unique identifier of the logical volume unchanged, so that the target volume can take over the source volume as the back-end storage of the logical volume to provide storage services to the host.

[0193] In one possible implementation, migration module 63 is specifically used for:

[0194] Monitor the host's read / write load on the logical volume.

[0195] When the read / write load exceeds the preset load threshold, the data transfer rate of the background migration will be limited to no more than the first rate threshold.

[0196] When the read / write load is less than or equal to a preset load threshold, the data transfer rate of the background migration is set to a second rate threshold, where the second rate threshold is greater than the first rate threshold.

[0197] In one possible implementation, the device further includes a consistency module, which is specifically used for:

[0198] Before starting the background migration, perform a snapshot operation on the source volume to generate a migration start point snapshot.

[0199] During the migration process, all data blocks modified by newly generated data write requests after the migration start point snapshot are recorded as difference data blocks.

[0200] After the background migration is completed and before the mapping relationship is switched, copy the difference data blocks to the target volume to ensure that the target volume and the source volume are completely consistent.

[0201] In one possible implementation, the dual-write module 65 is also used for:

[0202] If the source volume write fails, a write error is returned and the original mapping is maintained.

[0203] If the source volume is successfully written but the target volume fails to be written, pause the background migration and generate a fault alarm message.

[0204] During a target volume failure, data write requests received during the target volume failure will only be forwarded to the source volume to maintain service availability.

[0205] After the target volume is recovered from the failure, the differential data blocks are copied back to the target volume and the migration process continues.

[0206] In one possible implementation, the apparatus further includes a migration preparation module, which is specifically used for:

[0207] Verify that the capacity of the target volume is not less than the sum of the used capacity of the source volume and the preset safety margin.

[0208] Set the block size, access permissions, and cache configuration of the target volume to be consistent with those of the source volume.

[0209] Reserve metadata storage space in the target volume for recording migration progress and verification information.

[0210] In one possible implementation, the switching module 66 is further configured to:

[0211] When switching mapping relationships, the target volume is marked as the inherited volume of the source volume, and the association information between the inherited volume and the source volume is persistently stored. The association information includes the binding relationship between the globally unique identifier of the source volume and the internal identifier of the target volume. The inherited volume refers to the target volume that logically inherits the identity information of the source volume.

[0212] Update the path mapping table used to maintain the mapping relationship, and change the backend storage path corresponding to the logical volume from the source volume to the target volume.

[0213] After the switch is complete, the association between the source volume and the logical volume is removed, and the data and metadata of the source volume are retained.

[0214] After the storage controller that maintains the mapping relationship fails and recovers, the target volume is identified as the inherited volume of the source volume based on the association information of the persistent storage, and the mapping relationship between the logical volume and the target volume is rebuilt to maintain the globally unique identifier presented to the outside world of the logical volume unchanged.

[0215] The online data migration device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0216] Figure 7 This is a schematic diagram of an online data migration device provided in an embodiment of this application. Please refer to... Figure 7 The online data migration device may include a memory 71 and a processor 72. Exemplarily, the memory 71 and the processor 72 are interconnected via a bus 73.

[0217] Memory 71 is used to store program instructions.

[0218] The processor 72 is used to execute the program instructions stored in the memory, so that the data processing device performs the method shown in the above method embodiment.

[0219] The online data migration device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0220] This application provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above-described method when executed by a processor.

[0221] This application embodiment may also provide a computer program product, including a computer program that, when executed by a processor, can implement the above-described method.

[0222] All or part of the steps in the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above-described method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), random access memory (RAM), flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disc, and any combination thereof.

[0223] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0224] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0225] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0226] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

[0227] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

Claims

1. An online data migration method, characterized in that, The method includes: Receive a data migration instruction sent by the user terminal, wherein the data migration instruction indicates that the back-end storage of a logical volume be migrated from the source volume to the target volume, and the logical volume is currently mapped to the source volume; Create a migration pair consisting of the source volume and the target volume according to the data migration instructions; The data in the source volume is migrated to the target volume in the background, and the migration progress is monitored. During the background migration, data write requests sent by the host through the logical volume are continuously received, and the host's access to the logical volume remains available. When the migration progress is less than the first threshold, the data write request is sent to the source volume, and a success response is returned to the host after the source volume is successfully written. When the migration progress is greater than or equal to the first threshold, the data write request is simultaneously sent to the source volume and the target volume, and a success response is returned to the host after both the source volume and the target volume are successfully written. After the data migration is completed, the mapping relationship of the logical volume is switched from the source volume to the target volume, while keeping the globally unique identifier of the logical volume unchanged, so that the target volume can take over the source volume as the backend storage of the logical volume to provide storage services to the host.

2. The method according to claim 1, characterized in that, The process of migrating data from the source volume to the target volume in the background also includes: Monitor the read / write load of the host on the logical volume; When the read / write load exceeds a preset load threshold, the data transmission rate of the background migration is limited to no more than a first rate threshold. When the read / write load is less than or equal to the preset load threshold, the data transfer rate of the background migration is set to a second rate threshold, wherein the second rate threshold is greater than the first rate threshold.

3. The method according to claim 1, characterized in that, Also includes: Before starting the background migration, a snapshot operation is performed on the source volume to generate a migration start point snapshot; During the migration process, all data blocks modified by newly generated data write requests after the migration start point snapshot are recorded as difference data blocks; After the background migration is completed and before the mapping relationship is switched, the difference data block is copied to the target volume to ensure that the target volume and the source volume are completely consistent.

4. The method according to claim 3, characterized in that, After sending the data write request to both the source volume and the target volume simultaneously, the method further includes: When the source volume write fails, a write error is returned and the original mapping relationship is maintained; When the source volume is successfully written but the target volume fails to be written, the background migration is paused and a fault alarm message is generated. During the target volume failure, data write requests received during the target volume failure will be sent only to the source volume to maintain service availability; After the target volume recovers from the failure, the differential data blocks are copied back to the target volume and the migration process continues.

5. The method according to claim 1, characterized in that, After creating a migration pair consisting of the source volume and a target volume according to the data migration instruction, the process further includes: Verify that the capacity of the target volume is not less than the sum of the used capacity of the source volume and the preset safety margin; Set the block size, access permissions, and cache configuration of the target volume to be consistent with those of the source volume; Metadata storage space is reserved in the target volume for recording migration progress and verification information.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: When switching the mapping relationship, the target volume is marked as the inherited volume of the source volume, and the association information between the inherited volume and the source volume is persistently stored. The association information includes the binding relationship between the globally unique identifier of the source volume and the internal identifier of the target volume. The inherited volume refers to the target volume that logically inherits the identity information of the source volume. Update the path mapping table used to maintain the mapping relationship, and change the backend storage path corresponding to the logical volume from the source volume to the target volume; After the switch is completed, the association between the source volume and the logical volume is removed, and the data and metadata of the source volume are retained; After the storage controller that maintains the mapping relationship fails and recovers, the target volume is identified as the inherited volume of the source volume based on the association information of the persistent storage, and the mapping relationship between the logical volume and the target volume is reconstructed to maintain the globally unique identifier presented to the outside world of the logical volume unchanged.

7. An online data migration device, characterized in that, The device includes: A receiving module is used to receive a data migration instruction sent by a user terminal, wherein the data migration instruction indicates that the back-end storage of a logical volume is migrated from a source volume to a target volume, and the logical volume is currently mapped to the source volume; A creation module is used to create a migration pair consisting of the source volume and the target volume according to the data migration instructions; The migration module is used to migrate data from the source volume to the target volume in the background and monitor the migration progress. During the background migration process, it continuously receives data write requests sent by the host through the logical volume, and the host's access to the logical volume remains available. The single-write module is used to send the data write request to the source volume when the migration progress is less than the first threshold, and return a success response to the host after the source volume is successfully written. The dual-write module is used to simultaneously send the data write request to the source volume and the target volume when the migration progress is greater than or equal to the first threshold, and to return a success response to the host after both the source volume and the target volume are successfully written. The switching module is used to switch the mapping relationship of the logical volume from the source volume to the target volume after the data migration is completed, while keeping the globally unique identifier of the logical volume unchanged, so that the target volume can take over the source volume as the backend storage of the logical volume to provide storage services to the host.

8. An online data migration device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, in, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images