Data migration method, device, electronic device and medium

Abstract

The present disclosure provides a data migration method, device, electronic device, and medium, which belong to the field of computer technology and are intended to solve the problem that traditional data migration methods cannot achieve data migration without interrupting business. The method includes: dividing a data segment in a volume into multiple data blocks, wherein the data block represents a unit of data migration; determining the migration status of a target data block to be migrated among the multiple data blocks; if the migration status of the target data block is occupied by a write operation, storing the write operation data of the target data block in a target queue and migrating the target data block from a source address to a target address; if the migration status of the target data block is occupied by a read operation, reading the target data block from the source address and migrating the target data block from the source address to the target address.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data migration method, device, electronic device, and medium. Background Art

[0002] Data migration is the copying of data across hard disks and hard disk cabinets. During the data migration process, data consistency needs to be guaranteed. During this period, the upper-level system or host still has IO (Input / Output) read and write requests. If the online IO read and write cannot be handled properly, a fatal scenario of inconsistent data before and after migration may easily occur. In response to the above problems, two processing methods have been proposed in the relevant technology. The first method is to migrate data offline, and the second method is to only suspend the IO read and write requests of the data blocks to be migrated. However, the processing methods proposed in the relevant technology still cannot effectively handle the IO read and write requests during the data migration process, and there is a risk of business interruption and data loss due to downtime for maintenance. Summary of the Invention

[0003] To overcome the problems existing in the related art, the present disclosure provides a data migration method, device, electronic device and medium. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a data migration method is provided, including:

[0005] Dividing a data segment in a volume into a plurality of data blocks, wherein the data blocks represent units of data migration;

[0006] determining a migration status of a target data block to be migrated among the plurality of data blocks;

[0007] When the migration state of the target data block is occupied by a write operation, storing the write operation data of the target data block in a target queue, and migrating the target data block from a source address to a target address;

[0008] When the migration state of the target data block is occupied by a read operation, the target data block is read from the source address, and the target data block is migrated from the source address to the target address.

[0009] Optionally, the length of the target queue is determined in the following manner:

[0010] Determining theoretical data based on the business load data model, wherein the theoretical data represents theoretical usage information of resources; determining the theoretical optimal length of the target queue and an adjustment step size based on the theoretical data;

[0011] Determining real-time data based on a business load data model and a waiting time for real-time read and write operations, wherein the real-time data represents real-time usage information of resources;

[0012] According to the real-time data, the length of the target queue determined according to the theoretical data is adjusted according to an adjustment step, and / or the number of the target queues is adjusted.

[0013] Optionally, it also includes:

[0014] If the migration status of the target data block is occupied by a write operation, after migrating the target data block from the source address to the target address, writing the write operation data stored in the target queue to the source address and the target address, and updating the migration status of the target data block from occupied by a write operation to migration completed;

[0015] In a case where the migration status of the target data block is occupied by a read operation, after the target data block is migrated from the source address to the target address, the migration status of the target data block is updated from occupied by a read operation to migration completed.

[0016] Optionally, writing the write operation data stored in the target queue to the source address and the target address includes:

[0017] When the available space in the target queue is exhausted, or when the first write operation data stored in the target queue exceeds the preset processing cycle, the various write operation data stored in the target queue are written in batches to the source address and the corresponding target address corresponding to the write operation data, and the target queue includes the write operation data of the target data block once or multiple times.

[0018] Optionally, the migration status includes write operation occupied, read operation occupied, no read or write operation, and migration completed. The determining the migration status of the target data block to be migrated among the multiple data blocks includes:

[0019] Scan the target data block to obtain the read and write status of the target data block;

[0020] Determining the latest migration status of the target data block according to the read and write status;

[0021] The migration status of the target data block is determined according to the latest migration status.

[0022] Optionally, it also includes:

[0023] When the migration status of the target data block is no read or write operation, the target data block is migrated from the source address to the target address, and the migration status of the target data block is updated from write operation occupied to migration completed.

[0024] Optionally, it also includes:

[0025] Performing a read operation on a data block to determine a migration status of the data block;

[0026] When the migration status of the data block is migration completed, reading from the target address corresponding to the data block; the migration status being migration completed indicates that the data block has completed migration from the source address to the target address;

[0027] If the migration status of the data block is occupied by a write operation, a one-to-one correspondence is performed in the target queue according to the physical address of the data block to determine the position of the data block in the target queue; and data corresponding to the data block is read from the target queue. The migration status being occupied by a write operation indicates that the write operation data corresponding to the data block is stored in the target queue and the write operation data is not written back to the source address and target address corresponding to the data block.

[0028] Performing a write operation on the data block to determine a migration status of the data block;

[0029] When the migration status of the data block is migration completed, the write operation data is simultaneously written into the source address and the target address corresponding to the data block.

[0030] According to a second aspect of an embodiment of the present disclosure, there is provided a data migration apparatus, including:

[0031] a partitioning module, configured to partition a data segment in a volume into a plurality of data blocks, wherein the data blocks represent units of data migration;

[0032] a determination module, configured to determine a migration status of a target data block to be migrated among the plurality of data blocks;

[0033] a first migration module, configured to, when the migration state of the target data block is occupied by a write operation, store the write operation data of the target data block in a target queue and migrate the target data block from a source address to a target address;

[0034] The second migration module is configured to read the target data block from the source address and migrate the target data block from the source address to the target address when the migration state of the target data block is occupied by a read operation.

[0035] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the data migration method described in the first aspect are implemented.

[0036] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of data migration as described in the first aspect are implemented.

[0037] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the data migration method described in the first aspect are implemented.

[0038] The present disclosure divides the data volume into multiple data blocks and independently processes data migration for each data block according to the migration status, thereby achieving fine-grained control in the data migration process and realizing data migration without interrupting business. When the migration status is occupied by write operations, the write operation of the target data block is temporarily stored in a target queue to ensure that the data migration and the write operation are carried out synchronously. When the migration status is occupied by read operations, the read operation of the source address is continued to be allowed, and the migration process will not block normal read operations, thereby realizing the parallel development of read and write IO and data migration, and having good upper-layer application reliability and disaster tolerance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0040] Figure 1 This is a data migration diagram shown in an embodiment of the present disclosure;

[0041] Figure 2 This is a data segment division method shown in an embodiment of the present disclosure;

[0042] Figure 3 is a schematic diagram illustrating a method of performing data migration using relevant technologies according to an embodiment of the present disclosure;

[0043] Figure 4 This is a schematic diagram of the steps of a data migration method shown in an embodiment of the present disclosure;

[0044] Figure 5This is a data migration diagram shown in an embodiment of the present disclosure;

[0045] Figure 6 This is a schematic diagram of data block data migration shown in an embodiment of the present disclosure;

[0046] Figure 7 is a block diagram of a data migration device shown in an embodiment of the present disclosure;

[0047] Figure 8 is a schematic diagram of an electronic device shown in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0049] The terms "first", "second", etc. in the specification and claims of the present disclosure are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present disclosure can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects related to each other are in an "or" relationship.

[0050] Data migration technology is based on the principle of tiered storage management, treating high-speed, high-capacity, offline storage devices as the next level of disk storage. Using specified policies, frequently used data on disks is automatically migrated to secondary, high-capacity storage devices such as tape libraries. When this data is needed, the tiered storage system automatically transfers it from the next-level storage device back to the previous-level disk. This technology allows enterprises to store infrequently accessed data on less expensive storage devices without sacrificing access speed, thereby reducing investment and management costs.

[0051] Data migration is the copying of data across hard drives and hard drive cabinets. Figure 1 This is a data migration diagram shown in an embodiment of the present disclosure. Figure 1As shown, there are two storage pools, storage pool A and storage pool B. A storage pool is a logical unit composed of multiple physical storage devices and is used to uniformly manage and allocate storage resources. Storage pool A includes physical disk 0, physical disk 1, and physical disk 2, and storage pool B includes physical disk 3 and physical disk 4. Physical disks are components of storage pools. When migrating data, the X of physical disk 0 in storage pool A is migrated to the corresponding position of physical disk 3 in storage pool B. The location of the data in the source storage pool corresponds to the location in the target storage pool.

[0052] Figure 2 This is a data segment division method shown in the embodiment of the present disclosure. Figure 2 As shown, a virtual disk can be mapped to a volume. When migrating data in the storage system, the Lun (volume) in the storage system is migrated in units of Extent (data segment). The migration of a single Extent also needs to be divided into chunks of 16MB. Chunks are the units of data transmission in the storage system. Among them, when the storage system performs IO reading and writing, the 16MB chunks (data blocks) are divided into 256K chunks for IO reading and writing.

[0053] During the data migration process, data consistency needs to be guaranteed. During this period, the upper-layer system or host still has IO read and write requests. If the online IO read and write cannot be handled properly, a fatal scenario of inconsistent data before and after migration may easily occur.

[0054] In the related art, two processing solutions are proposed.

[0055] Solution 1: When migrating data on a storage volume, customer I / O must be suspended, and data migration must be performed offline. However, Solution 1 completely suspends online I / O, which is very unfriendly to online systems and can disrupt upper-layer applications. It also lacks high disaster recovery and reliability.

[0056] Solution 2: The storage volume migrates data in units of extents. After the entire extent is completed, the next extent is migrated.

[0057] The specific process for Solution 2 is as follows: ① Stop all I / O on the source extent; ② Remove the I / O pause restriction except for the chunk to be migrated; Perform the following operations on the selected node to implement the migration: ③ Synchronously read a chunk (256KB) from the source extent; ④ Synchronously write the read data to the destination extent; ⑤ After the chunk migration is complete, host write I / O needs to mirror the writes to the source and destination extents. After the chunk migration is complete, repeat ②, ③, ④, and ⑤ for the next chunk. Figure 3FIG. 1 is a schematic diagram of an embodiment of the present disclosure showing a method of using relevant technologies to perform data migration. Figure 3 As shown, if the data block has I / O, data migration cannot be performed; if the data block does not have I / O, data migration can be performed. However, Solution 2 uses the chunk as the smallest unit and suspends I / O. Although this ensures some I / O reading and writing, there are 4096 chunks in the Extent, and there are many and scattered chunks with I / O operations. This results in many chunks in the Extent being unable to perform data migration, resulting in a slow data migration process and frequent waiting. Migration cannot be performed in physical address order, making the process complex and prone to other unexpected problems in the event of multiple failures.

[0058] In order to solve the above technical problems, the present disclosure proposes a data migration method, which can migrate data without interrupting read and write IO, and can avoid the risks of business interruption and data loss caused by downtime maintenance in traditional migration methods.

[0059] Figure 4 This is a schematic diagram of the steps of a data migration method shown in an embodiment of the present disclosure. Figure 4 As shown, the data migration method may specifically include the following steps:

[0060] Step S11: Divide the data segments in the volume into multiple data blocks, and the data blocks represent the units of data migration; when migrating data, multiple data segments can be migrated in parallel. For any data segment, when performing IO reading and writing on the data in the storage system, the Chunks are divided into 256K Chunks for IO reading and writing. Based on this, the migration unit when migrating data can be determined. The volume can be divided into data segments of equal size, and then the data segments can be divided into data blocks of equal size, and the data blocks can be determined as the smallest migration units. For example, the Extent is divided into Chunks of 16MB in size, and then the Chunks are divided into 256K Chunks, and the Chunks are determined as the migration unit. Determining the Chunk as the migration unit can be compatible with IO reading and writing.

[0061] Step S12: determining the migration status of a target data block to be migrated among the plurality of data blocks;

[0062] A corresponding migration state may be set for each data block, the data block being migrated is determined as the target data block, the migration state of the target data block is determined, and the migration action for the target data block is determined based on the migration state of the target data block.

[0063] Step S13: When the migration state of the target data block is occupied by a write operation, the write operation data of the target data block is stored in a target queue, and the target data block is migrated from the source address to the target address;

[0064] When the migration status of the target data block is occupied by a write operation, it means that when the target data block is undergoing data migration, there is a write IO request, and the write IO request will cause the data to change. The data corresponding to the target database can be regarded as two parts, one part is the original data, that is, the data before the write IO request is initiated in the source address, and the other part is the new data, that is, the data corresponding to the write IO. These two parts of data can be processed separately, and the original data can be migrated from the source address to the target address according to the traditional migration method, and the new data can be written into the target queue. This target queue is a queue independent of data migration, which can temporarily store the write IO data corresponding to the target data segment, and can avoid conflicts between data migration and write operations.

[0065] Only when a data block is being migrated will it be defined as the target data block and the write operation data of the target data block will be stored in the target queue. Therefore, if a data block has a corresponding write operation before data migration is performed, the write operation at this time will be executed as a normal operation and the write operation data of the data block will not be stored in the target queue.

[0066] Step S14: When the migration status of the target data block is "read occupied," the target data block is read from the source address and migrated from the source address to the target address. When the migration status of the target data block is "read occupied," it means that a read IO request is still in progress while the target data block is being migrated. The read IO request will not cause any changes, so the data block is migrated from the source address to the target address using a traditional migration method. While the data at the source address is complete while the target data block is being migrated, the data at the target address may be incomplete. Therefore, in order for the read IO request to obtain complete data, the corresponding data may be read from the source address of the target data block.

[0067] By adopting the embodiments of the present disclosure, data blocks are used as the smallest migration units, which can match the access units of read and write IO requests and provide higher flexibility. When migrating data blocks, by determining the migration status of the data blocks and performing corresponding migration operations based on the migration status, the conflict between read and write operations and data migration can be reduced, and the integrity and consistency of data before and after migration can be ensured. The migration strategy can be dynamically adjusted according to the migration status of the data blocks, which can more effectively improve the migration efficiency and ensure the disaster recovery of the storage system. Data migration can be performed without affecting the normal operation of the application, and the system can complete the data migration task without interrupting the service.

[0068] Figure 5 This is a data migration diagram shown in an embodiment of the present disclosure. Figure 5 As shown, a data block can be determined as a unit of data migration, and a migration state can be set for the data block according to the read and write status of the data block. Each migration state uses a different identifier to migrate each data block from the source address to the corresponding target address. For example, when the source address of the data block is a, the target address corresponding to this data block is aa. In order to avoid conflicts between data migration and read and write operations, a target queue is constructed, which is used to store the write operation data of the data block being migrated.

[0069] In one optional embodiment, the length of the target queue is determined in the following manner: according to the business load data model, theoretical data is determined, and the theoretical data represents theoretical usage information of resources; according to the theoretical data, the theoretical optimal length and adjustment step of the target queue are determined; according to the business load data model and the waiting time of real-time read and write operations, real-time data is determined, and the real-time data represents real-time usage information of resources; according to the real-time data, the length of the target queue determined according to the theoretical data is adjusted according to the adjustment step, and / or the number of the target queues is adjusted.

[0070] The Smart Online Data Migration (SODM) management module is configured on the corresponding storage system board. This module dynamically manages the detection module, front-end module, main control module, back-end module, and disk array module. The detection module monitors CPU utilization and memory usage; the front-end module manages the storage system's I / O queue; the main control module manages the target queue; the back-end module manages read and write operations on data blocks during data migration; and the disk array module manages the hard drives.

[0071] The business data model can include the IO read-write ratio, as well as the write ratio of new and old addresses.

[0072] The SODM management module embeds various theoretical data for various business data models. These include theoretical CPU resource utilization (C), theoretical memory resource utilization (D), and the theoretical ratio (F) of the new to old addresses of the write object target address for the target queue in write I / O requests. These theoretical data represent the theoretical system usage under the current business data model. If the actual system usage exceeds the theoretical usage, it indicates that the system is overloaded and underperforming. If the actual system usage does not exceed the theoretical usage, it indicates that the system has sufficient performance.

[0073] The SODM management module also embeds a relationship table between theoretical data and target queues under various business data models; based on the relationship table, the optimal length of the target queue under the current business data model and the adjustment step can be determined, and the adjustment step is used to adjust the length of the target queue.

[0074] The front-end module can obtain real-time CPU performance and memory utilization of each main control module under different business data models, as well as real-time I / O queue wait time. The real-time I / O queue wait time represents the time difference between an I / O write request entering the target queue and actually writing to the target address. Therefore, the front-end module can obtain the real-time CPU resource utilization rate A, the real-time memory resource utilization rate B, and the real-time ratio E of the new and old target addresses of the write object in the target queue in the write I / O request under the business data model. This real-time data represents the real-time usage information of the system under the current business data model.

[0075] The write IO object address can be used to distinguish whether it is a new write IO or a modification IO of an old object. Specifically, the write IO identifier can be used to distinguish the new and old addresses of the write object target address, thereby obtaining the theoretical ratio of the new and old addresses of the write object target address. For example, when the write IO data corresponding to a data block has been stored in the target queue and the target queue has not written back, the address of this data block is the old address; when the write IO data corresponding to a data block has not been stored in the target queue, the address of this data block is the new address.

[0076] The SODM management module adjusts the length and number of target queues based on real-time data acquired by the front-end module. The number of target queues is equal to the number of extents available for synchronous data migration. The target queue length is adjusted using a preset adjustment step size. The target queue length is represented by L, the target queue adjustment step size is represented by M, and the number of target queues is represented by N. Based on theoretical and real-time data, either a Level 1 data migration warning or a Level 2 data migration alarm can be triggered. Both alarms trigger adjustments to the target queue length and / or number. The Level 1 data migration warning includes two scenarios: If the SODM management module detects that A is less than C*Y1 within time period T1, the length of L is increased by M and the number of N is increased by 1. If the SODM management module detects that B is greater than D*Y2 within time period T2, the length of L is reduced by M and the number of N is reduced by 1. Increases or decreases in the number of target queues are accompanied by increases or decreases in the number of data segments to be migrated synchronously. When there is a discrepancy between theoretical and real-time data, the system adjusts the number of data segments to be migrated synchronously, ensuring that system performance meets the current workload. The secondary data migration warning includes two situations. The first situation is that if the SODM management module continuously monitors that E is greater than F*Y3 within the time period T3, the length of L is +M; if the SODM management module continuously monitors that E is less than F*Y4 within the time period T4, the length of L is -M.

[0077] Among them, B\D\F\Y1\Y2\Y3\Y4\T1\T2\T3\T4\L\M\ are system preset parameters and can be adjusted in the system or through serial port modules.

[0078] By adopting the embodiments of the present disclosure, the corresponding target queue length and number can be adaptively selected according to different business data models; by comparing theoretical data and real-time data in real time, it is ensured that resources are not overused or wasted, thereby optimizing resource utilization; by dynamically adjusting the length and number of IO processing queues, the queue length and number can be increased when resources are sufficient to improve the migration speed, and the queue length and number can be reduced when resources are tight to avoid system overload and maintain stable system operation; based on theoretical data, the theoretical optimal length of the target queue and the adjustment step size can be determined, and flexible configuration and optimization can be performed according to different application scenarios and needs.

[0079] Among them, in an optional embodiment, it also includes: when the migration status of the target data block is occupied by write operation, after the target data block is migrated from the source address to the target address, the write operation data stored in the target queue is written to the source address and the target address, and the migration status of the target data block is updated from occupied by write operation to migration completed; when the migration status of the target data block is occupied by read operation, after the target data block is migrated from the source address to the target address, the migration status of the target data block is updated from occupied by read operation to migration completed.

[0080] For a target data block whose migration status is occupied by a write operation, after the data block is migrated from the source address to the target address, it will not be determined that the migration of the target data block is completed, because the write operation data of the target data block is still stored in the target queue. Only when the write operation data corresponding to the target data block in the target queue is written back to the source address and target address of the target data block, will it be determined that the migration of the target data block is completed, and the migration status of the target data block will be updated from occupied by a write operation to migration completed.

[0081] For a target data block whose migration status is occupied by a read operation, the read operation does not change the original data. Therefore, after the target data block is migrated from the source address to the target address, it is determined that the migration of the target data block is completed, and the migration status of the target data block is updated from occupied by a read operation to completed migration.

[0082] By adopting the embodiments of the present disclosure, the integrity and consistency of the data can be ensured by writing the backlog of write operation data in the target queue to the source address and the target address at the same time after the migration of the data block from the source address to the target address is completed; the migration status of the data block is updated to migration completed after the migration is completed, which helps to quickly release the migrated data block, thereby reducing the system's response time to user operations; by precisely controlling the read and write operations and state transitions during data migration, business continuity and data accuracy can be ensured.

[0083] Among them, in an optional embodiment, the write operation data stored in the target queue is written to the source address and the target address, including: when the available space of the target queue is exhausted, or when the first write operation data stored in the target queue exceeds the preset processing cycle, the various write operation data stored in the target queue are batch-written to the source address and the corresponding target address corresponding to the write operation data, and the target queue includes the write operation data of the target data block once or multiple times.

[0084] The target queue has a certain length and can only hold a limited amount of write operation data. Therefore, space in the target queue needs to be released to meet system needs. There are two conditions for releasing the target queue: one is when the target queue space is exhausted, and the other is when the first write operation stored in the target queue exceeds the preset processing period. If either condition is met, space in the target queue needs to be released. The preset processing period is the time difference between the moment the write operation data corresponding to a data block is stored in the target queue and the moment the write operation data is written back to the target address corresponding to the data block.

[0085] The target queue is monitored in real time. When it is detected that the space of the target queue is exhausted, or when the first write operation data stored in the target queue exceeds the preset processing cycle, the space release of the target queue is triggered. Specifically, the write operation data stored in the target queue is batch-written into the source address and the corresponding target address corresponding to the write operation data.

[0086] When the target queue is used to store write operation data of the target data block, the write operation data is written to the target queue, the write operation is recorded once, and the write operation data is written back from the target queue to the source address and the target address, the write operation is recorded twice, and a total of three write operations.

[0087] The target queue can store one or more write operation data for the same data block. For example, when a data block is performing a migration operation, this data block is determined as the target data block. During the migration of the target data block, there may be a write operation on the target data block. The write operation data corresponding to the write operation data is stored in the target queue. There is a time difference between the time when the target queue receives the write operation data and the time when the write operation data is written back. During this time, multiple write operations may be performed on the target data block. The write operation data corresponding to the multiple write operations are also stored in the target queue and written back to the source address and target address corresponding to the target data block together during the write back. Only one write back is performed. Therefore, no matter how many times the target queue stores the write operation data of the target data block, it will only be written back once, and two write operations will be recorded during the write back.

[0088] By adopting the embodiments of the present disclosure, by triggering batch writing when the available space in the target queue is exhausted or the first data times out, the system can process the write operation data in the queue more promptly, avoiding processing delays caused by queue overflow or long waiting time for data; through asynchronous processing and batch writing, the system can reduce interference with normal business operations, and maintain high business continuity and availability even when performing large-scale data migration or synchronization; because the target queue includes one or more write operation data of the target data block, in extreme scenarios, such as repeated erasing and writing of data in the same area, the additional write cost can be minimized. Figure 6 This is a schematic diagram of data block migration shown in an embodiment of the present disclosure. Figure 6 As shown, in the process of migrating the data block from the source address to the target address, the corresponding migration method is selected according to the migration status of the data block. When the migration status of the data block is occupied by the write operation, the write operation data is written into the target queue according to the corresponding write IO; when the data in the target queue reaches a certain amount (when the write-back condition is met), the write operation data in the target queue is written to the source address and the target address in batches, and before writing back, if the data of the data block needs to be read, it can be read from the target queue. Among them, the source address, target address and address of the data block in the target queue are one-to-one corresponding. For example, the source address of a data block is a, the target address is aa, and the address in the target queue is aaa.

[0089] In one optional embodiment, the migration status includes write operation occupation, read operation occupation, no read or write operation, and migration completion, and determining the migration status of the target data block to be migrated among the multiple data blocks includes: scanning the target data block to obtain the read and write status of the target data block; determining the latest migration status of the target data block based on the read and write status; and determining the migration status of the target data block based on the latest migration status.

[0090] Two bits can be added to the flag bit of the data block to mark the migration status. 00 can indicate no read or write operation, 01 indicates read operation is in use, 10 indicates write operation is in use, and 11 indicates migration completion. Therefore, the migration status includes write operation in use, read operation in use, no read or write operation, and migration completion.

[0091] When performing data migration, the data segment (Extent) can respond to the data migration command, scan the various data blocks (Chunks) contained in the data segment, and add a corresponding migration status for the data block based on the current read and write status of each data block. The migration status added at this time is the most initial migration status. When the data block performs data migration according to the data migration rules (physical address from low to high), the read and write status of the data block is rescanned, and the migration status of the data block is updated based on the read and write status of the data block, and it is determined as the latest migration status. According to the latest migration status, the corresponding migration operation is performed.

[0092] The initial migration state of a data block does not affect the read and write operations of the data block. For example, if the initial migration state of a data block is occupied by write operations, the write operation data of the data block will not be stored in the target queue, but the normal action will be performed. However, when the migration state of a data block is occupied by write operations during data migration, the write operation data of the data block will be stored in the target queue.

[0093] The migration status of the data block will change, and the corresponding data migration method will be determined according to the migration status determined when the data block performs data migration. For example, the initial migration status of a data block is occupied by write operations, but when this data block performs data migration, the data block has no read or write operations. At this time, the migration status of the data block is no read or write operations, so the data block will be migrated using the migration method corresponding to no read or write operations. By adopting the embodiments of the present disclosure, the migration status of the data block can be updated in a timely manner during the migration process. By clarifying the migration status of the data block and determining a more reasonable migration strategy based on the migration status of the data block, conflicts between read and write operations and data migration can be avoided; by finely controlling the migration process, the system can complete data migration without affecting or with minimal impact on business operations; through automated migration status management and classification processing logic, the system can simplify the management of the migration process.

[0094] Among them, in an optional embodiment, it also includes: when the migration status of the target data block is no read or write operation, migrating the target data block from the source address to the target address, and updating the migration status of the target data block from write operation occupied to migration completed.

[0095] The migration status of the target data block is determined. If the migration status of the target data block is "no read / write operation", it indicates that there is no need to consider the parallel operation of data migration and read / write operations. Therefore, for the data block with the migration status of "no read / write operation", it is only necessary to migrate the target data block from the source address to the target address according to the conventional method. After the migration reaches the target address, the migration status of the data block is updated to "migration completed".

[0096] By adopting the embodiments of the present disclosure, data blocks in a migration state of no read or write operations are migrated in a conventional manner, thereby maximizing the utilization of system resources and improving the speed and efficiency of migration.

[0097] Among them, in an optional embodiment, it also includes: performing a read operation on the data block to determine the migration status of the data block; when the migration status of the data block is migration completed, reading from the target address corresponding to the data block; the migration status is migration completed, indicating that the data block has completed the migration from the source address to the target address; when the migration status of the data block is write operation occupied, a one-to-one correspondence is performed in the target queue according to the physical address of the data block to determine the position of the data block in the target queue; reading data corresponding to the data block from the target queue; the migration status is write operation occupied, indicating that the write operation data corresponding to the data block is stored through the target queue, and the write operation data is not written back to the source address and target address corresponding to the data block; performing a write operation on the data block to determine the migration status of the data block; when the migration status of the data block is migration completed, writing the write operation data to the source address and target address corresponding to the data block at the same time. There are three situations for reading a data block based on the migration status of the data block, whether it is undergoing data migration or has completed migration. Among them, for a data block that has not undergone data migration, the migration status does not affect the read and write operations of the data block. When there are read and write operations on the data block before data migration, regular read and write operations are performed on the data block.

[0098] The first case is that when the migration status of the data block is migration completed, the data block is read from the target address. This is because the migration status of migration completed indicates that the data block has completed the data migration from the source address to the target address, so the data can be read from the target address to ensure that the read data is the latest data. The second case is that when the migration status of the data block is write operation occupied, the write operation occupied indicates that the data of the current data block is divided into two parts, one part is the original data (data before the write operation is performed), and the other part is the write operation data (data after the write operation is performed). The write operation data will be stored in the target queue during the data block data migration process, and the write operation data is not written back to the source address and target address corresponding to the data block. Therefore, accessing the write operation data corresponding to the data block from the target queue can read the latest data. Specifically, according to the physical address of the data block, a one-to-one correspondence can be made in the target queue to determine the position of the data block in the target queue and read the data corresponding to the data block from the target queue. The third case is that when the migration status of the data block is occupied by the read operation, the data block is read from the source address. The reason for reading the data block from the source address is that when the data block is being migrated, the data at the source address is complete, and the data at the target address may be incomplete. Therefore, reading the data block from the source address can ensure the integrity of the data.

[0099] Perform a write operation on a data block and determine corresponding steps based on the migration status of the data block. When the migration status of the data block is migration completed, write the write operation data to the source address and target address corresponding to the data block at the same time.

[0100] Therefore, for data blocks that have been migrated, write operation data is written to both the source and target addresses corresponding to the data block, and read operations are performed from the target address of the data block. For data blocks where write operation data is stored in the target queue and unwritten operation data is not written back to the source and target addresses, the data block can be accessed from the target queue.

[0101] According to the embodiments of the present disclosure, when the data block migration is completed, the data is read directly from the target address, avoiding the delay or erroneous access to the source address that may be caused by incomplete migration, thereby improving the immediacy and accuracy of data access. In the write operation scenario, if the data block migration is completed, the source address and the target address are updated at the same time, ensuring data consistency and fast response. If the migration is not completed and the migration state is not occupied by the write operation, the data is read from the target queue to ensure data integrity. Regardless of the migration state of the data block, the system can ensure the correctness of the read and write operations and the consistency of the data. The self-recovery capability and fault tolerance of the system are improved, reducing system failures and downtime caused by improper data migration or read-write conflicts. In an optional embodiment, a fault recovery mechanism can be set up to automatically recover the data or roll back to the state before migration when an error or data corruption occurs during data migration. First, the system monitors and detects various errors in the migration process in real time, including network failures, data corruption, hardware failures, etc. For example, checksums and / or hash comparisons can be performed before and after the data block migration to ensure the integrity of the data during transmission. When an error is detected, recovery measures corresponding to the error can be taken. If a migration fails due to network issues or temporary errors, the system automatically retry the migration operation. If data corruption or inconsistency is detected, the system halts the current migration task and rolls back the migrated data blocks to their original state. During the migration process, detailed status information for each data block is maintained, and restore points are created regularly. In the event of a failure, the system can resume operations from the most recent restore point.

[0102] Figure 7 This is a block diagram of a data migration device shown in an embodiment of the present disclosure. Figure 7 As shown, the data migration device includes:

[0103] A partitioning module 710 is configured to partition a data segment in a volume into a plurality of data blocks, wherein the data blocks represent units of data migration;

[0104] a determination module 720 configured to determine a migration status of a target data block to be migrated among the plurality of data blocks; a first migration module 730 configured to, if the migration status of the target data block is occupied by a write operation, store the write operation data of the target data block in a target queue and migrate the target data block from a source address to a target address;

[0105] The second migration module 740 is configured to read the target data block from the source address and migrate the target data block from the source address to the target address when the migration state of the target data block is occupied by a read operation.

[0106] Optionally, the data migration device further includes a queue length determination module, wherein the queue length determination module is configured to:

[0107] Determining theoretical data based on the business load data model, wherein the theoretical data represents theoretical usage information of resources; determining the theoretical optimal length of the target queue and an adjustment step size based on the theoretical data;

[0108] Determining real-time data based on a business load data model and a waiting time for real-time read and write operations, wherein the real-time data represents real-time usage information of resources;

[0109] According to the real-time data, the length of the target queue determined according to the theoretical data is adjusted according to an adjustment step, and / or the number of the target queues is adjusted.

[0110] Optionally, the data migration device further includes:

[0111] a first updating module, configured to, if the migration status of the target data block is occupied by a write operation, write the write operation data stored in the target queue to the source address and the target address after migrating the target data block from the source address to the target address, and update the migration status of the target data block from occupied by a write operation to migration completed;

[0112] The second update module is configured to update the migration status of the target data block from read operation occupied to migration completed after migrating the target data block from the source address to the target address when the migration status of the target data block is read operation occupied.

[0113] Optionally, the first update module is specifically configured to execute:

[0114] When the available space in the target queue is exhausted, or when the first write operation data stored in the target queue exceeds the preset processing cycle, the various write operation data stored in the target queue are written in batches to the source address and the corresponding target address corresponding to the write operation data, and the target queue includes the write operation data of the target data block once or multiple times.

[0115] Optionally, the migration status includes write operation occupied, read operation occupied, no read or write operation, and migration completed, and the determining module is specifically configured to execute:

[0116] Scan the target data block to obtain the read and write status of the target data block;

[0117] Determining the latest migration status of the target data block according to the read and write status;

[0118] The migration status of the target data block is determined according to the latest migration status.

[0119] Optionally, the data migration device further includes:

[0120] The third migration module is used to migrate the target data block from the source address to the target address when the migration status of the target data block is no read or write operation, and update the migration status of the target data block from write operation occupied to migration completed.

[0121] Optionally, the data migration device further includes:

[0122] A first determining module is configured to perform a read operation on a data block to determine a migration status of the data block;

[0123] A first reading module is configured to read from a target address corresponding to the data block when the migration status of the data block is migration completed; the migration status being migration completed indicates that the data block has completed migration from the source address to the target address;

[0124] A second reading module is configured to, when the migration state of the data block is occupied by a write operation, determine, in the target queue according to the physical address of the data block, a position of the data block in the target queue by performing a one-to-one correspondence; and read data corresponding to the data block from the target queue; the migration state being occupied by a write operation is characterized in that the write operation data corresponding to the data block is stored in the target queue and the write operation data is not written back to the source address and target address corresponding to the data block;

[0125] The second determination module is used to perform a write operation on the data block to determine the migration status of the data block; the first writing module is used to write the write operation data into the source address and target address corresponding to the data block at the same time when the migration status of the data block is migration completed.

[0126] It should be noted that the system embodiment is similar to the method embodiment, so the description is relatively simple. For relevant details, please refer to the method embodiment.

[0127] The present disclosure also provides an electronic device, Figure 8 , Figure 8 FIG is a schematic diagram of an electronic device according to an embodiment of the present disclosure. Figure 8 As shown, the electronic device 800 includes: a memory 810 and a processor 820. The memory 810 and the processor 820 are connected via a bus communication. A computer program is stored in the memory 810. The computer program can be run on the processor 820 to implement the steps in the data migration method disclosed in the embodiment of the present disclosure.

[0128] The embodiments of the present disclosure further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the data migration method disclosed in the embodiments of the present disclosure are implemented.

[0129] An embodiment of the present disclosure further provides a computer program product, including a computer program, which implements the steps of the data migration method described in the first aspect when executed by a processor.

[0130] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0131] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, apparatuses, or computer program products. Therefore, the embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0132] The embodiments of the present disclosure are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0133] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps of the functions specified in a box or multiple boxes. Although some embodiments of the present disclosure have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present disclosure. The above is a detailed introduction to a data migration method, device, electronic device and storage medium provided by the present disclosure. Specific examples are used in this article to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method of the present disclosure and its core idea. At the same time, for those skilled in the art, according to the ideas of the present disclosure, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. A data migration method, characterized in that: include: Dividing a data segment in a volume into a plurality of data blocks, wherein the data blocks represent units of data migration; determining a migration status of a target data block to be migrated among the plurality of data blocks; When the migration state of the target data block is occupied by a write operation, storing the write operation data of the target data block in a target queue, and migrating the target data block from a source address to a target address; When the migration state of the target data block is occupied by a read operation, the target data block is read from the source address, and the target data block is migrated from the source address to the target address; The length of the target queue is determined as follows: Determine theoretical data based on the business load data model, where the theoretical data represents theoretical resource usage information, including theoretical CPU resource usage, theoretical memory resource usage, and theoretical ratio of new and old addresses of a write object target address for the target queue in a write IO request; Determine the theoretical optimal length of the target queue and the adjustment step size based on the theoretical data; Determining real-time data based on a business load data model and a waiting time for real-time read and write operations, wherein the real-time data represents real-time usage information of resources; According to the real-time data, the length of the target queue determined according to the theoretical data is adjusted according to an adjustment step, and / or the number of the target queues is adjusted.

2. The method according to claim 1, characterized in that Also includes: If the migration status of the target data block is occupied by a write operation, after migrating the target data block from the source address to the target address, writing the write operation data stored in the target queue to the source address and the target address, and updating the migration status of the target data block from occupied by a write operation to migration completed; In a case where the migration status of the target data block is occupied by a read operation, after the target data block is migrated from the source address to the target address, the migration status of the target data block is updated from occupied by a read operation to migration completed.

3. The method according to claim 2, characterized in that Writing the write operation data stored in the target queue to the source address and the target address includes: When the available space in the target queue is exhausted, or when the first write operation data stored in the target queue exceeds the preset processing cycle, the various write operation data stored in the target queue are written in batches to the source address and the corresponding target address corresponding to the write operation data, and the target queue includes the write operation data of the target data block once or multiple times.

4. The method according to claim 1, wherein The migration status includes write operation occupied, read operation occupied, no read or write operation, and migration completed. The determining the migration status of the target data block to be migrated among the multiple data blocks includes: Scan the target data block to obtain the read and write status of the target data block; Determining the latest migration status of the target data block according to the read and write status; The migration status of the target data block is determined according to the latest migration status.

5. The method according to any one of claims 1 to 4, characterized in that: Also includes: When the migration status of the target data block is no read or write operation, the target data block is migrated from the source address to the target address, and the migration status of the target data block is updated from write operation occupied to migration completed.

6. The method according to claim 1, characterized in that Also includes: Performing a read operation on a data block to determine a migration status of the data block; When the migration status of the data block is migration completed, reading from the target address corresponding to the data block; the migration status being migration completed indicates that the data block has completed migration from the source address to the target address; If the migration status of the data block is occupied by a write operation, a one-to-one correspondence is performed in the target queue according to the physical address of the data block to determine the position of the data block in the target queue; and data corresponding to the data block is read from the target queue. The migration status being occupied by a write operation indicates that the write operation data corresponding to the data block is stored in the target queue and the write operation data is not written back to the source address and target address corresponding to the data block. Performing a write operation on the data block to determine a migration status of the data block; When the migration status of the data block is migration completed, the write operation data is simultaneously written into the source address and the target address corresponding to the data block.

7. A data migration device, characterized in that: include: a partitioning module, configured to partition a data segment in a volume into a plurality of data blocks, wherein the data blocks represent units of data migration; a determination module, configured to determine a migration status of a target data block to be migrated among the plurality of data blocks; a first migration module, configured to, when the migration state of the target data block is occupied by a write operation, store the write operation data of the target data block in a target queue and migrate the target data block from a source address to a target address; a second migration module, configured to read the target data block from the source address and migrate the target data block from the source address to the target address when the migration state of the target data block is occupied by a read operation; The length of the target queue is determined as follows: Determine theoretical data based on the business load data model, where the theoretical data represents theoretical resource usage information, including theoretical CPU resource usage, theoretical memory resource usage, and theoretical ratio of new and old addresses of write object target addresses for a target queue in write IO requests; Determine the theoretical optimal length of the target queue and the adjustment step size based on the theoretical data; Determining real-time data based on a business load data model and a waiting time for real-time read and write operations, wherein the real-time data represents real-time usage information of resources; According to the real-time data, the length of the target queue determined according to the theoretical data is adjusted according to an adjustment step, and / or the number of the target queues is adjusted.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the steps of the data migration method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data migration method according to any one of claims 1 to 6 are implemented.

Images