Data migration method and device, equipment and medium

By gradually increasing the migration speed and analyzing the changes in the storage system resource in the data migration task, and adjusting the migration speed according to the business IO resource requirements, the problem of the migration speed in the existing technology cannot be dynamically adjusted, and data migration efficiency is improved.

CN119088288AActive Publication Date: 2024-12-06INSPUR SUZHOU INTELLIGENT TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410947256.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-12-06
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

During the data migration process of existing storage systems, the migration speed cannot be dynamically adjusted, resulting in increased service IO response delay or ineffective utilization of storage system resources.

Method used

After the data migration task is started, the migration speed is gradually increased, and the target relationship is determined by analyzing the changes in the storage system resources, and the migration speed is adjusted according to the business IO resource requirements, so as to dynamically adjust the target migration speed of the data migration task.

Benefits of technology

It realizes that the migration speed of data migration tasks is dynamically adjusted while ensuring the demand for business IO resources, and maximizes the utilization of storage system resources, improving the execution efficiency of data migration tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088288A_ABST
    Figure CN119088288A_ABST
Patent Text Reader

Abstract

The invention provides a data migration method and device, equipment and a medium, relates to the technical field of storage systems, and can improve the data migration efficiency by increasing the migration speed of a data migration task for multiple times in an observation stage of the earlier stage of executing the data migration task. The method comprises the following steps: analyzing a target association relationship between the variable quantity of a plurality of storage system resources associated with a data migration task and the variable quantity of a migration speed according to a data migration task, and according to the target association relationship and resource requirements of a plurality of storage system resources associated with a service IO; according to the method, the liftable speed increments corresponding to the multiple storage system resources associated with the data migration task are determined, and the target migration speed used for executing the data migration task can be determined through the multiple liftable speed increments, so that the migration speed of the data migration task is dynamically adjusted on the premise of ensuring the resource requirements of the service IO, and the data migration efficiency is improved. The target migration speed is obtained, storage system resources are utilized to the maximum extent, and the execution efficiency of the data migration task is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of storage systems, and in particular to a data migration method, device, equipment and medium. Background Art

[0002] The background data migration function of the storage system, including mirroring at the volume level, cloning, backup, remote replication and other functions at the disaster recovery level, all require double data backup and data migration across storage pools at the volume level. Because data migration tasks also occupy storage system resources, including bandwidth resources, CPU resources, memory resources, hard disk resources, etc., the migration speed of data migration tasks is too fast, which may increase the response delay of business IO and cause business jams.

[0003] At present, most storage vendors can reduce the migration speed of data migration tasks by adjusting the speed to give priority to business IO. However, the migration speed can only be adjusted to a fixed speed, and the migration speed cannot be adjusted dynamically according to the business IO load. When the business IO load is high, the data migration task may occupy the storage system resources of the business IO, thereby increasing the latency of the business IO. When the business IO load is low, the migration speed cannot be increased, and the storage system resources may not be effectively utilized. Summary of the invention

[0004] In view of this, embodiments of the present application provide a data migration method, apparatus, device and medium to overcome the above problems or at least partially solve the above problems.

[0005] A first aspect of an embodiment of the present application provides a data migration method, the method comprising:

[0006] After the data migration task is started, during the observation phase, the migration speed of the data migration task is increased multiple times according to a preset unit speed increment, and a target association relationship is determined by analyzing the change amounts of multiple storage system resources associated with the data migration task, where the target association relationship is used to characterize the association relationship between the change amounts of the multiple storage system resources associated with the data migration task and the change amount of the migration speed of the data migration task;

[0007] Determine the speed increments that can be increased for each of the multiple storage system resources associated with the data migration task according to the target association relationship and the resource demand of the multiple storage system resources associated with the business IO;

[0008] Based on the multiple increaseable speed increments, a target migration speed of the data migration task is determined, and the data migration task is executed according to the target migration speed.

[0009] Optionally, the target association relationship is determined by the following steps:

[0010] For each observation period of the observation phase, the migration speed is increased according to the preset unit speed increment, and a change amount of multiple storage system resources associated with the data migration task in the observation period is determined;

[0011] After the observation phase is finished, statistics are collected on the changes in the plurality of storage system resources corresponding to each observation period to obtain statistical results;

[0012] Based on the statistical results, the target association relationship is determined.

[0013] Optionally, the multiple storage system resources include at least CPU resources, memory resources, and hard disk resources, and determining the target association relationship based on the statistical result includes:

[0014] Based on the statistical result, for each storage system resource associated with the data migration task, calculate an average change amount of the storage system resource after the migration speed of the data migration task is increased by a preset unit speed increment;

[0015] Determining the target association relationship based on the corresponding relationship between the preset unit speed increment and the average change amount of the plurality of storage system resources;

[0016] The determining, according to the target association relationship and the resource demand of the multiple storage system resources associated with the business IO, the speed increments that can be increased corresponding to the multiple storage system resources associated with the data migration task respectively include:

[0017] Based on the first resource margins of the multiple storage system resources currently available for the data migration task and the target association relationship, calculate the speed increments that can be increased corresponding to each of the multiple storage system resources;

[0018] The first resource margin is determined based on the total amount of the multiple storage system resources, the resource demand of the business IO for the multiple storage system resources, and the reserved resource amount of the multiple storage system resources.

[0019] Optionally, the method further comprises:

[0020] Determining, based on the type of a hard disk corresponding to the hard disk resource associated with the data migration task, a statistical time interval corresponding to the hard disk;

[0021] Based on the statistical time interval, within a first statistical period before the start of the observation phase, the number of times the hard disk is in a busy state is counted, and a first hard disk resource occupancy rate of the hard disk resource before the start of the observation phase is calculated;

[0022] Based on the statistical time interval, within multiple observation cycles after the start of the observation phase, the number of times the hard disk is in a busy state is counted, and multiple second hard disk resource occupancy rates of the hard disk resource in the observation phase are calculated;

[0023] Calculating an average change amount of the hard disk resource according to the plurality of second hard disk resource occupancy rates and the first hard disk resource occupancy rate;

[0024] After obtaining the average change of the hard disk resource, the average change of the hard disk resource is updated based on the current change of the hard disk resource in the latest observation period, the average change of the hard disk resource, the preset unit speed increment, and the first weight of the average change of the hard disk resource and the second weight of the current change of the hard disk resource in the latest observation period.

[0025] Optionally, when the multiple storage system resources include bandwidth resources, determining the target association relationship based on the statistical result includes:

[0026] When the data migration task and the service IO share the same IO port, determining the bandwidth resources available for the data migration task based on the bandwidth resources occupied by the service IO;

[0027] Based on the bandwidth resources available for the data migration task, a speed increment corresponding to the bandwidth resources is determined.

[0028] Optionally, the method further comprises:

[0029] In the first observation period of the observation phase, when the change amount of the storage system resources corresponding to the business IO is greater than the preset change amount or the storage system resources occupied by the business IO is greater than the first preset threshold, the change amounts of the multiple storage system resources obtained in the first observation period are not counted;

[0030] In the second observation period of the observation phase, when the load corresponding to the business IO is greater than the second preset threshold and the data volume of the data migration task is less than the third preset threshold, the changes in the multiple storage system resources obtained in the second observation period are not counted.

[0031] Optionally, determining a target migration speed of the data migration task based on the plurality of increaseable speed increments includes:

[0032] The target migration speed of the data migration task is calculated based on the smallest scalable speed increment among the multiple scalable speed increments and the current migration speed of the data migration task after the observation phase ends.

[0033] Optionally, the method further comprises:

[0034] In the process of executing the data migration task according to the target migration speed, the target migration speed is adjusted based on the change of the resource demand of the service IO on the multiple storage system resources and the target association relationship.

[0035] A second aspect of an embodiment of the present application provides a data migration device, the device comprising:

[0036] An observation module is used for, after the data migration task is started, increasing the migration speed of the data migration task multiple times according to a preset unit speed increment during an observation phase, and determining a target association relationship by analyzing the change amounts of multiple storage system resources associated with the data migration task, wherein the target association relationship is used to characterize the association relationship between the change amounts of the multiple storage system resources associated with the data migration task and the change amount of the migration speed of the data migration task;

[0037] A module for determining the increase in speed that can be increased is used to determine the increase in speed that can be increased corresponding to each of the multiple storage system resources associated with the data migration task according to the target association relationship and the resource demand of the multiple storage system resources associated with the business IO;

[0038] The data migration task execution module is used to determine a target migration speed of the data migration task based on the multiple speed increments that can be increased, and execute the data migration task according to the target migration speed.

[0039] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the data migration method as described in the first aspect.

[0040] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the data migration method as described in the first aspect is implemented.

[0041] Beneficial effects of this application:

[0042] The embodiments of the present application provide a data migration method, apparatus, device and medium, which increases the migration speed of the data migration task multiple times according to preset unit speed increments during the observation phase, and determines the target association relationship by analyzing the change amount of multiple storage system resources associated with the data migration task. The target association relationship is used to characterize the association relationship between the change amount of multiple storage system resources associated with the data migration task and the change amount of the migration speed of the data migration task. Then, based on the target association relationship and the resource demand of the multiple storage system resources associated with the business IO, the speed increments that can be increased corresponding to each of the multiple storage system resources associated with the data migration task are determined. Finally, based on the multiple speed increments that can be increased, the target migration speed of the data migration task is determined, and the data migration task is executed according to the target migration speed.

[0043] In the early observation stage of executing the data migration task, the migration speed of the data migration task can be increased multiple times to analyze the target correlation relationship between the change amount of multiple storage system resources associated with the data migration task and the change amount of the migration speed, and then the speed increments corresponding to the multiple storage system resources associated with the data migration task can be determined according to the target correlation relationship and the resource requirements of the multiple storage system resources associated with the business IO. Through the multiple speed increments, the target migration speed for executing the data migration task can be determined, so as to achieve the target migration speed by dynamically adjusting the migration speed of the data migration task under the premise of ensuring the resource requirements of the business IO. The data migration task is executed by using the target migration speed, which can maximize the use of storage system resources and improve the execution efficiency of the data migration task. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application.

[0045] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 It is a flowchart of a data migration method provided by an embodiment of the present application;

[0047] Figure 2 is a schematic diagram of a storage resource occupancy situation provided by an embodiment of the present application;

[0048] Figure 3This is an overall framework diagram of a data migration method provided by an embodiment of the present application;

[0049] Figure 4 It is a schematic diagram of a data migration device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0050] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] The background data migration function of the storage system, including mirroring at the volume level, cloning, backup, remote replication and other functions at the disaster recovery level, all require double data backup and data migration across storage pools at the volume level. Because data migration will also occupy storage resources, including transmission networks, CPUs, back-end hard disks, etc., if the migration speed of the data migration task is too fast, it may cause the response delay of business IO to increase, causing business jams.

[0053] Currently, most storage vendors can reduce the data migration speed by adjusting the speed to give priority to business IO. However, speed adjustment can only set a fixed migration speed and cannot automatically adjust the migration speed according to the business load. When the business load is high, it may cause the business IO delay to increase. When the business load is low, the migration speed cannot be increased.

[0054] Figure 1 is a flow chart of a data migration method provided by an embodiment of the present application, such as Figure 1 As shown, the method includes:

[0055] In step S101, after the data migration task is started, during the observation phase, the migration speed of the data migration task is increased multiple times according to a preset unit speed increment, and a target association relationship is determined by analyzing the change amounts of multiple storage system resources associated with the data migration task, wherein the target association relationship is used to characterize the association relationship between the change amounts of the multiple storage system resources associated with the data migration task and the change amount of the migration speed of the data migration task;

[0056] In step S102, according to the target association relationship and the resource demand of the multiple storage system resources associated with the business IO, the speed increments that can be increased corresponding to the multiple storage system resources associated with the data migration task are determined;

[0057] In step S103, a target migration speed of the data migration task is determined based on the plurality of increaseable speed increments, and the data migration task is executed according to the target migration speed.

[0058] First, in step S101, when the data migration task is started, it immediately enters the observation phase, which is the early stage of the entire data migration task execution process. During the observation phase, the migration speed of the data migration task is increased multiple times according to a preset unit speed increment, that is, the migration speed is gradually increased.

[0059] After each increase in the migration speed, record the changes in multiple storage system resources associated with the data migration task, such as CPU resources, bandwidth resources, hard disk resources, and memory resources.

[0060] By analyzing the relationship between the change in multiple storage system resources associated with the data migration task and the change in the migration speed of the data migration task, a target association relationship characterizing the association relationship between the change in multiple storage system resources and the change in the migration speed is determined, which may be a functional relationship or a model.

[0061] Further, in step S102, the target association relationship between the change amount of the storage system resource corresponding to the data migration task and the change amount of the migration speed of the data migration task, and the resource demand amount of multiple storage system resources associated with the business IO of the storage system are determined by correlation analysis, regression analysis, decision tree algorithm, support vector machine algorithm or artificial neural network algorithm, so that multiple speed increments that can be improved for the data migration task can be determined. The multiple speed increments that can be improved correspond to the multiple storage system resources one-to-one. According to the target association relationship between the change amount of each storage system resource and the change amount of the migration speed of the data migration task, and in combination with the resource demand amount of multiple storage system resources associated with the business IO, the speed increment that can be improved for the data migration task corresponding to each storage system resource can be determined, wherein the speed increment that can be improved is the speed increment by which the current migration speed of the data migration task can be improved, and the speed increment that can be improved for the data migration task corresponding to each storage system resource is: it can ensure that the corresponding storage system resource can maximize the use of the speed increment of the corresponding storage system resource under the premise of meeting the resource demand amount of the business IO.

[0062] For example, based on the target correlation between the change in CPU resource occupancy caused by data migration tasks and the change in migration speed of data migration tasks, combined with the resource demand for CPU resources by the business IO of the storage system, the corresponding speed increase of CPU resources can be obtained.

[0063] According to the target correlation between the change in the memory resource occupancy rate of the data migration task and the change in the migration speed of the data migration task, combined with the resource demand for memory resources by the business IO of the storage system, the corresponding speed increase of the memory resource can be obtained.

[0064] According to the target correlation between the change in the occupancy rate of the hard disk resources of the data migration task and the change in the migration speed of the data migration task, combined with the resource demand of the storage system's business IO for the hard disk resources, the corresponding speed increase of the hard disk resources can be obtained.

[0065] According to the target correlation between the change in bandwidth resource occupancy rate of the data migration task and the change in the migration speed of the data migration task, combined with the resource demand for bandwidth resources of the storage system's business IO, the corresponding speed increment of the bandwidth resource can be obtained.

[0066] Finally, in step S103, after obtaining multiple scalable speed increments, one scalable speed increment is selected from the multiple scalable speed increments to determine a target migration speed for executing the data migration task, and the data migration task is executed according to the target migration speed.

[0067] Through the above-mentioned embodiments, in the observation stage before executing the data migration task, by repeatedly increasing the migration speed of the data migration task, the target association relationship between the change amount of multiple storage system resources associated with the data migration task and the change amount of the migration speed can be analyzed, and then according to the target association relationship and the resource requirements of multiple storage system resources associated with the business IO, the speed increments that can be increased corresponding to each of the multiple storage system resources associated with the data migration task can be determined. Through the multiple speed increments that can be increased, the target migration speed for executing the data migration task can be determined, so as to achieve the target migration speed by dynamically adjusting the migration speed of the data migration task under the premise of ensuring the resource requirements of the business IO. The data migration task is executed by using the target migration speed, which can maximize the use of storage system resources and improve the execution efficiency of the data migration task.

[0068] Optionally, the target association relationship is determined by the following steps:

[0069] In step S201, for each observation period of the observation phase, the migration speed is increased according to the preset unit speed increment, and the change amount of multiple storage system resources associated with the data migration task in the observation period is determined;

[0070] In step S202, after the observation phase ends, statistics are collected on the changes in the plurality of storage system resources corresponding to each observation period to obtain statistical results;

[0071] In step S203, the target association relationship is determined based on the statistical result.

[0072] Specifically, in one embodiment, in step S201, when a data migration task is received, the amount of data required for data migration contained in the data migration task can be obtained first, and the number of observation cycles included in the observation phase corresponding to the data migration task can be determined based on the amount of data (the more data, the more observation cycles, and vice versa, the less data, the fewer observation cycles), and the duration of the observation phase accounts for a smaller proportion than the execution duration of the entire data migration task.

[0073] After the storage system receives the start signal of the data migration task, the migration speed of the data migration task is the default initial migration speed (the value of the initial migration speed is 0 or less than the value of the preset speed threshold). At the same time, the data migration task immediately enters the observation phase. The observation phase is the early stage of the data migration task execution process. The observation phase includes multiple observation cycles. In the observation phase, at the beginning of each observation cycle, the migration speed of the data migration task is increased by a preset unit speed increment. The preset unit speed increment can be set according to the actual speed regulation requirements of the migration speed of the migration task.

[0074] For example, taking the initial migration speed as 5MB / s and the preset unit speed increment as 10MB / s, at the beginning of the first observation period, the migration speed of the data migration task is increased from 5MB / s by 10MB / s, so that the current migration speed is 15MB / s, and the migration speed of the data migration task is increased from 15MB / s by 10MB / s, so that the current migration speed is 25MB / s. The speed adjustment process from the third observation period to the last observation period is similar until the end of the observation phase.

[0075] During each observation cycle in the observation phase, changes in multiple storage system resources associated with the data migration task are collected, wherein the change in each storage system resource in the multiple storage system resources associated with the data migration task is the difference between the start time and the end time of the observation cycle for each storage system resource, and the multiple storage system resources include CPU resources, memory resources, hard disk resources, bandwidth resources, etc.

[0076] Further, in step S202, after the observation phase ends, the change amount of each storage system resource associated with the data migration task in multiple observation periods can be counted, that is, the statistical results are obtained, wherein in the statistical results, each storage system resource will correspond to multiple changes, and the number of changes of each storage system resource is the number of observation periods.

[0077] Finally, in step S203, the changes in the multiple storage system resources associated with the data migration task and the changes in the migration speed in each observation period of the observation phase in the statistical results are analyzed to obtain a target association relationship for characterizing the association relationship between the changes in the multiple storage system resources associated with the data migration task and the changes in the migration speed of the data migration task.

[0078] The above process of analyzing the change amount of multiple storage system resources and the change amount of migration speed in each observation period during the observation phase may adopt the following methods:

[0079] 1. The correlation coefficients between the changes in the resources of multiple storage systems and the changes in the migration speed in each observation period of the observation phase are calculated, such as the Pearson correlation coefficient, the Spearman rank correlation coefficient, etc., to measure the correlation between the changes in the resources of multiple storage systems and the changes in the migration speed in each observation period of the observation phase. The closer the correlation coefficient is to 1 or -1, the higher the correlation degree is, and the target correlation relationship is determined by the correlation degree.

[0080] For example, by calculating the correlation coefficient between the change in the utilization of the CPU resources of the data migration task and the change in the migration speed of the data migration task, if the correlation coefficient is close to 1, it means that the change in the utilization of the CPU resources of the data migration task is positively correlated with the change in the migration speed of the data migration task, that is, the target correlation relationship is that the higher the utilization of the CPU resources, the higher the migration speed of the data migration task.

[0081] 2. Use regression models such as linear regression and multivariate linear regression, take the change in storage system resources as the independent variable, and the change in the migration speed of the data migration task as the dependent variable, and establish a mathematical model. The mathematical model is used to describe the correlation between the change in multiple storage system resources and the change in the migration speed in each observation period of the observation stage, so as to determine the target correlation based on the correlation.

[0082] For example, a linear regression model is used to take the bandwidth resources occupied by the data migration task as the independent variable and the migration speed of the data migration task as the dependent variable, and a mathematical model is established to describe the relationship between the two. By analyzing the coefficients and significance of the linear regression model, the degree of influence of bandwidth resources on the migration speed of the data migration task can be determined, and the target association relationship can be determined through the degree of influence.

[0083] 3. Use the decision tree algorithm, take the change in storage system resources as a feature, and the change in the migration speed of the data migration task as the target variable, to build a decision tree model. The branches of the decision tree are used to represent the degree of influence of the change in different storage system resources on the change in the migration speed, and the target association relationship is determined by the degree of influence.

[0084] For example, the hard disk resources occupied by the data migration task are used as features, and the migration speed of the data migration task is used as the target variable to build a decision tree model. Through the branches of the decision tree, the influence of different hard disk resource occupancy rates on the migration speed of the data migration task can be determined, and the target association relationship can be determined based on the influence degree.

[0085] Through the above-mentioned embodiments, in the observation phase after the data migration task is started, by repeatedly increasing the migration speed of the data migration task and analyzing the changes in the storage system resources managed by the data migration task, the target correlation relationship between the changes in multiple storage system resources associated with the data migration task and the changes in the migration speed in each observation period of the observation phase can be accurately determined.

[0086] Optionally, the multiple storage system resources include at least CPU resources, memory resources and hard disk resources, and the step S203 includes:

[0087] In step S2031, based on the statistical result, for each storage system resource associated with the data migration task, an average change amount of the storage system resource after the migration speed of the data migration task is increased by a preset unit speed increment is calculated;

[0088] In step S2032, the target association relationship is determined based on the correspondence between the preset unit speed increment and the average change amount of the plurality of storage system resources;

[0089] The step S102 includes:

[0090] In step S1021, based on the first resource margins of the multiple storage system resources currently available for the data migration task and the target association relationship, the speed increments that can be increased corresponding to the multiple storage system resources are calculated;

[0091] The first resource margin is determined based on the total amount of the multiple storage system resources, the resource demand of the business IO for the multiple storage system resources, and the reserved resource amount of the multiple storage system resources.

[0092] Specifically, the speed increments corresponding to the CPU resources, memory resources, and hard disk resources in the plurality of storage system resources are determined by the following steps:

[0093] First, in step S2031, based on the statistical results of each storage system resource (i.e., CPU resources, memory resources, and hard disk resources) associated with the data migration task, the average change in each storage system resource corresponding to the data migration task in multiple observation cycles during the observation phase is calculated after the migration speed is increased by multiple preset unit speed increments, wherein the average change refers to the average value of the change in the storage system resources corresponding to multiple observation cycles.

[0094] The calculation formula for the average change is: A=(A1+A2+…+An) / n, where A is the average change of storage system resources, A1 is the change of storage system resources in the first observation period, A2 is the change of storage system resources in the second observation period, An is the change of storage system resources in the third observation period, and n is the number of observation periods.

[0095] For example, taking the initial migration speed as 5MB / s and the preset unit speed increment as 10MB / s, during the observation phase of memory resources, when the data migration task is started, the migration speed of the data migration task is increased by 10MB / s at the beginning of each observation cycle, and at the end of the observation cycle, the change in memory resources is recorded. The change in memory resources can be represented by the growth in memory usage.

[0096] 1st observation period: memory usage increased by 5%;

[0097] 2nd observation period: memory usage increased by 7%;

[0098] 3rd observation period: memory usage increased by 9%;

[0099] 4th observation period: memory usage increased by 11%;

[0100] 5th observation period: memory usage increased by 13%;

[0101] It can be calculated that when the data migration speed is increased by 10MB / s, the average change in memory resources is: A=(5%+7%+9%+11%+13%) / 5=9.8%.

[0102] The calculation methods of CPU resources and hard disk resources are similar and will not be repeated here.

[0103] Further, in step S2032, a target association relationship is determined based on the correspondence between the preset unit speed increment and the average change of multiple storage system resources, that is, the target association relationship is: the correspondence between the preset unit speed increment and the average change of multiple storage system resources.

[0104] For example, when the average change in CPU resources is 10%, the average change in memory resources is 9.8%, and the average change in hard disk resources is 12%, the target association relationship is: for each increase in the budget unit speed increment of the data migration task, the CPU resource occupancy rate corresponding to the data migration task will increase by 10%, the memory resource occupancy rate will increase by 9.8%, and the hard disk resource occupancy rate will increase by 12%.

[0105] Further, in step S1021, based on the total amount of multiple storage system resources, the resource demand of the multiple storage system resources by the business IO, and the reserved resource amount of the multiple storage system resources, the first resource margin of each storage system resource is determined. The first resource margin indicates the resource amount of each storage system resource that can currently be allocated to the data migration task, that is, the first resource margin of each storage system resource is: the total amount of each storage system resource minus the demand of the storage system resource by the business IO, minus the reserved resource amount of the storage system resource. The reserved resource amount is set according to the actual usage of each storage system resource, for example 20%. The reserved resource amount is used to ensure that there are sufficient storage system resources for a task when it is running, thereby ensuring the normal operation of the task and avoiding slow running or crash of the task.

[0106] Figure 2 is a schematic diagram of a storage resource occupancy situation provided by an embodiment of the present application, such as Figure 2 As shown, each storage system resource is divided into three parts: reserved resource amount, the first resource margin of the data migration task, and the demand for storage system resources by business IO. The water level between the first resource margin of the data migration task and the demand for storage system resources by business IO is dynamically adjusted according to the change of the migration speed of the data migration task.

[0107] Then, according to the first resource margin and the target association relationship of each storage system resource, the accelerable speed increment corresponding to each storage system resource is calculated. The accelerable speed increment indicates the migration speed that can be increased corresponding to each storage system resource.

[0108] The calculation formula for the increaseable speed increment of each storage system resource is: the first resource margin of each storage system resource / the average change amount of each storage system resource*the preset unit speed increment.

[0109] For example, the target association relationship is: when the migration speed of the data migration task increases by each budget unit speed increment, the CPU resource occupancy rate corresponding to the data migration task will increase by 10%, the memory resource occupancy rate will increase by 9.8%, and the hard disk resource occupancy rate will increase by 12%. The first resource margin of the CPU resource is 50%, the first resource margin of the memory resource is 30%, and the first resource margin of the hard disk resource is 70%. Then, the speed increment corresponding to the CPU resource is: 50% / 10%*10MB / s=50MB / s, the speed increment corresponding to the memory resource is: 30% / 9.8%*10MB / s=30.6MB / s, and the speed increment corresponding to the hard disk resource is: 70% / 12%*10MB / s=58.3MB / s.

[0110] Through the above embodiment, the increase in speed of each storage system resource can be calculated according to the first resource margin and target association relationship of multiple storage system resources currently available for the data migration task. This increase in speed can be used to optimize the performance of the data migration task and further adjust the configuration of the storage system resources to achieve better migration speed and performance.

[0111] Optionally, the method further comprises:

[0112] The method further comprises:

[0113] In step S301, based on the type of the hard disk corresponding to the hard disk resource associated with the data migration task, a statistical time interval corresponding to the hard disk is determined;

[0114] In step S302, based on the statistical time interval, within a first statistical period before the start of the observation phase, the number of times the hard disk is in a busy state is counted, and a first hard disk resource occupancy rate of the hard disk resource before the start of the observation phase is calculated;

[0115] In step S303, based on the statistical time interval, within multiple observation cycles after the start of the observation phase, the number of times the hard disk is in a busy state is counted, and multiple second hard disk resource occupancy rates of the hard disk resource in the observation phase are calculated;

[0116] In step S304, the average change amount of the hard disk resource is calculated according to the plurality of second hard disk resource occupancy rates and the first hard disk resource occupancy rate;

[0117] In step S305, after obtaining the average change of the hard disk resource, the average change of the hard disk resource is updated based on the current change of the hard disk resource in the latest observation period, the average change of the hard disk resource, the preset unit speed increment, and the first weight of the average change of the hard disk resource and the second weight of the current change of the hard disk resource in the latest observation period.

[0118] Specifically, in one embodiment, first, in step S301, according to the type of the hard disk corresponding to the hard disk resource associated with the data migration task, a statistical time interval corresponding to the hard disk is determined;

[0119] Different types of hard disks have different IO response delays. Therefore, when obtaining the current occupancy rate of the hard disk, it is necessary to set different statistical time intervals of statistical cycles according to different types of hard disks. The statistical cycle is used to count the proportion of time that the hard disk is in a busy state. The proportion of time in the busy state can be used to determine the occupancy rate of the hard disk resources. As a result, for hard disks of different media, the statistical time interval t of the statistical cycle can be set differently when the statistical cycle remains unchanged.

[0120] For example, for a 7200 rpm disk with a larger capacity, set the statistical interval t = 10ms; for a 10,000 rpm disk, set the statistical interval t = 5ms; for a SAS interface SSD, set the statistical interval t = 2ms; for an NVMe interface SSD, set the statistical interval t = 1ms. The statistical period n is set to 5s.

[0121] The calculation formula for the hard disk resource occupancy rate is: B = (n*t) / T, where B is the hard disk resource occupancy rate, n is the number of times the hard disk is busy within the statistical cycle, t is the statistical time interval of the statistical cycle, and T is the statistical cycle.

[0122] Further, in step S302, according to the statistical time interval of the hard disk type, the number of times the hard disk is in a busy state is counted within the first statistical period before the observation phase of the data migration task begins, and the first hard disk resource occupancy rate before the observation phase of the hard disk resource begins is calculated.

[0123] For example, for a disk with a rotation speed of 10,000, the statistical time interval t of the statistical period is set to 5ms, and the statistical period is 5s. The number of IO queues is counted every 5ms during the statistical period. If the IO queue is empty, the hard disk is in an idle state within 5ms; if the number of IO queues is greater than 0, the hard disk is in a busy state within 5ms.

[0124] Further, in step S303, according to the statistical time interval, the number of times the hard disk is in a busy state is counted within multiple observation cycles after the observation phase of the data migration task begins, and multiple second hard disk resource occupancy rates of the hard disk resources during the observation phase are calculated, that is, for each observation cycle, according to the statistical time interval, the number of times the hard disk is in a busy state in each observation cycle is counted to obtain the second hard disk resource occupancy rate corresponding to each observation cycle.

[0125] Further, in step S304, based on the multiple second hard disk resource occupancy rates obtained in step S303 and the first hard disk resource occupancy rate, the changes in the multiple hard disk resource occupancy rates between the multiple second hard disk resource occupancy rates and the first hard disk resource occupancy rate are calculated, and the average of the changes in the multiple hard disk resource occupancy rates is calculated to obtain the average change in the hard disk resources.

[0126] Finally, in step S305, after obtaining the average change in the hard disk resources, since the weight of the hard disk resource occupancy rate corresponding to the data migration task is different at different times, for example, at the latest moment, the change in the hard disk resources can better reflect the current usage of the hard disk. Therefore, it is necessary to update the average change in the hard disk resources based on the average change in the hard disk resources during the observation phase and the current change in the hard disk in the latest observation period to ensure that the target correlation between the obtained average change in the hard disk resources and the preset unit speed increment is more accurate.

[0127] Specifically, a first weight can be set for the average change in the hard disk resources obtained during the observation phase, and a second weight can be set for the current change in the hard disk resources in the latest observation period. The update formula for the average change in the hard disk resources is as follows: C=w2*C2+w1*C1, wherein C is the average change in the hard disk resources after the update, C1 is the average change in the hard disk resources obtained during the observation phase, C2 is the current change in the hard disk resources in the latest observation period, w1 is the first weight, w2 is the second weight, and w1+w2=1. The first weight and the second weight can be set according to actual conditions.

[0128] Through the above embodiment, the average change of hard disk resources can be updated according to the average change of hard disk resources in the observation phase and the current change of the hard disk in the latest observation period to obtain accurate target association relationship corresponding to the hard disk resources.

[0129] Optionally, when the plurality of storage system resources include bandwidth resources, the method further comprises:

[0130] In step S401, when the data migration task and the service IO share the same IO port, the bandwidth resources available for the data migration task are determined based on the bandwidth resources occupied by the service IO;

[0131] In step S402, based on the bandwidth resources available for the data migration task, the increase in speed corresponding to the bandwidth resources is determined.

[0132] Specifically, in step S401, the server where the storage system is located has multiple IO ports. When processing data migration tasks and business IO at the same time, there may be a situation where the data migration task and the business IO share the same IO port. In this case, there will be competition for bandwidth resources between the data migration task and the business IO. Therefore, it is necessary to ensure that the bandwidth resources of the business IO are not preempted by the data migration task. First, it is necessary to determine the bandwidth resources available for the data migration task based on the bandwidth resources occupied by the business IO. The bandwidth resources occupied by the business IO are the bandwidth resources required to ensure that the business IO can perform IO operations normally. Therefore, the bandwidth resources available for the data migration task can be determined based on the difference between the total bandwidth resources and the bandwidth resources occupied by the business IO. For example, the speed of the total bandwidth resources of the shared IO port is 1Gbps, and the business IO occupies 0.8Gbps of bandwidth resources per second. Therefore, the bandwidth resources available for the data migration task are 0.2Gbps.

[0133] Further, in step S402, calculation is performed based on the available bandwidth resources and the target association relationship for the data migration task, and the speed increment corresponding to the bandwidth resources can be determined. For example, after determining that the available bandwidth resources for the data migration task are 0.2 Gbps, it can be known through conversion that the amount of data that can be transmitted per second at 0.2 Gbps is 25 MB. Therefore, it can be determined that the upper limit of the migration speed of the data migration task for the bandwidth resources is 25 MB / s. Therefore, based on the upper limit of the migration speed of the data migration task for the bandwidth resources and the current migration speed of the data migration task, the speed increment corresponding to the bandwidth resources can be determined.

[0134] It should be noted that the scalable speed increment may be greater than zero or less than or equal to 0. For example, when the current migration speed of the data migration task is 50 MB / s, the scalable speed increment corresponding to the bandwidth resource is -25 MB / s.

[0135] Through the above embodiments, the speed increment that can be increased can be determined based on the bandwidth resources, so as to avoid the occupation of the bandwidth resources required for the business IO during the speed adjustment process of the data migration task, so as to ensure that the IO operation of the business IO can be performed normally when the data migration task shares the IO port with the business IO.

[0136] Optionally, the method further comprises:

[0137] In step S501, in the first observation period of the observation phase, when the change amount of the storage system resources corresponding to the business IO is greater than a preset change amount or the storage system resources occupied by the business IO are greater than a first preset threshold, the change amounts of the multiple storage system resources obtained in the first observation period are not counted;

[0138] In step S502, in the second observation period of the observation phase, when the load corresponding to the business IO is greater than the second preset threshold and the data volume of the data migration task is less than the third preset threshold, the change amounts of multiple storage system resources obtained in the second observation period are not counted.

[0139] Specifically, in step S501, during the first observation period of the observation phase, the first observation period is an observation period in which the change in storage system resources corresponding to the business IO is greater than a preset change or the storage system resources occupied by the business IO is greater than a first preset threshold. In order to ensure that in the first observation period, due to the large fluctuation in business IO or the business IO almost completely occupies the storage system resources, the acquired change in storage system resources may be affected by the business IO, thereby interfering with subsequent statistical results. Therefore, the changes in multiple storage system resources acquired in the first observation period will not be counted, so as to avoid large errors in the average changes in multiple storage system resources calculated.

[0140] Furthermore, in the observation phase, when the data migration task causes the storage system resource usage to reach or approach a peak value, in order to ensure the normal operation of the storage system, the observation phase of the data migration task will be directly terminated.

[0141] In step S502, in the second observation period of the observation phase, the second observation period is an observation period in which the load corresponding to the business IO is greater than the second preset threshold and the data volume of the data migration task is less than the third preset threshold. When the load of the business IO is high (the load of the business IO is greater than the second preset threshold, indicating that the load of the business IO is high), and the data volume of the data migration task is small (the data volume of the data migration task is less than the third preset threshold, indicating that the data volume of the data migration task is small), therefore, the changes in the multiple storage system resources obtained in the second observation period will not be counted, but statistics will be counted after the load of the business IO is reduced. This ensures the reliability and accuracy of the statistical results and avoids deviations in the statistical results.

[0142] Optionally, the step S103 includes:

[0143] In step S1031, a target migration speed of the data migration task is calculated based on the smallest scalable speed increment among the multiple scalable speed increments and the current migration speed of the data migration task after the end of the observation phase.

[0144] Specifically, in step S1031, the multiple speed increments corresponding to the multiple storage system resources may be of different sizes. Therefore, it is necessary to take into account the various storage system resources and select one of the multiple speed increments and the current migration speed of the data migration task after the end of the observation phase to determine the target migration speed of the data migration task.

[0145] For example, according to the above embodiment, assuming that the multiple speed increments that can be increased are 20MB / s, 30MB / s and 40MB / s respectively, and the current migration speed of the data migration task after the observation phase is 30MB / s, then 20MB / s can be selected to increase the migration speed of the data migration task according to the principle of "the smallest speed increment that can be increased among the multiple speed increments that can be increased", so that the final target migration speed is 50MB / s.

[0146] Through the above embodiments, it is possible to ensure that the data migration task can maximize the use of various storage system resources while taking into account the resource demand of the business IO on various storage system resources and not being preempted by the data migration task.

[0147] Optionally, the method further comprises:

[0148] In step S701, in the process of executing the data migration task according to the target migration speed, the target migration speed is adjusted based on the change of the resource demand of the business IO for multiple storage system resources and the target association relationship.

[0149] Specifically, in step S701, after determining the target migration speed, the data migration task should be executed according to the target migration speed. In addition, in the process of executing the data migration task according to the target migration speed, it is also necessary to monitor in real time the changes in the resource demands of the business IO for the multiple storage system resources. When the change range of the monitored business IO resource demands for the multiple storage system resources is greater than the preset change range, it is also necessary to dynamically adjust the target migration speed based on the changes in the resource demands of the business IO for the multiple storage system resources and the target association relationship.

[0150] When the amount of reduction (increase) in the resource demand of multiple storage system resources by business IO is greater than the preset change range, the target migration speed of the data migration task can be increased (reduced). The process of increasing (reducing) can dynamically adjust the target migration speed according to the amount of reduction in the resource demand of multiple storage system resources by business IO and the target association relationship. In addition, in the process of dynamically adjusting the target migration speed, it is necessary to ensure that the resource demand of business IO for each storage system resource is not preempted by the data migration task.

[0151] Figure 3 is an overall framework diagram of a data migration method provided by an embodiment of the present application, such as Figure 3 As shown, after obtaining the target migration speed, the change in the demand for storage system resources of the business IO can be continuously monitored during the process of executing the data migration task according to the target migration speed. When the change in the demand for storage system resources of the business IO is increasing, the migration speed of the data migration task is reduced. When the change in the demand for storage system resources of the business IO is decreasing, the migration speed of the data migration task is increased, and monitoring is performed until the data migration task is completed.

[0152] Through the above embodiments, it is possible to ensure that during the execution of the data migration task at the target migration speed, the target migration speed is dynamically adjusted according to the changes in the resource demand of the business IO for multiple storage system resources, thereby realizing an adaptive speed control process of the data migration task.

[0153] Figure 4 is a schematic diagram of a data migration device provided by an embodiment of the present application. Figure 4 As shown, the device comprises:

[0154] The observation module 11 is used for, after the data migration task is started, increasing the migration speed of the data migration task multiple times according to a preset unit speed increment during the observation phase, and determining a target association relationship by analyzing the change amount of multiple storage system resources associated with the data migration task, wherein the target association relationship is used to characterize the association relationship between the change amount of the multiple storage system resources associated with the data migration task and the change amount of the migration speed of the data migration task;

[0155] The speed increment determination module 12 is used to determine the speed increments that can be increased corresponding to the multiple storage system resources associated with the data migration task according to the target association relationship and the resource demand of the multiple storage system resources associated with the business IO;

[0156] The data migration task execution module 13 is used to determine a target migration speed of the data migration task based on the multiple speed increments that can be increased, and execute the data migration task according to the target migration speed.

[0157] Optionally, the device further comprises:

[0158] a determination module, configured to increase the migration speed according to the preset unit speed increment for each observation period of the observation phase, and determine the change amount of multiple storage system resources associated with the data migration task in the observation period;

[0159] A statistical result acquisition module, used for, after the observation phase is over, performing statistics on the changes in the plurality of storage system resources corresponding to each observation period to obtain statistical results;

[0160] The target association relationship determination module is used to determine the target association relationship based on the statistical result.

[0161] Optionally, the multiple storage system resources include at least CPU resources, memory resources and hard disk resources, and the target association relationship determination module includes:

[0162] an average change amount determination module, configured to calculate, based on the statistical result, for each storage system resource associated with the data migration task, an average change amount of the storage system resource after the migration speed of the data migration task is increased by a preset unit speed increment;

[0163] a target association relationship determination unit, configured to determine the target association relationship based on a correspondence between the preset unit speed increment and an average change amount of the plurality of storage system resources;

[0164] The speed increment determination module 12 includes:

[0165] A speed increment determination unit, configured to calculate the speed increments corresponding to the plurality of storage system resources based on first resource margins currently available to the data migration task and the target association relationship;

[0166] The first resource margin is determined based on the total amount of the multiple storage system resources, the resource demand of the business IO for the multiple storage system resources, and the reserved resource amount of the multiple storage system resources.

[0167] Optionally, the device further comprises:

[0168] A statistical time interval determination module, configured to determine a statistical time interval corresponding to a hard disk based on the type of the hard disk corresponding to the hard disk resource associated with the data migration task;

[0169] A first hard disk resource occupancy rate calculation module is used to count the number of times the hard disk is in a busy state within a first statistical period before the start of the observation phase based on the statistical time interval, and calculate the first hard disk resource occupancy rate of the hard disk resource before the start of the observation phase;

[0170] A second hard disk resource occupancy rate calculation module is used to count the number of times the hard disk is in a busy state within multiple observation cycles after the start of the observation phase based on the statistical time interval, and calculate multiple second hard disk resource occupancy rates of the hard disk resource in the observation phase;

[0171] an average change amount calculation module for hard disk resources, used for calculating the average change amount of the hard disk resources according to the plurality of second hard disk resource occupancy rates and the first hard disk resource occupancy rate;

[0172] The average change amount update module of the hard disk resource is used to update the average change amount of the hard disk resource based on the current change amount of the hard disk resource in the latest observation period, the average change amount of the hard disk resource, the preset unit speed increment, and the first weight of the average change amount of the hard disk resource and the second weight of the current change amount of the hard disk resource in the latest observation period after obtaining the average change amount of the hard disk resource.

[0173] Optionally, when the plurality of storage system resources include bandwidth resources, the target association relationship determination module includes:

[0174] An available bandwidth resource determination module, used to determine the available bandwidth resources for the data migration task based on the bandwidth resources occupied by the business IO when the data migration task and the business IO share the same IO port;

[0175] The module for determining the increase in speed corresponding to the bandwidth resources is used to determine the increase in speed corresponding to the bandwidth resources based on the bandwidth resources available for the data migration task.

[0176] Optionally, the device further comprises:

[0177] A first execution module is used for not counting the changes of multiple storage system resources obtained in the first observation period when the change of the storage system resources corresponding to the business IO is greater than a preset change or the storage system resources occupied by the business IO is greater than a first preset threshold in the first observation period of the observation phase;

[0178] The second execution module is used to not count the changes in multiple storage system resources obtained in the second observation period during the observation phase when the load corresponding to the business IO is greater than the second preset threshold and the data volume of the data migration task is less than the third preset threshold.

[0179] Optionally, the data migration task execution module 13 includes:

[0180] The data migration task execution unit is used to calculate the target migration speed of the data migration task based on the smallest scalable speed increment among the multiple scalable speed increments and the current migration speed of the data migration task after the observation phase ends.

[0181] Optionally, the device further comprises:

[0182] The target migration speed adjustment module is used to adjust the target migration speed based on the change of the resource demand of the business IO for multiple storage system resources and the target association relationship during the process of executing the data migration task according to the target migration speed.

[0183] Based on the same inventive concept, another embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the data migration method as described in any of the above embodiments.

[0184] Based on the same inventive concept, another embodiment of the present application further provides a computer program product, including a computer program, which is executed by a processor to perform the data migration method described in any of the above embodiments.

[0185] Based on the same inventive concept, another embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, wherein when the program is executed by a processor, the data migration method as described in any of the above embodiments is implemented.

[0186] As for the device, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0187] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0188] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0189] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0190] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0191] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0192] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0193] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0194] The above is a detailed introduction to a data migration method, device, equipment and medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data migration method, characterized in that: The method comprises: After the data migration task is started, during the observation phase, the migration speed of the data migration task is increased multiple times according to a preset unit speed increment, and a target association relationship is determined by analyzing the change amounts of multiple storage system resources associated with the data migration task, where the target association relationship is used to characterize the association relationship between the change amounts of the multiple storage system resources associated with the data migration task and the change amount of the migration speed of the data migration task; Determine the speed increments that can be increased for each of the multiple storage system resources associated with the data migration task according to the target association relationship and the resource demand of the multiple storage system resources associated with the business IO; Based on the multiple increaseable speed increments, a target migration speed of the data migration task is determined, and the data migration task is executed according to the target migration speed.

2. The data migration method according to claim 1, characterized in that: The target association relationship is determined by the following steps: For each observation period of the observation phase, the migration speed is increased according to the preset unit speed increment, and a change amount of multiple storage system resources associated with the data migration task in the observation period is determined; After the observation phase is finished, statistics are collected on the changes in the plurality of storage system resources corresponding to each observation period to obtain statistical results; Based on the statistical results, the target association relationship is determined.

3. The data migration method according to claim 2, characterized in that: The multiple storage system resources include at least CPU resources, memory resources, and hard disk resources, and determining the target association relationship based on the statistical result includes: Based on the statistical result, for each storage system resource associated with the data migration task, calculate an average change amount of the storage system resource after the migration speed of the data migration task is increased by a preset unit speed increment; Determining the target association relationship based on the corresponding relationship between the preset unit speed increment and the average change amount of the plurality of storage system resources; The determining, according to the target association relationship and the resource demand of the multiple storage system resources associated with the business IO, the speed increments that can be increased corresponding to the multiple storage system resources associated with the data migration task respectively include: Based on the first resource margins of the multiple storage system resources currently available for the data migration task and the target association relationship, calculate the speed increments that can be increased corresponding to each of the multiple storage system resources; The first resource margin is determined based on the total amount of the multiple storage system resources, the resource demand of the business IO for the multiple storage system resources, and the reserved resource amount of the multiple storage system resources.

4. The data migration method according to claim 3, characterized in that: The method further comprises: Determining, based on the type of a hard disk corresponding to the hard disk resource associated with the data migration task, a statistical time interval corresponding to the hard disk; Based on the statistical time interval, within a first statistical period before the start of the observation phase, the number of times the hard disk is in a busy state is counted, and a first hard disk resource occupancy rate of the hard disk resource before the start of the observation phase is calculated; Based on the statistical time interval, within multiple observation cycles after the start of the observation phase, the number of times the hard disk is in a busy state is counted, and multiple second hard disk resource occupancy rates of the hard disk resource in the observation phase are calculated; Calculating an average change amount of the hard disk resource according to the plurality of second hard disk resource occupancy rates and the first hard disk resource occupancy rate; After obtaining the average change of the hard disk resource, the average change of the hard disk resource is updated based on the current change of the hard disk resource in the latest observation period, the average change of the hard disk resource, the preset unit speed increment, and the first weight of the average change of the hard disk resource and the second weight of the current change of the hard disk resource in the latest observation period.

5. The data migration method according to claim 2, characterized in that: In the case where the plurality of storage system resources include bandwidth resources, the method further comprises: When the data migration task and the service IO share the same IO port, determining the bandwidth resources available for the data migration task based on the bandwidth resources occupied by the service IO; Based on the bandwidth resources available for the data migration task, a speed increment corresponding to the bandwidth resources is determined.

6. The data migration method according to claim 2, characterized in that: The method further comprises: In the first observation period of the observation phase, when the change amount of the storage system resources corresponding to the business IO is greater than the preset change amount or the storage system resources occupied by the business IO is greater than the first preset threshold, the change amounts of the multiple storage system resources obtained in the first observation period are not counted; In the second observation period of the observation phase, when the load corresponding to the business IO is greater than the second preset threshold and the data volume of the data migration task is less than the third preset threshold, the changes in the multiple storage system resources obtained in the second observation period are not counted.

7. The data migration method according to claim 2, characterized in that: The step of determining a target migration speed of the data migration task based on the plurality of speed-increase increments includes: The target migration speed of the data migration task is calculated based on the smallest scalable speed increment among the multiple scalable speed increments and the current migration speed of the data migration task after the observation phase ends.

8. The data migration method according to any one of claims 1 to 7, characterized in that: The method further comprises: In the process of executing the data migration task according to the target migration speed, the target migration speed is adjusted based on the change of the resource demand of the service IO on the multiple storage system resources and the target association relationship.

9. A data migration device, characterized in that: The device comprises: An observation module is used for, after the data migration task is started, increasing the migration speed of the data migration task multiple times according to a preset unit speed increment during an observation phase, and determining a target association relationship by analyzing the change amounts of multiple storage system resources associated with the data migration task, wherein the target association relationship is used to characterize the association relationship between the change amounts of the multiple storage system resources associated with the data migration task and the change amount of the migration speed of the data migration task; A module for determining the increase in speed that can be increased is used to determine the increase in speed that can be increased corresponding to each of the multiple storage system resources associated with the data migration task according to the target association relationship and the resource demand of the multiple storage system resources associated with the business IO; The data migration task execution module is used to determine a target migration speed of the data migration task based on the multiple speed increments that can be increased, and execute the data migration task according to the target migration speed.

10. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the data migration method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the data migration method according to any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Migration speed control method and device for distributed object storage system and medium

    CN111966301A

  • Data migration method and device and medium

    CN113032369A

  • Cloud platform storage migration speed control method, device and system

    CN115967724A

  • Data migration method and device, electronic equipment and storage medium

    CN118051501A

  • Method of data migration

    US20010011324A1