A Method, Device and System for Controlling the Storage Migration Speed of a Cloud Platform

Abstract

The present invention relates to a method, device and system for controlling the storage migration speed of a cloud platform. The method includes: obtaining the migration parameters of the current migration task, where the migration parameters include the amount of migrated data, the migration progress and the priority parameter, and the priority parameter is set according to the importance and urgency of the current migration task; obtaining the task migration situation of the node where the current migration task is located, where the task migration situation includes the number of tasks being migrated, the total amount of migrated data, the total migration progress and the total sum of the priority parameters; calculating the bandwidth allocation threshold of the current migration task based on the migration parameters of the current migration task and the task migration situation of the node where the current migration task is located, and adjusting the migration speed of the corresponding migration task based on the bandwidth allocation threshold. The storage migration speed and the overall migration concurrency number of the cloud platform can be dynamically adjusted according to the operating state of the cloud platform and the actual needs of customers.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing storage, and in particular to a cloud platform storage migration speed control method, device and system. Background Art

[0002] With the popularization and large-scale application of cloud computing, the amount of data generated by cloud services is becoming increasingly large, the storage scale used by the corresponding cloud platform is also increasing, and the business scenarios are becoming more and more complex. A cloud platform often provides a variety of different types of storage resources to meet different business needs, and must also have the ability to migrate data between different types of storage resources. At the same time, in order to prevent storage migration from occupying host bandwidth and affecting other business systems on the cloud platform, it is necessary to limit the speed and concurrency of storage migration.

[0003] In the related art, the speed limit of storage migration is adopted, and usually a fixed migration speed threshold is set or the migration speed limit needs to be set manually, and the migration speed limit cannot be modified after the storage migration starts. When the amount of migration data is large and the span is long, it is easy to cause unreasonable speed limit due to the initial estimation deviation of the system load or the peak and valley changes of the business, resulting in excessive bandwidth preemption affecting normal business operation, or the migration speed is too low, resulting in idle network but too long storage migration time, etc. Summary of the invention

[0004] Embodiments of the present invention provide a cloud platform storage migration speed control method, device and system, which can dynamically adjust the storage migration speed and the concurrent number of overall migrations of the cloud platform according to the operating status of the cloud platform and actual customer needs.

[0005] On the one hand, an embodiment of the present invention provides a cloud platform storage migration speed control method, characterized in that it includes the steps of:

[0006] Obtaining migration parameters of the current migration task, wherein the migration parameters include the migration data volume DiskSize, the migration progress MigProg, and the priority parameter MigPri, and the priority parameter is set according to the importance and urgency of the current migration task;

[0007] Obtain the task migration status of the node where the current migration task is located, wherein the task migration status includes the number of migrated tasks TolNum, the total amount of migrated data TolSize, the total migration progress TolProg, and the total priority parameter TolPri;

[0008] The bandwidth allocation threshold MigSpd of the current migration task is calculated based on the migration parameter of the current migration task and the task migration status of the node where the current migration task is located, and the migration speed of the corresponding migration task is adjusted based on the bandwidth allocation threshold MigSpd.

[0009] In some embodiments, the task migration situation of the node where the current migration task is located further includes:

[0010] The average network speed BizSpd of the service network, the network card bandwidth BizBand of the service network, the network card bandwidth StorBand of the storage network, and the reserved bandwidth ratio StorReserve of the storage network of the node where the current migration task is located.

[0011] In some embodiments, calculating the bandwidth allocation threshold MigSpd of the current migration task based on the migration parameters of the current migration task and the task migration situation of the node where the current migration task is located includes the steps of:

[0012] Calculating the bandwidth allocation threshold MigSpd of the current migration task according to the first formula, and the first formula includes:

[0013]

[0014] Among them, the operation result of floor(a) is to round down the numerical value a, the operation result of max(a, b) is to take the larger number of a and b, TaskMinSpd is the minimum protection speed of the migration task, and OffsetRatio is the preset migration bandwidth offset rate, and the value range is [0, 1].

[0015] In some embodiments, the preset value of the migration bandwidth offset rate is 1 / 3.

[0016] In some embodiments, it further includes the steps of:

[0017] Obtaining the total migration situation parameters of the migration tasks completed in the previous cycle, where the total migration situation parameters include the number of migration tasks TaskNum, the total storage migration amount Data, and the average migration speed TaskAvgSpd corresponding to each migration task;

[0018] Calculating the cloud platform concurrent migration threshold TaskMaxNum based on the total migration situation parameters and the maximum migration protection speed of the storage cluster, and adjusting the number of cloud platform concurrent migration tasks in the next cycle based on the cloud platform concurrent migration threshold TaskMaxNum.

[0019] In some embodiments, calculating the cloud platform concurrent migration threshold TaskMaxNum based on the total migration situation parameters and the maximum migration protection speed of the storage cluster includes the steps of:

[0020] Calculating the cloud platform concurrent migration threshold based on the second formula, and the second formula includes:

[0021]

[0022] Among them, the result of ceil(a) operation is to round a up, the result of min(a, b) operation is to take the smaller number of a and b, the result of avg(a) operation is to calculate the arithmetic mean of variable a, ClusterMaxSpd is the maximum migration protection speed of the storage cluster, and Th is the speed limit period.

[0023] In some embodiments, the average migration speed TaskAvgSpd corresponding to each migration task is calculated according to the third formula, and the third formula is:

[0024]

[0025] Among them, Te is the task end time, and Ts is the task start time.

[0026] In a second aspect, an embodiment of the present invention further provides a cloud platform storage migration speed control device, which is characterized in that the device includes: at least one processor; and a memory coupled to the at least one processor, the memory containing instructions stored therein, and the instructions are loaded and executed by the processor to implement the above-mentioned cloud platform storage migration speed control method.

[0027] In a third aspect, an embodiment of the present invention further provides a cloud platform storage migration speed control system, which is characterized in that it includes:

[0028] A parameter acquisition module, which is used for:

[0029] Acquire the migration parameters of the current migration task, the migration parameters include the amount of migrated data, the migration progress, and the priority parameter, and the priority parameter is set according to the importance and urgency of the current migration task;

[0030] Acquire the task migration situation of the node where the current migration task is located, the task migration situation includes the number of tasks in migration, the total amount of migrated data, the total migration progress, and the total sum of priority parameters;

[0031] A speed limit module, which is used to calculate the bandwidth allocation threshold of the current migration task based on the migration parameters of the current migration task and the task migration situation of the node where the current migration task is located, and the bandwidth allocation threshold is used to adjust the migration speed of the corresponding migration task.

[0032] In some embodiments, the parameter acquisition module is further used to acquire the total migration situation parameters of the migration tasks completed in the previous period, and the total migration situation parameters include the number of migration tasks, the total storage migration volume, and the average migration speed corresponding to each migration task;

[0033] The speed limit module is further configured to calculate a cloud platform concurrent migration threshold according to the total migration situation parameter and the maximum migration protection speed of the storage cluster, and the cloud platform concurrent migration threshold is used to adjust the number of cloud platform concurrent migration tasks in the next cycle.

[0034] An embodiment of the present invention provides a cloud platform storage migration speed control method, device, and system, which are used to dynamically and intelligently control and adjust the migration speed of each migration task and the concurrent threshold of the overall migration task according to the environmental operating state when migrating task data across storage clusters in the powered-on state of virtual machines in the cloud platform. In the embodiment of the present invention, the data volume, migration progress, and priority parameters of the migration task are associated with the algorithm, which is convenient for adjusting the migration speed by setting the weight ratio of the three to affect the calculation result (the bandwidth allocation threshold of the current migration task). On the one hand, it can avoid the long migration of small-scale tasks (small data volume) in the large-concurrency scenario from affecting the user experience. At the same time, it can also accelerate the migration speed of highly completed services to reduce the time-consuming of multi-task preemption and reduce the system load; and it can achieve that the higher the task priority, the faster the migration speed, and the speed of long-time concurrent migration tasks can be flexibly regulated according to business requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0036] Figure 1 It is a schematic flowchart of a cloud platform storage migration speed control method provided by an embodiment of the present invention;

[0037] Figure 2 It is a schematic flowchart of a cloud platform storage migration speed control method provided by an embodiment of the present invention

[0038] Figure 3 It is a schematic structural diagram of a cloud platform storage migration speed control system provided by an embodiment of the present invention;

[0039] Figure 4 It is a schematic structural diagram of a cloud platform storage migration speed control system provided by an embodiment of the present invention;

[0040] Figure 5 It is a schematic flowchart of a method for calculating the bandwidth allocation threshold of the current migration task based on the cloud platform storage migration speed control system provided by an embodiment of the present invention;

[0041] Figure 6Schematic flow chart of calculating the concurrent migration threshold of a cloud platform in the storage migration speed control system provided by an embodiment of the present invention. Detailed implementation manners

[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0043] As Figure 1 shown, an embodiment of the present invention provides a cloud platform storage migration speed control method, including the steps of:

[0044] S100: Obtain the migration parameters of the current migration task. The migration parameters include the migration data volume DiskSize, the migration progress MigProg, and the priority parameter MigPri, and the priority parameter is set according to the importance and urgency of the current migration task;

[0045] S200: Obtain the task migration situation of the node where the current migration task is located. The task migration situation includes the number of tasks being migrated TolNum, the total migration data volume TolSize, the total migration progress TolProg, and the total priority parameter TolPri;

[0046] S300: Calculate the bandwidth allocation threshold MigSpd of the current migration task based on the migration parameters of the current migration task and the task migration situation of the node where the current migration task is located, and adjust the migration speed of the corresponding migration task based on the bandwidth allocation threshold MigSpd.

[0047] It should be noted that the migration parameters include the migration data volume DiskSize (unit: GB), the migration progress MigProg (percentage), the priority parameter MigPri (the value range can be selected from 1 to 5, default is 5), the number of tasks being migrated TolNum (positive integer), the total migration data volume TolSize (unit: GB), the total migration progress TolProg (percentage), and the total priority parameter TolPri (positive integer).

[0048] It should be noted that the priority parameter in S100 can be specified by the user when creating a migration task after evaluating the importance level and / or urgency level of the business. If the system has a total of 5 levels, the priority parameter can be selected from 1 to 5. For example, the highest importance level is specified as level 5 and the lowest as level 1. The priority parameter can also be selected according to other user-defined level reference criteria. Therefore, during the migration process, the migration speed of each task can be adjusted by adjusting the task priority at any time. The priority parameter provides the function of artificially intervening to adjust the migration speed.

[0049] It can be understood that S300 is used to periodically calculate the bandwidth upper limit available for each ongoing migration task. In the embodiment of the present invention, the data volume, migration progress, and priority parameter of the migration task are associated with the algorithm, which is convenient for adjusting the migration speed by setting the weight ratio of the three to affect the calculation result (the bandwidth allocation threshold of the current migration task). On the one hand, it can avoid the long migration of small-scale tasks (small data volume) in a large-concurrency scenario from affecting the user experience. At the same time, it can also accelerate the migration speed of services with a high degree of completion, so as to reduce the time-consuming of multi-task preemption and reduce the system load; and it realizes that the higher the task priority, the faster the migration speed, and the long-time concurrent migration task speed can be flexibly adjusted according to business requirements.

[0050] Preferably, the task migration situation of the node where the current migration task is located in S200 further includes: the average network speed of the business network, the network card bandwidth of the business network, the network card bandwidth of the storage network, and the reserved bandwidth ratio of the storage network of the node where the current migration task is located.

[0051] It should be noted that the storage network is for the traffic of virtual machines reading and writing hard disks, and other traffic goes through the business network. The storage network and the business network are isolated from each other and correspond to independent network devices on the hardware. The network card bandwidth can be obtained from the monitoring module, and the reserved bandwidth ratio of the storage network is obtained from the configuration module.

[0052] It can be understood that the network card traffic and bandwidth can be obtained from cloud platform monitoring components (such as ceilometer, zabbix, Prometheus, etc.). The reserved bandwidth ratio of the storage network is filled in by the administrator according to the actual situation when the cloud platform is built, which represents the minimum storage bandwidth percentage reserved for platform services. This part of the reserved bandwidth will not be occupied by migration tasks under any circumstances, and the administrator can also modify it at any time, and these parameters can also be obtained from the cloud platform in real time.

[0053] Furthermore, in S300, the bandwidth allocation threshold of the current migration task can be calculated according to the first formula, and the first formula includes:

[0054]

[0055] Among them, MigSpd is the bandwidth allocation threshold of the current migration task, the result of floor(a) operation is to round down the value a, the result of max(a, b) operation is to take the larger number of a and b, BizSpd is the average network speed of the business network of the node, BizBand is the network card bandwidth of the business network of the node, StorReserve is the reserved bandwidth ratio of the storage network of the node, StorBand is the network card bandwidth of the storage network of the node, DiskSize is the amount of migrated data of the current migration task (i.e., the disk size migrated by the current migration task), MigProg is the migration progress of the current migration task, MigPri is the priority parameter of the current migration task, TolNum is the number of tasks being migrated on the node, TolSize is the total amount of migrated data of the migration tasks on the node, TolProg is the total migration progress of the migration tasks on the node, TolPri is the total sum of the priority parameters of the migration tasks on the node, TaskMinSpd is the minimum protection speed of the migration task, OffsetRatio is the preset migration bandwidth offset rate, and the value range is [0, 1].

[0056] It should be noted that the value of the migration bandwidth offset rate OffsetRatio can range from 0 to 1, including decimals between 0 and 1. When this value is 0, the speeds of all migration tasks are the same, and the remaining bandwidth of the storage network is evenly divided. The larger this value is, the greater the influence of the calculation result of the bandwidth allocation threshold by the three-party weights (the amount of data, migration progress, and priority parameter of the migration task), and the greater the difference in the bandwidth allocated among the migration tasks. That is, the migration task with a high weight has a faster migration speed, and the one with a low weight is slower, and the adjustment effect of the algorithm is more obvious. Preferably, OffsetRatio is set to 1 / 3. The parameter OffsetRatio can be modified by the administrator at any time, and the result of the first formula is extracted and calculated according to the actual value extracted in each cycle according to the speed limit period.

[0057] It can be understood that in the embodiments of the present invention, the remaining bandwidth is used for storage migration, which can always ensure that the business traffic is not affected; at the same time, by real-time monitoring the network load (including the average network speed of the business network and the network card bandwidth), human estimation errors can be avoided, the network bandwidth can be maximally utilized, and the migration speed of each migration task can be accurately controlled, automatically adjusted according to the business peak and valley, and the overall migration concurrency threshold of the cloud platform can be intelligently adjusted while taking into account the business busyness and the performance of the storage cluster. In the case of concurrent migration, based on the business priority principle (based on the priority parameter of the current migration task and the total sum of the priority parameters of all migration tasks on the node), the traffic bandwidth occupied by each migration task is intelligently allocated, and the migration progress of each task is reasonably regulated.

[0058] As Figure 2 shown, in some embodiments, the cloud platform storage migration speed control method provided by the embodiments of the present invention further includes:

[0059] S400: Obtain the total migration parameters of the migration tasks completed in the previous cycle. The total migration parameters include the number of migration tasks, the total storage migration volume, and the average migration speed corresponding to each migration task.

[0060] S500: Calculate the cloud platform concurrent migration threshold based on the total migration parameters and the maximum migration protection speed of the storage cluster, and adjust the number of cloud platform concurrent migration tasks in the next cycle based on the cloud platform concurrent migration threshold.

[0061] It should be noted that the storage cluster refers to a complete storage product connected to the cloud platform, which can be a distributed storage cluster, or a single or clustered SAN storage device. The maximum migration protection speed of the storage cluster can be obtained by taking the smaller value of the maximum read / write performance index of the storage product and the network bandwidth of the cloud platform connecting to the storage, and then reserving a certain amount of service bandwidth. It is used to reflect the maximum bandwidth that this storage product can provide for storage migration. It can be understood that in this embodiment, the total data volume migrated by the platform in the previous period and the maximum migration protection speed of the storage cluster are used to affect the migration concurrency threshold in the next period, thereby further adjusting the number of cloud platform concurrent migration tasks in the next period.

[0062] Preferably, S500 calculates the cloud platform concurrent migration threshold based on the second formula, and the second formula includes:

[0063]

[0064] Among them, TaskMaxNum is the cloud platform concurrent migration threshold, the result of the ceil(a) operation is to round a up, the result of the min(a, b) operation is to take the smaller value of a and b, the result of the avg(a) operation is to take the arithmetic mean of the variable a, TaskAvgSpd is the average migration speed corresponding to each migration task, Data is the total storage migration volume, TaskNum is the number of migration tasks, ClusterMaxSpd is the maximum migration protection speed of the storage cluster, Th is the speed limit period (in seconds), and if the speed limit period is set to 1 hour, then Th is 3600.

[0065] Preferably, the average migration speed corresponding to each migration task is calculated according to the third formula, and the third formula is:

[0066]

[0067] Among them, DiskSize is the migration data volume of the current migration task, Te is the task end time, and Ts is the task start time.

[0068] Second aspect, as Figure 3As shown in the figure, an embodiment of the present invention further provides a cloud platform storage migration speed control system, which includes:

[0069] A parameter acquisition module, which is used for:

[0070] Acquire the migration parameters of the current migration task, where the migration parameters include the amount of migrated data, the migration progress, and the priority parameter, and the priority parameter is set according to the importance and urgency of the current migration task;

[0071] Acquire the task migration situation of the node where the current migration task is located, where the task migration situation includes the number of tasks in migration, the total amount of migrated data, the total migration progress, and the sum of the priority parameters;

[0072] A speed limit module, which is used to calculate the bandwidth allocation threshold of the current migration task based on the migration parameters of the current migration task and the task migration situation of the node where the current migration task is located, and the bandwidth allocation threshold is used to adjust the migration speed of the corresponding migration task.

[0073] It can be understood that in the embodiment of the present invention, the amount of data, the migration progress, and the priority parameter of the migration task are associated with the algorithm, which is convenient for adjusting the migration speed by setting the weight ratio of the three to affect the calculation result (the bandwidth allocation threshold of the current migration task). On the one hand, it can avoid the long migration of small-scale tasks (small amount of data) in a large concurrency scenario, which affects the user experience. At the same time, it can also accelerate the migration speed of services with high completion degrees, reduce the time-consuming of multi-task preemption, and reduce the system load; and it can achieve that the higher the task priority, the faster the migration speed, and the long-time concurrent migration task speed can be flexibly adjusted according to business requirements.

[0074] In some embodiments, the task migration situation of the node where the current migration task is located further includes: the average network speed of the business network of the node where the current migration task is located, the bandwidth of the business network network card, the bandwidth of the storage network network card, and the reserved bandwidth ratio of the storage network;

[0075] The speed limit module is also used to calculate the bandwidth allocation threshold of the current migration task according to the first formula. Preferably, OffsetRatio in the first formula is set to 1 / 3.

[0076] It can be understood that in the embodiments of the present invention, the remaining bandwidth is used for storage migration, which can always ensure that the service traffic is not affected. At the same time, by real-time monitoring of the network load (including the average network speed of the service network and the network card bandwidth of the network), human estimation errors can be avoided, the network bandwidth can be maximally utilized, and the migration speed of each migration task can be accurately controlled, automatically adjusted according to the business peak and valley, so as to intelligently adjust the overall migration concurrency threshold of the cloud platform while taking into account the business busyness degree and the performance of the storage cluster. In the case of concurrent migration, based on the service priority principle (based on the current migration task priority parameter and the total priority parameter sum of all migration tasks of the node), the traffic bandwidth occupied by each migration task is intelligently allocated, and the migration progress of each task is reasonably regulated.

[0077] In some embodiments, the parameter acquisition module is further configured to acquire the total migration situation parameters of the migration tasks completed in the previous period, and the total migration situation parameters include the number of migration tasks, the total storage migration volume, and the average migration speed corresponding to each migration task.

[0078] The speed limit module is further configured to calculate the cloud platform concurrent migration threshold according to the total migration situation parameters and the maximum migration protection speed of the storage cluster, and the cloud platform concurrent migration threshold is used to adjust the number of cloud platform concurrent migration tasks in the next period.

[0079] Preferably, the speed limit module is further configured to calculate the cloud platform concurrent migration threshold according to the second formula

[0080] As Figure 4 shown, in a specific embodiment, the cloud platform storage migration speed control system further includes a configuration module, a monitoring module, and a migration module; the parameter acquisition module can acquire corresponding parameters from the configuration module, the monitoring module, and the migration module.

[0081] Among them, the configuration module is used to configure the relevant parameters for global speed limit for the entire cloud platform, and provide corresponding parameter modification interfaces for other modules to call. The relevant parameters set and provided for query and modification by the configuration module include: the proportion of reserved bandwidth for the node storage network StorReserve (the value is a percentage, such as 5%), the minimum protection speed of the migration task TaskMinSpd (unit: Mb / s), the maximum migration protection speed of the storage cluster ClusterMaxSpd (unit: GB / s), and the cloud platform concurrent migration threshold TaskMaxNum.

[0082] It should be noted that default values are provided during the construction of the cloud platform, and users can modify them at any time according to the actual environment after construction, which can be understood as being initialized by the user. The minimum protection speed of the migration task is the minimum migration speed limit set by the administrator to ensure that the migration task is not interrupted, and can be configured after estimating according to the performance of the cloud platform storage device and the total network bandwidth.

[0083] During operation, the configuration module updates parameters according to the real-time calculation of the concurrent migration threshold TaskMaxNum of the cloud platform by the speed limit module.

[0084] The monitoring module is used to monitor the loads of each network plane of each node in the cloud platform and provide corresponding query interfaces for other components to call. The parameters collected and provided by the monitoring module include: the bandwidth of the business network network card BizBand (unit: Mb / s) of the specified node, the average network speed BizSpd of the business network of the specified node (which can be the average network speed in a previous time interval, unit: Mb / s), and the bandwidth of the storage network network card StorBand (unit: Mb / s) of the specified node.

[0085] The migration module is used to start and manage storage migration tasks and provide corresponding query interfaces for other components to call; the parameters managed and provided by the migration module include: the amount of migrated data DiskSize (unit: GB) of the current migration task, the migration progress MigProg of the current migration task (i.e., the completed migration progress, expressed as a percentage), the priority parameter MigPri of the current migration task (optional from 1 to 5, default 5), the virtual machine where the disk is located, the node where the migration task is located, the start time of the migration task, and the completion time of the migration task. It can be understood that after the speed limit module calculates the migration bandwidth (i.e., the bandwidth allocation threshold) of each migration task and the overall storage concurrent migration threshold of the cloud platform, it takes effect in real time and is migrated through the migration module.

[0086] It can be understood that the real-time effectiveness of the migration bandwidth calculated by the speed limit module means sending the calculated migration bandwidth to the migration module, and the migration module performs task migration according to the real-time calculated migration bandwidth. In addition, after the speed limit module calculates the cloud platform concurrent threshold in real time, it only needs to send it to the configuration module for parameter update and does not need to send it to the speed limit module. When the user issues a new migration task, the migration module obtains the latest cloud platform concurrent migration threshold from the configuration module to determine whether to allow the user to issue a new migration task.

[0087] Such as Figure 5 shown, in a specific embodiment, when performing migration speed control based on the cloud platform storage migration speed control system, it includes the steps:

[0088] S11. The parameter acquisition module accesses the migration module according to the speed limit period to obtain the list of ongoing migration tasks;

[0089] S12. The parameter acquisition module analyzes the ongoing migration task list. Taking nodes as the dimension, it statistically obtains the parameters of each node with migration tasks, including: the total number of migration tasks TolNum (a positive integer) that are being migrated on this node, the total disk size TolSize (GB) of the migration tasks being migrated on this node, the total migration progress TolProg (expressed as a percentage) of all the migration tasks being migrated on this node, and the sum of the total priority parameters TolPri (expressed as a positive integer) of all the migration tasks being migrated on this node;

[0090] S13. When the parameter acquisition module traverses the ongoing migration task list, it obtains parameters from the current migration task details, including the disk size DiskSize of the migration (i.e., the amount of migration data of the current migration task), the completed migration progress MigProg (i.e., the migration progress of the current migration task), the migration task priority parameter MigPri, and the node where the current migration task is located;

[0091] S14. The parameter acquisition module calls the configuration module interface to obtain parameters, including: the reserved bandwidth ratio StorReserve of the node storage network, and the minimum protection speed TaskMinSpd of the migration task;

[0092] S15. According to the node where the current migration task is located, the parameter acquisition module calls the monitoring module interface to obtain parameters, including the business network network card bandwidth BizBand of this node, the average network speed BizSpd in the previous 1 minute of the business network of this node, and the storage network network card bandwidth StorBand of this node;

[0093] S16. According to the node where the current migration task is located, the parameter acquisition module obtains the parameters of the node where the current migration task is located from the result of step S12, including: the total number of migration tasks TolNum that are being migrated on this node, the total disk size TolSize of the migration tasks being migrated on this node, and the total migration progress TolProg of all the migration tasks being migrated on this node;

[0094] S17. The speed limit module calculates the bandwidth (the bandwidth allocation threshold of the current migration task) that can be allocated to the current migration task using the first formula based on the parameters obtained by the parameter acquisition module. The OffsetRatio is set to 1 / 3. Therefore, the first formula is:

[0095]

[0096] In the above formula, the result of the floor(a) operation is to round down the value a, and the result of the max(a, b) operation is to take the larger number of a and b. It can be understood that the OffsetRatio can be managed by the configuration module for the administrator to modify at any time, and then obtained by the speed limit module every cycle and substituted into the first formula; it can also be managed by the configuration file of the speed limit module and loaded into the first formula when the speed limit module is started.

[0097] S18. The speed limit module calls the virtualization layer interface to set the migration bandwidth to take effect in real time, that is, the speed limit module sends the calculated migration bandwidth to the migration module, and the migration module performs task migration according to the real-time calculated migration bandwidth.

[0098] It should be noted that before step S11, the speed limit module starts and initializes the calculation task according to the speed limit period set in its configuration file. This calculation task is to calculate the bandwidth allocation threshold for the migration task. It can be understood that the speed limit module runs periodically, and steps S11 to S18 are executed in each cycle to calculate and adjust the migration speed of all ongoing migration tasks. S11 obtains the list; S12 obtains the overall statistical parameters from the list, and then traverses the list, and loops to execute S13 to S18 for each migration task in the list.

[0099] As Figure 6 shown, in a specific embodiment, when performing migration speed control based on the cloud platform storage migration speed control system, the speed limit module can also calculate the cloud platform storage concurrent migration threshold, which specifically includes the steps:

[0100] S21. When the speed limit module starts, according to the speed limit period set in its configuration file, it initializes the calculation task, and this calculation task is to calculate the cloud platform concurrent migration task threshold;

[0101] S22. When the cycle time arrives, the parameter acquisition module accesses the migration module to obtain the list of migration tasks completed in the previous cycle. If the list is empty, skip this cycle and return to S22, otherwise enter S23;

[0102] S23. The parameter acquisition module calls the configuration module interface to obtain the maximum migration protection speed ClusterMaxSpd of the storage cluster;

[0103] S24. The parameter acquisition module counts the parameters from the list of migration tasks completed in the previous cycle, including: the total storage migration amount Data completed in the previous cycle, the number of migration tasks TaskNum completed in the previous cycle, the migration data volume DiskSize of each migration task in the previous cycle, the task end time Te and task start time Ts of each migration task

[0104] S25. The speed limit module calculates the concurrent migration threshold of the cloud platform using the second formula based on the parameters provided by the parameter acquisition module. The second formula is:

[0105]

[0106] where the result of the ceil(a) operation is to round a up, the result of the min(a, b) operation is to take the smaller number of a and b, the result of the avg(a) operation is to take the arithmetic mean of the variable a, and Th is the speed limit period; TaskAvgSpd is the average migration speed of each migration task completed in the previous period, which can be calculated using the third formula. The third formula is:

[0107]

[0108] S26. The speed limit module calls the configuration module interface to set the concurrent migration threshold of the cloud platform to take effect in real time. It can be understood that after the speed limit module calculates the cloud platform concurrent threshold in real time, it sends it to the configuration module for parameter update. The migration module will obtain the latest cloud platform concurrent migration threshold from the configuration module and determine whether to allow the user to issue a new migration task.

[0109] It should be noted that since the calculation tasks started by S21 to S26 are different from the calculation tasks performed by S11 to S18, the speed limit periods involved in S21 and the speed limit periods involved in S11 are not the same period value. In practical applications, the speed limit period of S11 is used to calculate and adjust the migration task bandwidth, with the minute as the period unit. The speed limit period of S21 is used to calculate and adjust the concurrent migration task threshold of the cloud platform, with the hour as the period unit.

[0110] The beneficial effects of the cloud platform storage migration speed control method and system provided by the embodiments of the present invention include: using the remaining bandwidth for storage migration to always ensure that business traffic is not affected; real-time monitoring of network load to avoid human estimation errors and maximize the use of network bandwidth; accurately controlling the migration speed of each migration task and automatically adjusting with the business peak and valley; taking into account the business busy degree and the performance of the storage cluster, and intelligently adjusting the overall migration concurrent threshold of the cloud platform; in the case of concurrent migration, following the principle of business priority, intelligently allocating the traffic bandwidth occupied by each migration task, and reasonably regulating the migration progress of each task; associating the data size of the migration task with the migration speed limit algorithm to achieve that the smaller the total data volume of the task, the faster the migration, and avoiding the long migration time of small-scale tasks in large-concurrency scenarios from affecting the user experience; incorporating the migration progress indicator into the migration speed limit algorithm, the greater the migration progress, the faster the migration speed, accelerating the migration of services with high completion degrees to reduce the time-consuming of multi-task preemption and reduce the system load; incorporating the task priority that can be specified by the user at any time into the migration algorithm, the higher the task priority, the faster the migration speed, and realizing that the speed of long-time concurrent migration tasks can be flexibly regulated according to business requirements.

[0111] Based on the same inventive concept, an embodiment of the present invention further provides a device, which includes: at least one processor; and a memory coupled to the at least one processor, the memory containing instructions stored therein, the instructions being loaded and executed by the processor to implement the method according to any one of the method embodiments.

[0112] Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, and their appropriate combinations. In the hardware implementation, the division of the functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or one function or step may be executed by several physical components in cooperation. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable storage medium, which may include a computer-readable storage medium (or non-transitory medium) and a communication medium (or transitory medium).

[0113] It should be noted that in the present invention, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0114] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for controlling the storage migration speed of a cloud platform, characterized in that, It includes the steps: Obtain the migration parameters of the current migration task, where the migration parameters include the amount of migrated data DiskSize, the migration progress MigProg, and the priority parameter MigPri, and the priority parameter is set according to the importance and urgency of the current migration task; Obtain the task migration status of the node where the current migration task is located. The task migration status of the node where the current migration task is located includes the number of tasks being migrated TolNum, the total amount of migrated data TolSize, the total migration progress TolProg, the sum of priority parameters TolPri, the average network speed of the service network BizSpd, the network card bandwidth of the service network BizBand, the network card bandwidth of the storage network StorBand, and the reserved bandwidth ratio of the storage network StorReserve; Calculate the bandwidth allocation threshold MigSpd of the current migration task based on the migration parameters of the current migration task and the task migration status of the node where the current migration task is located, and adjust the migration speed of the corresponding migration task based on the bandwidth allocation threshold MigSpd; Calculating the bandwidth allocation threshold MigSpd of the current migration task based on the migration parameters of the current migration task and the task migration status of the node where the current migration task is located includes the steps: Calculate the bandwidth allocation threshold MigSpd of the current migration task according to the first formula, and the first formula includes: Where the result of the floor(a) operation is to round down the value a, the result of the max(a,b) operation is to take the larger of a and b, TaskMinSpd is the minimum protection speed of the migration task, and the OffsetRatio is the preset migration bandwidth offset rate, with a value range of [0,1].

2. The cloud platform storage migration speed control method according to claim 1, wherein The preset value of the migration bandwidth offset rate is 1 / 3.

3. The cloud platform storage migration speed control method according to claim 1, wherein It includes the steps: Obtain the total parameters of the migration status of the migration tasks completed in the previous cycle. The total parameters of the migration status include the number of migration tasks TaskNum, the total storage migration amount Data, and the average migration speed TaskAvgSpd corresponding to each migration task; Calculate the cloud platform concurrent migration threshold TaskMaxNum based on the total parameters of the migration status and the maximum migration protection speed of the storage cluster, and adjust the number of cloud platform concurrent migration tasks in the next cycle based on the cloud platform concurrent migration threshold TaskMaxNum.

4. The cloud platform storage migration speed control method according to claim 3, wherein Calculating the cloud platform concurrent migration threshold TaskMaxNum based on the total parameters of the migration status and the maximum migration protection speed of the storage cluster includes the steps: Calculate the cloud platform concurrent migration threshold based on the second formula, and the second formula includes: Where the result of the ceil(a) operation is to round up a, the result of the min(a,b) operation is to take the smaller of a and b, the result of the avg(a) operation is to take the arithmetic mean of the variable a, ClusterMaxSpd is the maximum migration protection speed of the storage cluster, and Th is the speed limit period.

5. The cloud platform storage migration speed control method according to claim 4, wherein The average migration speed TaskAvgSpd corresponding to each migration task is calculated according to the third formula, and the third formula is: Among them, Te is the task end time, and Ts is the task start time.

6. A cloud platform storage migration speed control device, characterized in that, The device includes: at least one processor; and a memory coupled to the at least one processor, the memory containing instructions stored therein, the instructions being loaded and executed by the processor to implement the method according to any one of claims 1-5.

7. A cloud platform storage migration speed control system, characterized in that It includes: A parameter acquisition module, which is used for: Acquiring the migration parameters of the current migration task, the migration parameters including the amount of migrated data, the migration progress, and the priority parameter, and the priority parameter being set according to the importance and urgency of the current migration task; Acquiring the task migration situation of the node where the current migration task is located, the task migration situation of the node where the current migration task is located including the number of tasks in migration, the total amount of migrated data, the total migration progress, the total sum of priority parameters, the average network speed BizSpd of the service network, the network card bandwidth BizBand of the service network, the network card bandwidth StorBand of the storage network, and the storage network reserved bandwidth ratio StorReserve of the node where the current migration task is located; A speed limiting module, which is used for calculating the bandwidth allocation threshold of the current migration task based on the migration parameters of the current migration task and the task migration situation of the node where the current migration task is located, and the bandwidth allocation threshold being used to adjust the migration speed of the corresponding migration task; The speed limiting module is used for: Calculating the bandwidth allocation threshold MigSpd of the current migration task according to the first formula, the first formula including: Among them, the operation result of floor(a) is to round down the value a, the operation result of max(a, b) is to take the larger number of a and b, TaskMinSpd is the minimum protection speed of the migration task, and OffsetRatio is a preset migration bandwidth offset rate, and the value range is [0, 1].

8. The cloud platform storage migration speed control system according to claim 7, characterized in that, The parameter acquisition module is further used for acquiring the total parameters of the migration situation of the migration tasks completed in the previous cycle, the total parameters of the migration situation including the number of migration tasks, the total storage migration amount, and the average migration speed corresponding to each migration task; The speed limiting module is further used for calculating the cloud platform concurrent migration threshold according to the total parameters of the migration situation and the maximum migration protection speed of the storage cluster, and the cloud platform concurrent migration threshold being used to adjust the number of cloud platform concurrent migration tasks in the next cycle.

Images