Data migration method, device, computer equipment and readable storage medium

Abstract

The embodiments of the present application provide a data migration method, apparatus, computer equipment and readable storage medium. The method includes: determining at least one source storage volume that needs to be migrated, and obtaining a negative load impact factor; migrating the data in the source storage volume, and determining the first utilization of the source storage volume during the migration process; determining the utilization threshold of the source storage volume, and determining the first adjustment granularity of the read and write rate of the target block device receiving the source storage volume; sending a read and write rate adjustment instruction for the corresponding target storage volume to the target block device, so that the target block device gradually adjusts the read and write rate of the target storage volume smoothly according to the first adjustment granularity according to the read and write rate adjustment instruction, so as to gradually adjust the read and write rate of the source storage volume smoothly until the first utilization of the source storage volume is adjusted to the utilization threshold; after the data migration is completed, the data migration operation is stopped. In this way, the flexibility of data migration can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data migration, and in particular to a data migration method, apparatus, computer equipment and readable storage medium. Background Art

[0002] Data migration refers to the process of transferring data from one system or storage medium to another. In scenarios such as technology upgrades or replacements, changes in business requirements, and data integration, data migration is required to ensure data integrity and business continuity.

[0003] In the related art, when migrating data, usually after selecting the storage volume to be migrated, the storage volume is directly migrated. However, when the available load of the system is insufficient, this will increase the burden on the system when the system is overloaded, or cause a waste of resources when the available load of the system is large. In other words, in the process of data migration, the flexibility of migration is low. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a data migration method, apparatus, computer equipment and readable storage medium, which can improve the flexibility of data migration.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a data migration method, which is applied to a source block device and includes:

[0006] In response to the data migration instruction, determining at least one source storage volume that needs to be migrated, and obtaining a preset negative load impact factor of the source storage volume;

[0007] Migrating data in the source storage volume, and determining a first utilization rate of the source storage volume during the migration process;

[0008] Determining a utilization threshold of the source storage volume based on the negative load impact factor corresponding to the source storage volume, and determining a first adjustment granularity of a read / write rate of a target block device receiving the source storage volume according to a gap between the first utilization and the utilization threshold;

[0009] Sending a read / write rate adjustment instruction for the corresponding target storage volume to the target block device according to the first adjustment granularity, so that the target block device gradually and smoothly adjusts the read / write rate of the target storage volume according to the first adjustment granularity according to the read / write rate adjustment instruction, so as to gradually and smoothly adjust the read / write rate of the source storage volume until the first utilization of the source storage volume is adjusted to the utilization threshold;

[0010] When the data in the source storage volume is migrated, the data migration operation is stopped.

[0011] Accordingly, a second aspect of an embodiment of the present application provides a data migration device, the device comprising:

[0012] A response module, configured to determine, in response to a data migration instruction, at least one source storage volume to be migrated, and obtain a preset negative load impact factor of the source storage volume;

[0013] a migration module, which migrates the data in the source storage volume and determines a first utilization rate of the source storage volume during the migration process;

[0014] a determination module, configured to determine a utilization threshold of the source storage volume based on the negative load impact factor corresponding to the source storage volume, and determine a first adjustment granularity of a read / write rate of a target block device receiving the source storage volume according to a gap between the first utilization and the utilization threshold;

[0015] An adjustment module is used to send a read / write rate adjustment instruction for the corresponding target storage volume to the target block device according to the first adjustment granularity, so that the target block device gradually and smoothly adjusts the read / write rate of the target storage volume according to the first adjustment granularity according to the read / write rate adjustment instruction, so as to gradually and smoothly adjust the read / write rate of the source storage volume until the first utilization of the source storage volume is adjusted to the utilization threshold.

[0016] The stop module is used to stop the data migration operation after the data in the source storage volume is migrated.

[0017] In some implementations, the data migration device further includes a solution module, which is used to:

[0018] Obtaining the initial utilization of the source storage volume in a time step before data migration and the upper limit of the storage space utilization of the source storage volume, and establishing an objective function of the negative load impact factor based on the storage space utilization, the upper limit of the storage space utilization and the factor to be solved;

[0019] Obtaining a constraint factor corresponding to at least one resource dimension and a constraint threshold corresponding to the constraint factor, and determining at least one first constraint function based on the initial utilization, the factor to be solved, the constraint factor and the constraint threshold; wherein the constraint factor corresponding to the resource dimension changes based on a change in the load negative impact factor;

[0020] Constructing a model to be solved according to the objective function and the at least one first constraint function;

[0021] The factor to be solved is solved according to the model to be solved to obtain a negative impact factor of the load of the source storage volume during the process of the source block device migrating the data of the source storage volume.

[0022] In some implementations, the determining module is further configured to:

[0023] Obtaining an absolute value of a difference between the first utilization rate and the utilization rate threshold;

[0024] Comparing the absolute value of the difference with the adjustment threshold to obtain a comparison result;

[0025] When the comparison result indicates that the absolute value of the difference is greater than the adjustment threshold, a first adjustment granularity of the read / write rate of the target block device corresponding to the migration of the source block device is determined from a preset adjustment granularity table according to the absolute value of the difference.

[0026] In some implementations, the data migration device further includes a correction module, which is used to:

[0027] Obtain the original shadow copy created at the last time step when the migration of data in the source storage volume is started;

[0028] In the process of migrating the data of the source storage volume, when the data of the source storage volume is updated, an updated shadow copy corresponding to the time step from the last time the shadow copy was created to the current time step is created;

[0029] After completing the data migration of the source storage volume, generating target reference data based on the original shadow copy and at least one updated shadow copy;

[0030] Comparing the target reference data with the data in the target storage volume to obtain a comparison result;

[0031] When the comparison result indicates that the data in the target storage volume is inconsistent with the data in the target reference data, the data in the target storage volume is modified based on the target reference data.

[0032] In some implementations, the data migration device further includes an updating module, which is used to:

[0033] Regularly obtain the available load of the source storage volume and the processing progress of each source storage volume;

[0034] When the processing progress at the current moment is less than the processing progress threshold preset at the current moment, the negative load impact factor of the corresponding source storage volume is updated based on the available load.

[0035] In some implementations, the data migration device further includes a marking module, which is used to:

[0036] Obtaining the location of the source storage volume to be migrated in the source block device;

[0037] For the source storage volume, obtaining multiple load negative impact factors corresponding to multiple time periods;

[0038] In the source block device, the location of the source storage volume and the multiple load negative impact factors corresponding to the multiple time periods are marked.

[0039] Accordingly, a third aspect of an embodiment of the present application proposes a data migration method, which is applied to a target block device. The method includes:

[0040] In response to a read / write rate adjustment instruction for a corresponding target storage volume sent by a source block device according to a first adjustment granularity, wherein the first adjustment granularity is determined by the source block device based on a load negative impact factor corresponding to the source storage volume, and is obtained according to a difference between the first utilization and the utilization threshold; the first utilization is determined based on the process of migrating data in the source storage volume by the source storage volume; the load negative impact factor is obtained after the source block device determines at least one source storage volume that needs to be migrated in response to a data migration instruction;

[0041] According to the read / write rate adjustment instruction, gradually and smoothly adjust the read / write rate of the target storage volume according to the first adjustment granularity, so as to gradually and smoothly adjust the read / write rate of the source storage volume until the first utilization rate of the source storage volume is adjusted to the utilization rate threshold;

[0042] When the data in the source storage volume is migrated, the operation of receiving the migrated data is stopped.

[0043] Correspondingly, the fourth aspect of the embodiments of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the data migration method described in any one of the first aspect embodiments or the second aspect embodiments of the present application.

[0044] Correspondingly, the fifth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the data migration method described in any one of the first aspect embodiments or the second aspect embodiments of the present application.

[0045] The embodiment of the present application determines at least one source storage volume that needs to be migrated in response to a data migration instruction, and obtains a preset negative load impact factor of the source storage volume; migrates data in the source storage volume, and determines a first utilization of the source storage volume during the migration process; determines a utilization threshold of the source storage volume based on the negative load impact factor corresponding to the source storage volume, and determines a first adjustment granularity of the read and write rate of a target block device that receives the source storage volume according to the difference between the first utilization and the utilization threshold; sends a read and write rate adjustment instruction for the corresponding target storage volume to the target block device according to the first adjustment granularity, so that the target block device gradually smoothly adjusts the read and write rate of the target storage volume according to the read and write rate adjustment instruction, so as to gradually smoothly adjust the read and write rate of the source storage volume until the first utilization of the source storage volume is adjusted to the utilization threshold; when the migration of the data in the source storage volume is completed, the data migration operation is stopped. In this way, the negative load impact factor can be set as a policy parameter for different time periods, the current load of the system can be fully considered, the speed of data migration can be dynamically adjusted, and the flexibility of data migration can be improved. At the same time, the application can more smoothly affect the read and write rates of the source storage volume by adjusting the read and write rates of the target block device, thereby avoiding the problem of unstable or interrupted data migration caused by sudden changes in the read and write rates, and improving the flexibility of data migration. In summary, the application can improve the flexibility of data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the architecture of the data migration system provided in an embodiment of the present application;

[0047] Figure 2 is a flow chart of a data migration method provided in an embodiment of the present application;

[0048] Figure 3 is an overall flow chart of the data migration method provided in the embodiment of the present application;

[0049] Figure 4 is a schematic diagram of functional modules of a data migration device provided in an embodiment of the present application;

[0050] Figure 5 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0054] Data migration refers to the process of transferring data from one system or storage medium to another. In scenarios such as technology upgrades or replacements, changes in business requirements, and data integration, data migration is required to ensure data integrity and business continuity.

[0055] In the related art, when migrating data, usually after selecting the storage volume to be migrated, the storage volume is directly migrated. However, when the available load of the system is insufficient, this will increase the burden on the system when the system is overloaded, or cause a waste of resources when the available load of the system is large. In other words, in the process of data migration, the flexibility of migration is low.

[0056] Based on this, the embodiments of the present application provide a data migration method, apparatus, computer device and readable storage medium, which can improve the flexibility of data migration.

[0057] The data migration method, apparatus, computer device, and readable storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the data migration system in the embodiments of the present application is described.

[0058] Please refer to Figure 1 In some implementations, an embodiment of the present application provides a data migration system, including a terminal 11 and a server 12 .

[0059] Exemplarily, the terminal 11 may be any device having a user interface and capable of initiating a data migration request. The terminal 11 may be a personal computer, a mobile device, etc. The operator may initiate a data migration request through a graphical user interface or a command line interface on the terminal 11, and display the progress and load of the migration in real time, while allowing the user to adjust the migration parameters or stop or restart the migration task.

[0060] Furthermore, the server end 12 may be a server in a data center, a virtual or physical server of a cloud service provider, a storage array or a storage area network device, etc. The server end 12 may perform a specific block device data migration task according to the request and configuration information sent by the terminal 11. During the migration process, the server end 12 may continuously monitor the load of the block device and dynamically adjust the migration speed according to the negative load impact factor. At the same time, the server end 12 may record the load and progress information during the migration process so that the terminal can monitor and perform subsequent operations.

[0061] In some implementations, the terminal 11 can communicate with the server 12 through a management system (such as a Web interface or CLI) to manage the migration task, including marking the block device objects to be migrated, obtaining migration object information, setting migration parameters (such as load negative impact factors), etc. The server 12 is responsible for the actual data transmission work and dynamically adjusts the migration speed during the execution process to adapt to load changes.

[0062] The data migration method in the embodiments of the present application can be illustrated by the following embodiments.

[0063] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first. Moreover, the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or separate consent, the necessary user-related data for enabling the normal operation of the embodiment of the present application will be obtained.

[0064] In the embodiment of the present application, the data migration device will be described from the perspective of the data migration device, which can be integrated into a computer device. Figure 2 , Figure 2 This is a flowchart of the steps of the data migration method provided in an embodiment of the present application. The data migration method is applied to a source block device. In the embodiment of the present application, a data migration device is specifically integrated in a terminal or a server as an example. When a processor on the terminal or the server executes a program instruction corresponding to the data migration method, the specific process is as follows:

[0065] Step 101 : In response to a data migration instruction, at least one source storage volume that needs to be migrated is determined, and a preset negative load impact factor of the source storage volume is obtained.

[0066] In some implementations, in order to dynamically migrate data, the negative load impact factors pre-set for different time periods may be obtained, and the corresponding negative load impact factors may be obtained in the current time period to automatically adjust the migration speed according to the business load conditions.

[0067] The source block device can be the original storage device that needs to be migrated. The source block device can be a hard disk, a logical volume, or any other form of storage unit, and the source block device contains the data to be migrated. In the data migration process, the source block device is the starting point of the data, that is, the data is read from here and migrated to the target block device.

[0068] The data migration instruction can be a command issued by the user through the management system (such as an application program interface or a command line interface accessed by a browser), instructing the system to start the data migration process. The data migration instruction contains basic information about the migration, such as the source block device, the target block device, and related migration parameters. The data migration instruction triggers the start of the entire data migration process, allowing the system to perform subsequent data migration actions according to the instruction.

[0069] The source storage volume can be a storage unit that serves as a data source during the data migration process, usually a logical volume (LV) in a logical volume management (LVM) system. The source storage volume represents the storage space where the data to be migrated is located. The source storage volume can be divided from one or more physical hard disks, on which the data to be migrated is stored.

[0070] The negative load impact factor can be a value pre-set by the user in the management system, or a value obtained by predicting the load changes of the source block device in different time periods through the target model. The negative load impact factor is used to adjust the load of the source storage volume (which can be understood as the logical storage space on the source block device) and is used to set the measurement of the increase in the load of the source storage volume during the data migration process.

[0071] In some implementations, a user can directly mark the source storage volume that needs to be migrated, as well as the negative load impact factor configured for the source storage volume in different time periods, through a management system (such as an application program interface or a command line interface accessed by a browser). When data needs to be migrated from the source storage volume to the target storage volume of the target block device, the management system or script program can directly read the configuration file generated after marking, and extract the required migration information without the need for command line queries, thereby improving the usability of the system.

[0072] Furthermore, the management system will generate one or more configuration files based on the user's tags, each configuration file contains information about a single or multiple migration tasks. When a specific migration task needs to be performed, the management system or script program can read the corresponding configuration file based on the user's input or specific conditions and extract the relevant information of the source block device that needs to be migrated.

[0073] In some embodiments, the source block device and the target block device can be located on the same physical device or on different physical devices. For example, in the case of data migration on the same physical device, that is, the situation where data needs to be relocated or the layout of storage space needs to be adjusted within the same server, internal reorganization or fault repair is required through data migration. At this time, the target block device and the source block device belong to the same physical device; while for data migration between data centers, cloud migration or storage upgrade, the target block device and the source block device belong to different physical devices respectively, and the two are transmitted through a network connection. Regardless of whether cross-server data migration is involved, the technical solution of the present application can be used to achieve flexible adjustment of the migration speed.

[0074] Through the above method, the source storage volume to be migrated and the corresponding negative load impact factor can be obtained, so as to facilitate the subsequent flexible migration of the source storage volume.

[0075] In some implementations, in order to ensure that the storage space utilization limit is not exceeded during the data migration process while ensuring the stable operation of the business system, a mathematical model can be established to solve the negative load impact factor to ensure that the utilization of the source storage volume during the data migration process does not exceed the preset safety threshold and meets the requirements of the quality of service or other resource dimensions. For example, the "negative load impact factor" in step 101 can be determined by the following method:

[0076] (A.1) Obtain the initial utilization of the source storage volume in the previous time step of data migration and the upper limit of the storage space utilization of the source storage volume, and establish the objective function of the negative load impact factor based on the initial utilization, the upper limit of the storage space utilization and the factor to be solved;

[0077] (A.2) Obtaining a constraint factor corresponding to at least one resource dimension and a constraint threshold corresponding to the constraint factor, and determining at least one first constraint function based on the initial utilization, the factor to be solved, the constraint factor and the constraint threshold; wherein the constraint factor corresponding to the resource dimension changes based on a change in a negative load impact factor;

[0078] (A.3) constructing a model to be solved according to the objective function and at least one first constraint function;

[0079] (A.4) Solving the factors to be solved according to the model to be solved, and obtaining the negative impact factor of the load of the source storage volume when the source block device migrates the data of the source storage volume.

[0080] The initial utilization rate may be the utilization rate of the source storage volume at a time step before the start of data migration. For example, if data is migrated at 12:15, the utilization rate of the corresponding source storage volume may be obtained at 12:01 as the initial utilization rate.

[0081] Among them, the upper limit of storage space utilization can be the maximum storage utilization allowed by the source block device. Exceeding the upper limit of storage space utilization may have a negative impact on the business system. Therefore, the sum of the initial utilization and the load negative impact factor is set to be less than or equal to the upper limit of storage space utilization.

[0082] The factor to be solved may be a negative load impact factor to be solved, which represents a maximum utilization increment allowed during the data migration process.

[0083] Among them, the objective function can be a function composed of initial utilization, load negative impact factor and storage space utilization upper limit. The objective function also includes the expectation of the solved load negative impact factor, such as making the load negative impact factor as large as possible.

[0084] The resource dimension may be various indicators related to service quality, such as response time, throughput, IOPS, etc.

[0085] The constraint factor may correspond to a resource dimension and may be a specific indicator representing the service quality, such as a response time not exceeding 50 milliseconds and a throughput not less than 100 MB per second.

[0086] The constraint threshold may be an upper limit or a lower limit of the constraint factor, which is used to ensure the quality of service.

[0087] The first constraint function may be a constraint function for ensuring that even if the load negative impact factor changes during the data migration process, any constraint factor of the service quality will not exceed its corresponding constraint threshold. The form of the first constraint function may be nonlinear, because the change of the load negative impact factor will cause the change of the constraint factor.

[0088] The model to be solved may be a model consisting of an objective function and at least one constraint function.

[0089] For example, we can first obtain the initial utilization and storage space utilization upper limit, assuming that the initial utilization (InitialLoad) of the current source storage volume is 40% and the storage space utilization upper limit (StorageSpaceUtilizationUpperLimit) is 70%. We can establish an objective function to ensure that the sum of the initial utilization and the negative load impact factor x is less than or equal to the storage space utilization upper limit, that is:

[0090] InitialLoad+x≤StorageSpaceUtilizationUpperLimit, that is, 40%+x≤70%.

[0091] Furthermore, it can be assumed that the resource dimensions include response time RT and throughput T. The constraint thresholds are that the response time does not exceed 50 milliseconds, that is, RTmax=50ms, and the throughput is not less than 100MB per second, that is, Tmin=100MB / s.

[0092] Furthermore, the first constraint functions of response time and throughput can be determined respectively. Since the change of negative load influencing factors will lead to the change of constraint factors, the first constraint function is to ensure RT≤RTmax and T≥Tmin. Assuming RT=f(InitialLoad+x), T=g(InitialLoad+x), there are the following two first constraint functions: f(40%+x)≤50ms, and g(40%+x)≥100MB / s.

[0093] Furthermore, the objective function and the first constraint function can be combined to construct a complete mathematical model, that is, the model to be solved. The constructed model to be solved can be as follows:

[0094] ;

[0095] ;

[0096] ;

[0097] By solving the model to be solved, the final negative load influence factor can be calculated.

[0098] It should be noted that in different cases, the initial utilization rate, constraint factor, constraint threshold storage space utilization upper limit will be different, and the form of the constructed model to be solved will also be different, but the construction idea is the same as above, and this application will not list them one by one.

[0099] In some embodiments, the change of the constraint factor caused by the change of the negative load influence factor can be obtained through model simulation, through a specific algorithm, or through empirical judgment, and the present application does not impose any specific restrictions on this.

[0100] By obtaining the initial utilization rate and the upper limit of storage space utilization before migration, and combining the objective function with the nonlinear constraint function to build a mathematical model, an optimal negative load impact factor is solved, so that while meeting the upper limit of storage space utilization, the service quality (such as response time, throughput, etc.) is not destroyed. In this way, not only the efficiency of data migration is improved, but also business interruption or performance degradation caused by migration is avoided, thereby achieving efficient data migration while ensuring stable system operation.

[0101] In some implementations, the negative load impact factor can be set by the user.

[0102] In some implementations, in order to achieve flexible management during data migration, the load data of the source storage volume in the past period of time can be collected to confirm the increase in the negative load impact factor of the source storage volume during the data migration process in different time periods, so as to ensure that the data migration is both efficient and does not adversely affect the current business. For example, the "negative load impact factor" in step 101 can also be determined in the following way:

[0103] (B.1) Obtain multiple historical available loads of the source storage volume in multiple historical time periods;

[0104] (B.2) Based on the multiple historical available loads, predict the available loads of the source storage volume in multiple future time periods to obtain multiple predicted available loads corresponding to the multiple time periods;

[0105] (B.3) Based on the predicted available load corresponding to each time period, determine the corresponding negative load impact factor.

[0106] The historical time period can be a time period in the past used to record and analyze system load conditions. These time periods can be fixed cycles (such as every hour, every day, every week, etc.) or arbitrarily defined time periods, such as daytime, nighttime, etc.

[0107] The historical available load may be the actual available load of the source storage volume in the historical time period. The historical available load may be the hard disk usage rate, which is used to reflect the usage of the device at a certain point in time in the past.

[0108] The predicted available load may be a result obtained by predicting system load conditions in multiple future time periods through a mathematical model or algorithm based on historical available load data.

[0109] In some implementations, the historical available load of the corresponding historical time period can be collected by first collecting key indicators such as the historical CPU usage rate and the historical disk I / O operation times of the source storage volume (such as hard disk / dev / sda1). The historical data is modeled using its learning algorithm (such as the long short-term memory model) to predict the predicted available load for multiple time periods in the future. And according to the predicted available load, the negative load impact factor of the source storage volume is set.

[0110] In some implementations, the waiting time for migration of each source storage volume in the source storage volume can also be determined first, and then the predicted available load of the source storage volume at the waiting time for migration can be predicted based on multiple historical available loads in multiple historical time periods, and the corresponding negative load impact factor can be determined based on the predicted available load.

[0111] In some implementations, the negative load impact factor can be automatically calculated based on user-defined weights. Users can set different weights for different influencing factors, such as central processing unit (CPU) usage, memory usage, and input / output (IO) usage. The system can automatically calculate the negative load impact factor based on these weights. For example, the negative load impact factor can be obtained by the following formula:

[0112] Negative load impact factor = a (1-CPU usage) + b (1-memory usage) + c (1-IO usage), where a is the first weight corresponding to CPU usage, b is the second weight corresponding to memory usage, and c is the third weight corresponding to IO usage. The values ​​of a, b, and c can be set according to actual needs. The sum of a, b, and c is less than or equal to the operating threshold allowed by the system setting, for example, 90%.

[0113] In some implementations, the load negative impact factor can also be dynamically adjusted through quality of service (QoS) indicators. QoS indicators such as response time, throughput, latency, etc. can directly reflect the health status of the system. For example, thresholds for QoS indicators can be set for different services, such as response time not exceeding 100ms, throughput: not less than 500MB / s, etc. The model can predict the QoS indicators of corresponding services in the future based on historical QoS indicators of different services, such as response time, throughput, etc. When the QoS indicator exceeds the preset indicator threshold, a lower load negative impact factor is set; otherwise, a higher load negative impact factor is set. For example, the formula for setting the load negative impact factor based on the predicted QoS indicator is as follows:

[0114] Negative load impact factor = (current response time / response time threshold + current throughput / throughput threshold) * 100%.

[0115] Among them, the response time threshold and the throughput threshold can be set according to actual conditions.

[0116] In some implementations, the negative load impact factor can also be set in combination with the business time period. For example, a smaller negative load impact factor can be set for business peak periods (such as 9:00-18:00 during the day), and a larger negative load impact factor can be set for business off-peak periods (such as 18:00-9:00 at night).

[0117] In some implementations, a negative load impact factor may be set according to the current load at a time step before the start of data migration.

[0118] Through the above method, the corresponding negative load impact factor is determined according to the predicted available load in different time periods in the future, so as to provide a stable reference framework to determine the tolerance for changes in system load during data migration. In this way, it can effectively avoid relying on real-time load to make frequent decisions and introduce unnecessary delays and complexity.

[0119] In some implementations, in order to quickly identify the source storage volume that needs to be processed during the data migration process, the location of the source storage volume and multiple load negative impact factors corresponding to multiple time periods can be marked in advance in the source block device, without the need for frequent command line queries, reducing the probability of frequent manual input errors, and making it easier to use. For example, before "determining at least one source storage volume that needs to be migrated in response to the data migration instruction" in step 101, it also includes:

[0120] (C.1) Obtain the location of the source storage volume that needs to be migrated in the source block device;

[0121] (C.2) For the source storage volume, obtain multiple load negative impact factors corresponding to multiple time periods;

[0122] (C.3) In the source block device, the location of the source storage volume and multiple load negative impact factors corresponding to multiple time periods are marked.

[0123] The location of the source storage volume may include the device name, start and end sectors, mount point, etc. Specifically, the device name indicates the specific block device and the partition corresponding to the source storage volume, such as / dev / sdb1; the start and end sectors are used to define the start and end positions of the source storage volume on the physical disk; the mount point is when the source storage volume is mounted, it displays the corresponding mount path, such as / data, and does not need to be obtained if it is not mounted.

[0124] For example, you can use the command lsblk to list the device name and mount point, and use fdisk-l / dev / sdb or parted / dev / sdbprint to obtain detailed sector information. Afterwards, mark the different load negative impact factors corresponding to multiple time periods at the location of the corresponding source storage volume, so that when performing data migration in the corresponding time period later, directly select the load negative impact factor of the corresponding time period. For example, the first time period of 9:00-18:00 is marked, and the corresponding load negative impact factor is 15%, and the second time period of 18:00-9:00 is marked, and the corresponding load negative impact factor is 30%. Or mark the corresponding load negative impact factor based on each hour as a time period, and specifically mark according to multiple load negative impact factors of multiple time periods calculated in advance. Each source storage volume needs to be marked accordingly.

[0125] In some embodiments, if the user chooses to mark components through an application programming interface (Application Programming Interface, API), the management system will provide an API endpoint for the user to request, in order to mark the storage volumes that need to be migrated and set the load negative impact factor. The API request content can be a user sending JSON data containing source storage volume information and load negative impact factors corresponding to each time period through a Hypertext Transfer Protocol (HTTP) request (usually a POST request), and the management system returns a successful marking result after processing the request. In this way, the subsequent system can obtain relevant information about the marked source storage volume through an API request, and use code to parse the API response data to extract the required information without having to manually query the device each time, thereby reducing the possibility of errors in manual input.

[0126] By marking the source storage volumes, the location of each source storage volume and at least one corresponding negative load impact factor can be obtained more conveniently and intuitively. It is easy to use for users who are not familiar with command lines and can greatly improve the efficiency and accuracy of system management.

[0127] Step 102: migrate data in a source storage volume and determine a first utilization rate of the source storage volume during the migration process.

[0128] In some implementations, since data migration may affect the performance of the source storage volume, if the utilization of the source storage volume is too high, existing business may be affected. Therefore, in order to ensure business continuity and stability, the utilization of the original storage volume may be detected to maximize the smooth progress of data migration.

[0129] The first utilization rate may be a storage space utilization rate of the source storage volume during the data migration process, and is used to indicate the current usage of the source storage volume.

[0130] Specifically, during the process of migrating data in the source storage volume, data may be read from the source storage volume and written to the target storage volume in the target block device to ensure consistency and integrity of the previous and subsequent data.

[0131] For example, in the process of migrating data in the source storage volume, the ratio between the used space and the total available space in the source storage volume can be used as the first utilization rate of the source storage volume. It indicates the current usage of the storage volume, usually expressed as a percentage. For example, if the used space is 600GB and the total available space is 1000GB, then the utilization rate (%) = (used space / total available space) * 100% = 60%.

[0132] In some implementations, the first utilization of the source storage volume may be obtained in real time or periodically, for example, once every 1 minute, once every 2 minutes, once every 5 minutes, etc., depending on actual conditions.

[0133] According to the actual utilization of the source storage volume (first utilization), the system can automatically adjust the speed of data migration. When the utilization of the source storage volume is lower than the preset threshold, the migration speed is increased to improve efficiency. When the utilization is close to or exceeds the threshold, the migration speed is slowed down to reduce the impact on the business. This dynamic adjustment method can effectively balance the relationship between migration efficiency and business performance.

[0134] Step 103: determine a utilization threshold of the source storage volume based on the negative load impact factor corresponding to the source storage volume, and determine a first adjustment granularity of the read and write rate of the target block device receiving the source storage volume according to the gap between the first utilization and the utilization threshold.

[0135] In some implementations, in order to dynamically adjust the speed of data migration according to the current load status of the source storage volume, a utilization threshold can be determined by setting a negative load impact factor, and then the adjustment range of the target block device read and write rate is determined based on the gap between the current first utilization and the utilization threshold, so as to ensure that the data migration process is both efficient and does not have a negative impact on the running business, and effectively balance the relationship between the migration speed and the business load.

[0136] The utilization threshold may be a maximum utilization standard allowed for the source storage volume during the data migration process.

[0137] The first adjustment granularity can be a specific adjustment range selected from the adjustment granularity table according to the comparison result when the absolute value of the difference is greater than the adjustment threshold. The first adjustment granularity determines how the read and write rates of the target block device will be adjusted in the next period of time, that is, how much percentage to increase or decrease.

[0138] Exemplarily, the utilization threshold can be obtained by setting the negative load impact factor based on the initial utilization of the current source storage volume. For example, when the negative load impact factor is set to 30% for the initial utilization at 12:00, the utilization threshold is obtained by adding the initial utilization 40% at 12:00 to the negative load impact factor 30%, and the utilization threshold is 70%. The utilization threshold represents the maximum utilization that the source storage volume can accept without affecting other indicators (such as service quality indicators).

[0139] It is understandable that the absolute value of the difference between the first utilization and the utilization threshold can be calculated first to measure the distance between the current first utilization and the maximum utilization allowed, so as to control the read and write rate of the target block device when receiving data according to the difference between the two, and then adjust the read and write rate of the source block device in the opposite direction. For example, when the utilization threshold is 70% and the first utilization is 40%, the read and write rate of the target block device can be increased until the utilization of the source storage volume reaches the utilization threshold, or until the difference between the utilization of the source storage volume and the utilization threshold is less than a preset utilization range, such as 5%.

[0140] In some embodiments, the first adjustment granularity may be the default adjustment granularity, for example, adjusting 1% of the current read / write rate every 30 seconds. In order to ensure business continuity and stability and avoid business interruption problems caused by sudden and large adjustments, a smaller adjustment granularity may be set to gradually adjust the read / write rate of the target block device to the target storage volume to achieve a smooth transition effect. Alternatively, the adjustment granularity corresponding to the absolute value of the difference between the first utilization rate and the utilization threshold may be selected. For example, the absolute value of the difference between the first utilization rate and the utilization threshold is 20%, and the adjustment granularity corresponding to the pre-adjustment of 20% is 2%. Therefore, 2% may be used as the first adjustment granularity.

[0141] In some embodiments, different gradient adjustment granularities may be set according to the absolute value of the difference between the first utilization and the utilization threshold. For example, the first adjustment granularity may include a first gradient adjustment granularity and a second gradient adjustment granularity. First, the read and write rate of the target block device to the target storage volume is adjusted by the first gradient adjustment granularity (e.g., 5%), and then, the read and write rate of the target block device to the target storage volume is adjusted by the second gradient adjustment granularity (e.g., 1%) until the utilization of the source storage volume reaches the utilization threshold, or until the difference between the utilization of the source storage volume and the utilization threshold is less than a preset utilization range, so as to gradually approach the threshold and reduce the impact on the business.

[0142] By determining the first adjustment granularity of the read and write rate of the target block device to the target storage volume, it is convenient to subsequently adjust the read and write rate of the target storage volume according to the first adjustment granularity, and then adjust the read and write rate of the source storage volume to achieve smooth data migration.

[0143] In some implementations, in order to ensure the stability of the system, the load negative impact factor can be set in the time step before the data migration is performed, so that a utilization threshold is determined before the data migration is performed, and the utilization threshold is used to control the migration speed in the time period corresponding to the migration process. In this way, the management process can be simplified, so that the impact of data migration can be better controlled even in a complex environment. For example, "determining the utilization threshold of the source storage volume based on the load negative impact factor corresponding to the source storage volume" in step 103 may include:

[0144] (103.a1) Obtain the initial utilization of the source storage volume when setting the negative load impact factor for the current time period;

[0145] (103.a2) Determine the utilization threshold of the source storage volume based on the load negative impact factor and the initial utilization.

[0146] The initial utilization rate may be the initial utilization rate of the source storage volume in the previous time step of data migration, that is, when the negative load impact factor of the current time period is set.

[0147] For example, if the data of the source storage volume needs to be migrated at 12:00, a negative load impact factor can be set at the migration time step of 12:00, for example, 12:59. At 12:59, the utilization rate of the source storage volume is 30%, and 30% is the initial utilization rate.

[0148] In some implementations, the utilization threshold of the source storage volume can be set by the sum of the initial utilization and the negative load impact factor. For example, if the initial utilization is 30% and the negative load impact factor is 20%, then the utilization threshold of the source storage volume is 50%.

[0149] In the above manner, a benchmark in the data migration process can be generated by determining the utilization threshold, so as to adjust the speed of data migration according to the benchmark.

[0150] In some implementations, in order to ensure that the data migration task is completed as efficiently as possible without interfering with existing services, the migration speed may be dynamically adjusted according to the gap between the first utilization rate and the utilization rate threshold, so as to avoid the risk of business interruption due to excessive adjustment, and also prevent low migration efficiency due to too small adjustment. For example, in step 103, "determining the first adjustment granularity of the read and write rate of the target block device receiving the source storage volume according to the gap between the first utilization rate and the utilization rate threshold" may include:

[0151] (103.b1) Obtaining an absolute value of a difference between the first utilization rate and the utilization rate threshold;

[0152] (103.b2) Compare the absolute value of the difference with the adjustment threshold to obtain a comparison result;

[0153] (103.b3) When the comparison result indicates that the absolute value of the difference is greater than the adjustment threshold, a first adjustment granularity of the read / write rate of the target block device corresponding to the source block device is determined from a preset adjustment granularity table according to the absolute value of the difference.

[0154] The absolute value of the difference may be the absolute value of the difference between the actual utilization (first utilization) of the current source storage volume and the set utilization threshold. For example, if the current utilization is 40% and the set threshold is 50%, the absolute value of the difference is 10%.

[0155] The comparison result may be a conclusion obtained by comparing the absolute value of the difference with one or more preset adjustment thresholds.

[0156] The adjustment granularity table may be a preset table or a set of rules, which lists different read / write rate adjustment ranges (ie, adjustment granularities) that should be adopted according to different absolute values ​​of the difference.

[0157] In some implementations, if the absolute value of the current difference is large, the system can select a larger adjustment granularity according to the adjustment granularity table to quickly change the read and write rates; if the absolute value of the difference is small, the default smaller adjustment granularity will be selected to gradually adapt to the new load situation to avoid unnecessary impact on the business caused by excessive adjustment, such as business interruption, etc. By adjusting the read and write rates of the target block device corresponding to the source block device for migration, the adjustment process is given greater flexibility, so that the migration speed can be flexibly adjusted according to the load situation monitored in real time. If it is found that the adjustment has a negative impact on the business, a quick response can be made to adjust back to the original speed or further fine-tune it.

[0158] Exemplarily, the adjustment granularity table stores adjustment granularities corresponding to different absolute values ​​of the difference. For example, the absolute value of the difference is 5% to 10%, and the adjustment granularity is 1%; the absolute value of the difference is 10% to 20%, and the adjustment granularity is 2%; the absolute value of the difference is 20% to 30%, and the adjustment granularity is 3%, and so on. The specific adjustment granularity can be set according to actual conditions.

[0159] In some implementations, the read and write rate of the target block device may be adjusted by adjusting the IOPS (input and output operations per second) and MBPS (megabytes per second, i.e., the read and write operation speed) of the target block device to the target storage volume. For example, when it is detected that the first utilization of the source storage volume is lower than the set utilization threshold, the absolute value of the difference between the first utilization and the utilization threshold may be calculated. For example, if the first utilization is 90% and the utilization threshold is 70%, the absolute value of the difference is 20%, and the first utilization of the source storage volume may be adjusted down to 70%.

[0160] For example, if the adjustment threshold is set to 15%, and the comparison result indicates that the absolute value of the difference is 20% greater than the adjustment threshold, then 20% can be used to query the adjustment granularity table to determine that the first adjustment granularity of the read and write rate of the target block device corresponding to the migration of the source block device is 2%, and the read and write rate of the target block device is adjusted according to the preset period, and adjusted by 2% each preset period until the absolute value of the difference between the first utilization rate and the utilization rate threshold is within the allowable range, such as 3%. Or, until the first utilization rate is adjusted to be equal to the utilization rate threshold.

[0161] For example, if the adjustment threshold is set to 15%, and the comparison result indicates that the absolute value of the difference is 10% less than the adjustment threshold, then there is no need to query the adjustment granularity table, and the default adjustment granularity, such as 1%, is used as the adjustment granularity of the read and write rate of the target block device corresponding to the migration of the source block device. The read and write rate of the target block device is adjusted according to the preset period, and the adjustment is made by 1% each preset period until the absolute value of the difference between the first utilization rate and the utilization rate threshold is within the allowable range, such as 3%. Or, until the first utilization rate is adjusted to be equal to the utilization rate threshold.

[0162] It should be noted that the preset period can be set according to actual conditions, such as 10 seconds, 30 seconds, etc., and the embodiments of the present application do not impose specific restrictions on this.

[0163] By using the adjustment granularity table and selecting the appropriate adjustment granularity based on the absolute value of the difference, different load conditions can be flexibly responded to. In addition, by using granularity adjustment instead of immediately and significantly adjusting the migration speed, the negative impact on the business system during the migration process can be minimized, and a smoother and more controllable data migration process can be achieved.

[0164] Step 104: Send a read / write rate adjustment instruction for the corresponding target storage volume to the target block device according to the first adjustment granularity, so that the target block device gradually and smoothly adjusts the read / write rate of the target storage volume according to the first adjustment granularity according to the read / write rate adjustment instruction, so as to gradually and smoothly adjust the read / write rate of the source storage volume until the first utilization of the source storage volume is adjusted to the utilization threshold.

[0165] It is understandable that if the read and write rates of the source storage volume are adjusted directly, it may have a direct impact on the running application. For example, if the read and write rates of the source storage volume are reduced, it may cause the application performance to degrade or the response time to increase, and more seriously, it may cause data inconsistency. Therefore, in some embodiments, by gradually adjusting the read and write rates of the target storage volume, a smoother and more stable data migration process can be achieved.

[0166] The read / write rate adjustment instruction may be a command generated by the system and sent to the target block device, which is used to dynamically adjust the read / write operation rate (IOPS and MBPS) of the target storage volume receiving the migrated data. The read / write rate adjustment instruction includes a first adjustment granularity and other related adjustment information.

[0167] In some implementations, since the target storage volume needs to read data from the source storage volume during the data migration process, an increase in the read and write rate of the target block device to the target storage volume will force the read rate of the source storage volume to increase accordingly.

[0168] For example, if during data migration, the current utilization of the source storage volume is 50%, and the preset utilization threshold is 70%. In order to achieve smooth data migration, the first adjustment granularity can be set to 1%. The system generates a read / write rate adjustment instruction, requiring the target block device to periodically increase the read / write rate of the target storage volume receiving the migrated data by 1% until the first utilization of the source storage volume is adjusted to the utilization threshold. Through this feedback mechanism, the system can dynamically adapt to changing load conditions to ensure that the migration process is both efficient and safe.

[0169] By gradually adjusting the read and write rates of the target storage volume, that is, adjusting the input / output operations per second (IOPS) and megabytes per second (MBPS) of the target storage volume, the read pressure of the source storage volume can be indirectly increased, thereby improving the utilization of the source storage volume. This process is dynamic and smooth, and data migration can be completed efficiently without affecting business operations.

[0170] In some implementations, in order to ensure efficient and stable migration, the available load and processing progress of the source storage volume can be obtained regularly, and the migration strategy can be adjusted according to the processing progress to improve the overall efficiency of the migration. For example, after "until the first utilization of the source storage volume is adjusted to the utilization threshold" in step 104, it can also include:

[0171] (D.1) Regularly obtain the available load of the source storage volume and the processing progress of each source storage volume;

[0172] (D.2) When the processing progress at the current moment is less than the preset processing progress threshold at the current moment, the negative load impact factor of the corresponding source storage volume is updated based on the available load.

[0173] The processing progress may be the ratio or percentage of the completed data migration volume to the total migration volume during the data migration process.

[0174] The processing progress threshold can be a pre-set baseline during the data migration process to determine whether the completion of the migration task meets expectations. Each moment corresponds to a processing progress threshold. If the current moment processing progress is lower than the processing progress threshold set at the corresponding moment, it indicates that the migration speed is slow and measures need to be taken to speed up the migration.

[0175] Exemplarily, the available load of the source storage volume can be obtained regularly through system monitoring tools, cloud platform detection services, etc. For example, the psutil library can be used in Python to obtain the disk I / O usage of the source block device and other indicator usage of the source block device.

[0176] Exemplarily, the processing progress can be obtained regularly by monitoring the log or status file of the migration tool. For example, when using rsync for data migration, the processing progress can be obtained by parsing the log output by rsync.

[0177] In some implementations, in order to ensure that data migration is carried out efficiently and stably, a preset processing progress threshold can be set at each moment, and the migration strategy can be adjusted according to the processing progress threshold. Specifically, the expected processing progress at each moment, that is, the processing progress threshold, can be determined based on the total duration and total data volume of the migration task. The processing progress threshold gradually increases over time to ensure that all migration tasks are completed before the migration deadline. Alternatively, the processing progress threshold at each moment can also be manually set by the operator, and this application does not limit the specific setting method.

[0178] In some embodiments, when the processing progress at the current moment is less than the processing progress threshold preset at the current moment, for example, if the processing progress threshold is set to 70% at 12:15, however, during the actual data migration process, the processing progress corresponding to 12:15 (current moment) is 55%, then the negative load impact factor can be updated, for example, the negative load impact factor can be appropriately increased to speed up the migration progress of the source storage volume.

[0179] Exemplarily, the negative load impact factor can be re-determined based on the available load of the current source storage volume, or the model to be solved can be established again to obtain the utilization threshold, and then the negative load impact factor is calculated. The specific process of determining the negative load impact factor has been introduced above and will not be repeated here.

[0180] The negative load impact factor is updated by the processing progress and available load of the source storage volume. On the basis of the pre-settings, fine-tuning can be made according to the current load situation to better adapt to changes in the actual operating environment, making the data migration process efficient and flexible.

[0181] Step 105: After the data in the source storage volume has been migrated, the data migration operation is stopped.

[0182] In some implementations, after completing the data migration, that is, after ensuring that all planned migrated data has been successfully transferred to the target storage volume, it is necessary to close or stop all ongoing data migration tasks to ensure data consistency. If the data needs to be verified, it is also possible to prepare for subsequent data verification, consistency checks, and possible correction operations.

[0183] In some implementations, after the migration is completed, relevant personnel may be notified of the completion of the migration via email or other means.

[0184] The embodiment of the present application determines at least one source storage volume that needs to be migrated in response to a data migration instruction, and obtains a preset negative load impact factor of the source storage volume; migrates data in the source storage volume, and determines a first utilization of the source storage volume during the migration process; determines a utilization threshold of the source storage volume based on the negative load impact factor corresponding to the source storage volume, and determines a first adjustment granularity of the read and write rate of a target block device that receives the source storage volume according to the difference between the first utilization and the utilization threshold; sends a read and write rate adjustment instruction for the corresponding target storage volume to the target block device according to the first adjustment granularity, so that the target block device gradually smoothly adjusts the read and write rate of the target storage volume according to the read and write rate adjustment instruction, so as to gradually smoothly adjust the read and write rate of the source storage volume until the first utilization of the source storage volume is adjusted to the utilization threshold; when the migration of the data in the source storage volume is completed, the data migration operation is stopped. In this way, the negative load impact factor can be set as a policy parameter for different time periods, the current load of the system can be fully considered, the speed of data migration can be dynamically adjusted, and the flexibility of data migration can be improved. At the same time, the application can more smoothly affect the read and write rates of the source storage volume by adjusting the read and write rates of the target block device, thereby avoiding the problem of unstable or interrupted data migration caused by sudden changes in the read and write rates, and improving the flexibility of data migration. In summary, the application can improve the flexibility of data migration.

[0185] In some implementations, when migrating data in a source storage volume, the source storage volume may generate new data, which may cause the data in the target storage volume to be inconsistent with the complete data of the source storage volume. In order to ensure consistency and integrity during the data migration process, a shadow copy may be generated at each update during the data migration, and the shadow copies may be aggregated after the data migration is completed to generate complete target reference data, thereby minimizing the risk of data inconsistency and ensuring data integrity and business continuity. For example, the data migration method may also include:

[0186] (E.1) obtaining the original shadow copy created at the last time step when the migration of data in the source storage volume was started;

[0187] (E.2) During the process of migrating the data of the source storage volume, when the data of the source storage volume is updated, an updated shadow copy corresponding to the time step from the last time the shadow copy was created to the current time step is created;

[0188] (E.3) after completing the data migration of the source storage volume, generating target reference data based on the original shadow copy and at least one updated shadow copy;

[0189] (E.4) Comparing the target reference data with the data in the target storage volume to obtain a comparison result;

[0190] (E.5) When the comparison result indicates that the data in the target storage volume and the target reference data are inconsistent, the data in the target storage volume is corrected based on the target reference data.

[0191] The original shadow copy may be a complete data backup of the source storage volume before the data migration begins.

[0192] The last time step for creating a shadow copy may be the time point when the shadow copy was last created.

[0193] The current time step may be a time point when the current system processes or creates the latest shadow copy, that is, a time point when the source storage volume receives new update data.

[0194] Among them, updating the shadow copy can be used to record all data updates from the last time step of creating the shadow copy to the current time step.

[0195] The target reference data may be data generated by integrating the original shadow copy and all updated shadow copies, and represents the final data state of the source storage volume.

[0196] The comparison result may be a result obtained by comparing the target reference data with the data of the target storage volume. When the comparison result is consistent, it indicates that there is no difference in the data between the source storage volume and the target storage volume; when the comparison result is inconsistent, it indicates that there is a difference in the data between the source storage volume and the target storage volume, and the data of the target storage volume needs to be corrected to ensure the accuracy and completeness of the migration process.

[0197] In some implementations, a shadow copy may be created at each update to ensure that all updates are fully recorded, thereby ensuring that no changes are lost during the migration process, thereby ensuring the continuity and integrity of the data in the target storage volume.

[0198] In some embodiments, when the data in the source storage volume needs to be migrated to the target storage volume, the original storage volume may continuously receive new data updates. In order to ensure the consistency of the data after the migration, a shadow copy can be created to record each update. Exemplarily, before the migration starts, an initial shadow copy can be created by full backup to obtain the state of the entire source storage volume before the migration starts, for example, the creation time is 12:15. Assuming that the source storage volume receives the first data update 5 minutes after the start of the data migration, then an updated shadow copy can be created immediately after the update occurs to record the changes that occurred from the time step (12:15) when the original shadow copy was created to the current time step (12:20). Specifically, all changes that occurred in the corresponding update time period can be obtained through logging or change data capture technology.

[0199] For example, if the source storage volume receives a second data update 15 minutes after the start of data migration, an updated copy can be created immediately after the update occurs, recording the changes that occurred from the time step (12:20) when the last shadow copy was created to the current time step (12:30). By triggering the creation of a new shadow copy each time the data in the source storage volume is updated, it can be used to generate target reference data after the migration is completed, and perform consistency checks to ensure the integrity and consistency of the migrated data.

[0200] In some embodiments, after the data migration of the source storage volume is completed, when the target reference data is generated based on the original shadow copy and at least one updated shadow copy, the originally created original shadow copy can be loaded, and the updated shadow copy created during the data migration process can be applied at one time to ensure that the target reference data finally generated is complete. For example, the directory structure of the final target reference data can be created using Path(final_copy).mkdir(parents=True, exist_ok=True), the initial shadow copy can be copied to the final reference data directory using shutil.copytree, and the files and directories in each updated shadow copy can be traversed using os.walk, and the corresponding directory structure in the target reference data can be created using Path(destination_root).mkdir(parents=True, exist_ok=True), and if the target file already exists and the content is different, the old file can be replaced using shutil.copy2; if the target file does not exist, a new file can be created.

[0201] In some embodiments, after the data migration is completed, when comparing the target reference data with the data in the target storage volume, the comparison can be performed by calculating hash values, file-by-file comparison, metadata comparison, etc. This application does not limit the specific comparison method, but only needs to ensure that the comparison result is accurate.

[0202] Taking the comparison by hash value as an example, the first hash value corresponding to the target reference data and the second hash value corresponding to the data in the target storage volume can be calculated first for quick comparison to obtain the comparison result. In another example, the comparison can be performed by metadata of the two, for example, obtaining the file size of the target reference data, that is, the first storage occupancy; and obtaining the file size of the data in the target storage volume, that is, the second storage occupancy, and comparing the first storage occupancy with the second storage occupancy to obtain the comparison result. Furthermore, the comparison can also be performed by other metadata, and the embodiment of the present application does not limit the form of the specific metadata for comparison.

[0203] In some embodiments, when the comparison result indicates that the data in the target storage volume and the target reference data are inconsistent, the difference between the target storage volume and the target reference data can be quickly synchronized through a file-level synchronization tool (rsync or other third-party tools). For example, the rsync -avz --delete --progress / path / to / referenced / data / path / to / target / storage command can be used to correct the files in the target reference data to the target storage volume.

[0204] In some implementations, the copying and replacement of the target storage volume and the target reference data may be handled by writing an automated script, and the present application does not limit the specific modification method.

[0205] By creating and managing shadow copies, the accuracy of the migration process and the consistency of data can be ensured while the source storage volume is constantly updated. By creating the original shadow copy before migration and creating an updated shadow copy when updating, all changes can be continuously tracked and integrated to generate reliable target reference data. After the migration is completed, the consistency and integrity of the target storage volume with the source storage volume can be ensured by comparing the data with the target storage volume and correcting the differences.

[0206] In some implementations, the present application also provides a data migration method, which is applied to a target block device, and the method includes:

[0207] In response to a read / write rate adjustment instruction for a corresponding target storage volume sent by a source block device according to a first adjustment granularity, wherein the first adjustment granularity is determined by the source block device based on a negative load impact factor corresponding to the source storage volume, and is obtained according to a difference between the first utilization and the utilization threshold; the first utilization is determined in a process of migrating data in the source storage volume based on the source storage volume; the negative load impact factor is obtained after the source block device determines at least one source storage volume that needs to be migrated in response to the data migration instruction;

[0208] According to the read / write rate adjustment instruction, the read / write rate of the target storage volume is gradually and smoothly adjusted according to the first adjustment granularity, so as to gradually and smoothly adjust the read / write rate of the source storage volume until the first utilization rate of the source storage volume is adjusted to the utilization rate threshold;

[0209] When the data in the source storage volume is migrated, stop receiving the migrated data.

[0210] The embodiment of the present application responds to a read / write rate adjustment instruction for a corresponding target storage volume sent by a source block device according to a first adjustment granularity, wherein the first adjustment granularity is determined by the source block device based on the load negative impact factor corresponding to the source storage volume, and is obtained according to the difference between the first utilization and the utilization threshold; the first utilization is determined in the process of migrating data in the source storage volume based on the source storage volume; the load negative impact factor is obtained after the source block device responds to the data migration instruction and determines at least one source storage volume that needs to be migrated; according to the read / write rate adjustment instruction, the read / write rate of the target storage volume is gradually and smoothly adjusted according to the first adjustment granularity, so as to gradually and smoothly adjust the read / write rate of the source storage volume until the first utilization of the source storage volume is adjusted to the utilization threshold; when the data in the source storage volume is migrated, the operation of receiving the migrated data is stopped. In this way, the negative load impact factor can be set as a policy parameter for different time periods, the current load of the system can be fully considered, the speed of data migration can be dynamically adjusted, and the flexibility of data migration can be improved. At the same time, the application can more smoothly affect the read and write rates of the source storage volume by adjusting the read and write rates of the target block device, thereby avoiding the problem of unstable or interrupted data migration caused by sudden changes in the read and write rates, and improving the flexibility of data migration. In summary, the application can improve the flexibility of data migration.

[0211] The specific implementation of the data migration method applied to the target block device is substantially the same as the specific embodiment of the data migration method applied to the source block device described above, and will not be described in detail herein.

[0212] Please refer to Figure 3 In some embodiments, in combination Figure 3The overall embodiment of the present application is introduced. First, the location of the source storage volume to be migrated and the corresponding negative load impact factor can be marked in advance, and the utilization threshold can be calculated. For example, if the initial utilization (current utilization) is 40% and the negative load impact factor is 30%, then the utilization threshold can be calculated to be 70%.

[0213] Furthermore, during the data migration process, the utilization of the source storage volume can be detected at regular intervals, and the read and write rates of the target storage volume can be adjusted based on the difference between the utilization of the source storage volume and the utilization threshold, thereby increasing the read and write pressure of the source storage volume in reverse and adjusting the source storage volume. For example, when the utilization of the source storage volume is less than the utilization threshold of 70%, the pressure of the target storage volume is increased at a certain time interval, with a granularity of 1% each time, thereby smoothly adjusting the read and write rates of the source storage volume. When the utilization of the source storage volume is greater than the utilization threshold of 70%, the pressure of the target storage volume is reduced at a certain time interval, with a smaller granularity of 2% each time, thereby smoothly adjusting the read and write rates of the source storage volume.

[0214] See also Figure 4 The embodiment of the present application further provides a data migration device, which can implement the above data migration method. The data migration device includes:

[0215] A response module 41 is used to determine at least one source storage volume that needs to be migrated in response to the data migration instruction, and obtain a preset negative load impact factor of the source storage volume;

[0216] A migration module 42 migrates data in a source storage volume and determines a first utilization rate of the source storage volume during the migration process;

[0217] A determination module 43 is used to determine a utilization threshold of the source storage volume based on the negative load impact factor corresponding to the source storage volume, and determine a first adjustment granularity of the read and write rate of the target block device receiving the source storage volume according to the gap between the first utilization and the utilization threshold;

[0218] The adjustment module 44 is configured to send a read / write rate adjustment instruction for the corresponding target storage volume to the target block device according to the first adjustment granularity, so that the target block device gradually and smoothly adjusts the read / write rate of the target storage volume according to the first adjustment granularity according to the read / write rate adjustment instruction, so as to gradually and smoothly adjust the read / write rate of the source storage volume until the first utilization rate of the source storage volume is adjusted to the utilization rate threshold;

[0219] The stop module 45 is used to stop the data migration operation after the data in the source storage volume is migrated.

[0220] The specific implementation of the data migration device is basically the same as the specific implementation of the data migration method described above, and will not be repeated here. On the premise of meeting the requirements of the embodiment of the present application, the data migration device can also be provided with other functional modules to implement the data migration method in the above embodiment.

[0221] The embodiment of the present application also provides a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data migration method when executing the computer program. The computer device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0222] See also Figure 5 , Figure 5 The hardware structure of a computer device according to another embodiment is shown, and the computer device includes:

[0223] The processor 51 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0224] The memory 52 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 52 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 52, and the processor 51 calls and executes the data migration method of the embodiment of this application;

[0225] Input / output interface 53, used to implement information input and output;

[0226] Communication interface 54, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0227] A bus 55 that transmits information between the various components of the device (e.g., the processor 51, the memory 52, the input / output interface 53, and the communication interface 54);

[0228] The processor 51 , the memory 52 , the input / output interface 53 and the communication interface 54 are connected to each other in communication within the device via a bus 55 .

[0229] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data migration method is implemented.

[0230] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0231] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0232] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0233] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0234] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0235] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0236] It should be understood that in the present application, "at least one (item)" and "several" refer to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0237] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0238] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0239] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0240] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0241] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data migration method, characterized in that: Applied to a source block device, the method comprises: In response to the data migration instruction, determining at least one source storage volume that needs to be migrated, and obtaining a preset negative load impact factor of the source storage volume; Migrating data in the source storage volume, and determining a first utilization rate of the source storage volume during the migration process; Determining a utilization threshold of the source storage volume based on the negative load impact factor corresponding to the source storage volume, and determining a first adjustment granularity of a read / write rate of a target block device receiving the source storage volume according to a gap between the first utilization and the utilization threshold; Sending a read / write rate adjustment instruction for the corresponding target storage volume to the target block device according to the first adjustment granularity, so that the target block device gradually and smoothly adjusts the read / write rate of the target storage volume according to the first adjustment granularity according to the read / write rate adjustment instruction, so as to gradually and smoothly adjust the read / write rate of the source storage volume until the first utilization of the source storage volume is adjusted to the utilization threshold; When the data in the source storage volume is migrated, the data migration operation is stopped.

2. The data migration method according to claim 1, characterized in that: The negative load impact factor is determined by the following method: Obtaining the initial utilization of the source storage volume in a time step before data migration and the upper limit of the storage space utilization of the source storage volume, and establishing an objective function of the negative load impact factor based on the storage space utilization, the upper limit of the storage space utilization and the factor to be solved; Obtaining a constraint factor corresponding to at least one resource dimension and a constraint threshold corresponding to the constraint factor, and determining at least one first constraint function based on the initial utilization, the factor to be solved, the constraint factor and the constraint threshold; wherein the constraint factor corresponding to the resource dimension changes based on a change in the load negative impact factor; Constructing a model to be solved according to the objective function and the at least one first constraint function; The factor to be solved is solved according to the model to be solved to obtain a negative impact factor of the load of the source storage volume during the process of the source block device migrating the data of the source storage volume.

3. The data migration method according to claim 1, characterized in that: The step of determining a first adjustment granularity of a read / write rate of a target block device receiving the source storage volume according to a gap between the first utilization and the utilization threshold comprises: Obtaining an absolute value of a difference between the first utilization rate and the utilization rate threshold; Comparing the absolute value of the difference with the adjustment threshold to obtain a comparison result; When the comparison result indicates that the absolute value of the difference is greater than the adjustment threshold, a first adjustment granularity of the read / write rate of the target block device corresponding to the migration of the source block device is determined from a preset adjustment granularity table according to the absolute value of the difference.

4. The data migration method according to claim 1, characterized in that: The method further comprises: Obtain the original shadow copy created at the last time step when the migration of data in the source storage volume is started; In the process of migrating the data of the source storage volume, when the data of the source storage volume is updated, an updated shadow copy corresponding to the time step from the last time the shadow copy was created to the current time step is created; After completing the data migration of the source storage volume, generating target reference data based on the original shadow copy and at least one updated shadow copy; Comparing the target reference data with the data in the target storage volume to obtain a comparison result; When the comparison result indicates that the data in the target storage volume is inconsistent with the data in the target reference data, the data in the target storage volume is modified based on the target reference data.

5. The data migration method according to claim 1, characterized in that: After the first utilization of the source storage volume is adjusted to the utilization threshold, the method further includes: Regularly obtain the available load of the source storage volume and the processing progress of each source storage volume; When the processing progress at the current moment is less than the processing progress threshold preset at the current moment, the negative load impact factor of the corresponding source storage volume is updated based on the available load.

6. The data migration method according to claim 1, characterized in that: Before determining at least one source storage volume to be migrated in response to the data migration instruction, the method further includes: Obtaining the location of the source storage volume to be migrated in the source block device; For the source storage volume, obtaining multiple load negative impact factors corresponding to multiple time periods; In the source block device, the location of the source storage volume and the multiple load negative impact factors corresponding to the multiple time periods are marked.

7. A data migration method, characterized in that: Applied to a target block device, the method comprises: In response to a read / write rate adjustment instruction for a corresponding target storage volume sent by a source block device according to a first adjustment granularity, wherein the first adjustment granularity is determined by the source block device based on a load negative impact factor corresponding to the source storage volume, and is obtained according to a difference between the first utilization and the utilization threshold; the first utilization is determined based on the process of migrating data in the source storage volume by the source storage volume; the load negative impact factor is obtained after the source block device determines at least one source storage volume that needs to be migrated in response to a data migration instruction; According to the read / write rate adjustment instruction, gradually and smoothly adjust the read / write rate of the target storage volume according to the first adjustment granularity, so as to gradually and smoothly adjust the read / write rate of the source storage volume until the first utilization rate of the source storage volume is adjusted to the utilization rate threshold; When the data in the source storage volume is migrated, the operation of receiving the migrated data is stopped.

8. A data migration device, characterized in that: The device comprises: A response module, configured to determine, in response to a data migration instruction, at least one source storage volume to be migrated, and obtain a preset negative load impact factor of the source storage volume; a migration module, which migrates the data in the source storage volume and determines a first utilization rate of the source storage volume during the migration process; a determination module, configured to determine a utilization threshold of the source storage volume based on the negative load impact factor corresponding to the source storage volume, and determine a first adjustment granularity of a read / write rate of a target block device receiving the source storage volume according to a gap between the first utilization and the utilization threshold; an adjustment module, configured to send a read / write rate adjustment instruction for the corresponding target storage volume to the target block device according to the first adjustment granularity, so that the target block device gradually smoothly adjusts the read / write rate of the target storage volume according to the first adjustment granularity according to the read / write rate adjustment instruction, so as to gradually smoothly adjust the read / write rate of the source storage volume until the first utilization rate of the source storage volume is adjusted to the utilization rate threshold; The stop module is used to stop the data migration operation after the data in the source storage volume is migrated.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the data migration method described in any one of claims 1 to 6 or claim 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data migration method described in any one of claims 1 to 6 or claim 7 is implemented.

Images