A data migration method, device, equipment and medium

Abstract

The present application relates to the field of data processing, in particular to a data migration method, device, equipment and medium, the data migration method is executed by distributed storage equipment, the data migration method comprises: when receiving a data migration request, determining a migration mode according to the number of read-write operations per second of current business, in the case of high business load, by limiting the number of read-write operations per second of migration, effectively avoid the interference of migration activity to business operation, ensure the smooth operation of business;After the completion of full migration, the partial business redirection mode is used, that is, only the business corresponding to the object accessed in the migration process is redirected to the target volume, while the business corresponding to the object not accessed still remains in the source volume, reducing the influence on business continuity in the migration process, and improving the flexibility of migration.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a data migration method, apparatus, device, and medium. Background Technology

[0002] Digital transformation is profoundly impacting all industries, and data has evolved from a supporting tool to a core asset for enterprise decision-making and operations. During data center construction and maintenance, operations such as storage device replacement, system upgrades, and load balancing are frequently required, often involving the migration of large amounts of data. Data migration is the technology that moves data from a source storage location to a target location.

[0003] The relevant technical solution is online migration, which moves data from the source storage location to the target location while ensuring continuous business access; the large number of migration IO requests and business IO has led to a significant decrease in business IO.

[0004] Therefore, how to migrate data to ensure smooth business I / O operation has become an urgent problem to be solved in this field. Summary of the Invention

[0005] The purpose of this invention is to provide a data migration method, apparatus, device, and medium that can determine the migration mode based on the number of read and write operations per second of the current business. Under high business load, by limiting the number of read and write operations per second during migration, the interference of migration activities on business operations is effectively avoided, ensuring the smooth operation of the business.

[0006] Firstly, a data migration method is provided, executed by a distributed storage device, the data migration method comprising:

[0007] When a migration request is received to migrate data from the source volume to the target volume, the number of read and write operations per second for the current business is obtained.

[0008] Based on the number of read and write operations per second of the current service, a migration mode is determined. The migration mode includes a first mode and a second mode. The first mode requires limiting the number of read and write operations per second during migration, and the second mode is a migration-priority processing mode. During the migration process according to the migration mode, the number of read and write operations per second during the migration process is limited by the migration mode.

[0009] In the migration mode, all objects from the source volume are migrated to the target volume.

[0010] If the full migration is completed, the first service will be redirected to the target volume, while the second service will remain directed to the source volume. The object corresponding to the first service is an object that was not accessed during the migration process, and the object corresponding to the second service is an object that was accessed during the migration process.

[0011] In a preferred embodiment, the present invention can be further configured as follows: after a full migration is completed, the first service is redirected to the target volume, while the second service remains directed to the source volume, and the invention further includes:

[0012] Incrementally synchronize the object corresponding to the second service in the source volume to the target volume;

[0013] The second service is redirected to the target volume.

[0014] In a preferred embodiment, the present invention may further be configured to include:

[0015] Obtain record information corresponding to multiple objects, wherein the record information records whether each object has been accessed;

[0016] Based on the record information corresponding to multiple objects, determine the accessed objects and the unaccessed objects from among the multiple objects.

[0017] In a preferred embodiment, the present invention can be further configured such that: the first mode uses a token bucket algorithm to limit the number of read and write operations per second for migration IO;

[0018] The process of using the token bucket algorithm to limit the number of read and write operations per second during migration includes:

[0019] When an object migration request for the target object of the source volume arrives, if a token exists in the token bucket, the token is retrieved so that the object migration request can be processed using the token;

[0020] The token bucket is set with a maximum number of tokens, and tokens are periodically added to the token bucket according to the token generation rate.

[0021] In a preferred embodiment, the present invention may further be configured to include:

[0022] The system acquires in real time the maximum number of read / write operations per second, the number of read / write operations per second for the current business, and the fluctuation value, wherein the fluctuation value is determined based on the historical number of read / write operations per second for the business.

[0023] The number of read / write operations available per second is determined based on the maximum number of read / write operations per second and the current number of read / write operations per second for the business.

[0024] Based on a preset formula, the token generation rate is determined according to the number of read / write operations per second available, the fluctuation value, the maximum number of tokens, and the token generation cycle.

[0025] The preset formula is: Token generation rate R = (Number of read / write operations per second available - Security threshold) / (Token bucket size × Token generation cycle).

[0026] In a preferred embodiment, the present invention can be further configured such that: the full migration of objects from the source volume to the target volume includes:

[0027] Obtain detailed information about the objects in the source volume, including: object type, object access frequency, and object size;

[0028] Based on the preset correspondence relationship corresponding to the target information in the detailed information, the sub-adjustment value corresponding to the target information in the detailed information is determined. The preset correspondence relationship is the correspondence relationship between information and sub-adjustment value. The target information is any one of object category, object access frequency, and object size.

[0029] For each object, if there exists a sub-adjustment value that is not less than the preset highest priority, then the object's priority is set to the preset highest priority; if there exists a sub-adjustment value that is not greater than the preset lowest priority, and there is no sub-adjustment value that is not less than the preset highest priority, then the object's priority is set to the preset lowest priority; otherwise, the sub-adjustment levels corresponding to each piece of information are added together to obtain the adjustment level, and the object's priority is determined based on the adjustment level and the base value.

[0030] Based on the priority of each object in the source volume, they are fully migrated to the target volume in descending order of priority.

[0031] In a preferred embodiment, the present invention may further be configured to include:

[0032] During the migration process, when the number of migrated objects reaches a preset number, the objects that have been migrated to the target volume are clustered according to object category to obtain objects corresponding to multiple object categories.

[0033] For each object category, select at least one object from the objects corresponding to the object category, and perform a consistency check between the at least one object and the corresponding object of the source volume;

[0034] When the verification fails, it is marked so that after the full migration is completed, all objects of the object category corresponding to the marked object will be migrated again.

[0035] Secondly, a data migration apparatus is provided, comprising:

[0036] The acquisition module is used to acquire the number of read and write operations per second of the current business when a migration request for data to be migrated from the source volume to the target volume is received;

[0037] The determination module is used to determine the migration mode based on the number of read and write operations per second of the current service. The migration mode includes a first mode and a second mode. The first mode requires limiting the number of read and write operations per second during migration, and the second mode is a migration-priority processing mode. During the migration process according to the migration mode, the number of read and write operations per second during the migration process is limited by the migration mode.

[0038] A migration module is used to migrate all objects from the source volume to the target volume in the migration mode.

[0039] The redirection module is used to redirect the first service to the target volume and keep the second service directed to the source volume if the full migration is completed. The object corresponding to the first service is an object that has not been accessed during the migration process, and the object corresponding to the second service is an object that has been accessed during the migration process.

[0040] Thirdly, a distributed storage device is provided, comprising:

[0041] One or more processors;

[0042] Memory;

[0043] One or more applications, wherein the applications are stored in memory and configured to be executed by one or more processors, the applications being configured to: perform operations corresponding to the data migration method shown in any possible implementation of the first aspect.

[0044] Fourthly, a computer-readable storage medium is provided, the storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded by a processor and executed according to the steps of the data migration method shown in any possible implementation of the first aspect.

[0045] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements operations corresponding to the data migration method shown in any possible implementation of the first aspect.

[0046] In summary, the data migration method provided by this invention is executed by a distributed storage device, and the data migration method has the following beneficial technical effects:

[0047] When a migration request to migrate data from the source volume to the target volume is received, the number of read / write operations per second for the current service is obtained. Based on the number of read / write operations per second for the current service, the migration mode is determined. The migration mode includes a first mode and a second mode. The first mode requires limiting the number of read / write operations per second during the migration. In the migration mode, all objects in the source volume are migrated to the target volume. If the full migration is completed, the first service is redirected to the target volume, while the second service remains redirected to the source volume. The objects corresponding to the first service are those that were not accessed during the migration process, while the objects corresponding to the second service are those that were accessed during the migration process.

[0048] In this invention, when a data migration request is received, the migration mode is determined based on the number of read and write operations per second of the current business. Under high business load, by limiting the number of read and write operations per second during migration, interference with business operations during migration is effectively avoided, ensuring smooth business operation. After the full migration is completed, a partial business redirection method is adopted, that is, only the business corresponding to the accessed objects during the migration process is redirected to the target volume, while the business corresponding to the unaccessed objects remains in the source volume, reducing the impact on business continuity during migration and improving the flexibility of migration.

[0049] In addition, the present invention also provides a data migration device, equipment and medium, all of which have the above-mentioned beneficial technical effects. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 This is a schematic diagram of a data migration method provided in an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of another data migration method provided in an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the structure of a data migration device provided in an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of the structure of a distributed storage device provided in an embodiment of the present invention. Detailed Implementation

[0055] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but as long as they are within the scope of the present invention, they are protected by patent law.

[0056] It should be noted that, in the optional embodiments of the present invention, the data related to object information, etc., requires the permission or consent of the object when the embodiments of the present invention are applied to specific products or technologies. Furthermore, the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of the present invention involve data related to an object, it needs to be obtained with the object's authorization and consent, the authorization and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If the embodiments involve personal information, the acquisition of all personal information requires the individual's consent. If sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the object's authorization and consent.

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0059] Digital transformation is profoundly impacting all industries, and data has evolved from a supporting tool to a core asset for enterprise decision-making and operations. During data center construction and maintenance, operations such as storage device replacement, system upgrades, and load balancing are frequently required, often involving the migration of large amounts of data. Data migration is the technology of moving data from a source storage location to a target location. Volume migration faces numerous technical challenges: firstly, data consistency must be guaranteed to ensure no data loss or corruption during the migration process. The migration speed is also a key concern for enterprises; and a robust fault handling mechanism is essential.

[0060] Related technical solutions include offline migration, which requires stopping business access. Its advantage is that the migration method is simple and it is not easy to cause problems such as data inconsistency. However, its disadvantage is that modern enterprises often require business to run 24 hours a day, which does not meet the actual needs. Therefore, the current trend of volume migration is mostly online migration, that is, the technology of moving data from the source storage location to the target location while ensuring continuous business access.

[0061] The technical implementation of online volume migration needs to properly handle the relationship between newly generated IO requests and the migration of existing data. Furthermore, it requires controlling the impact of migration on business performance and possessing a robust fault handling mechanism. Existing solutions mostly employ IO redirection, redirecting business operations to the target volume. The drawback is that the large number of migration IO requests and business IO significantly reduces business IO. Specifically, the number of read / write operations per second can drop by up to 50%, having a significant impact on business operations, especially during periods of high business pressure. However, when business pressure is low, the migration task rate decreases considerably, making it impossible to dynamically adjust the migration speed based on actual business conditions.

[0062] Online volume migration is mainly implemented at the DBS layer and can be divided into three stages: prepare, execute, and commit. The prepare stage is responsible for preparing the target volume, creating a target volume snapshot, and recording migration relationships; the execute stage is responsible for data migration; and the commit stage is responsible for cleaning up the source volume and other finishing work to complete the entire online volume migration.

[0063] This invention primarily focuses on optimizing the workflow design during the execute phase.

[0064] This invention addresses the problem of fixed and singular migration modes during online migration, which cannot adapt to different business situations, and proposes a new migration processing solution. This solution includes two migration modes: a first mode and a second mode. When business pressure is high, the first mode, prioritizing business I / O, can be adopted to ensure that normal business operations are not affected. When business pressure is low, the second mode can be adopted to focus on online data migration and complete the migration task more quickly. The migration task is completed in two phases: the first phase is a full migration, where business I / O is located on the source volume; the second phase is an incremental migration, where business I / O is located on the target volume, through a bitmap object (…). Accurately track data changes.

[0065] This invention provides a data migration method, such as... Figure 1 As shown, the method provided in this embodiment of the invention can be executed by a distributed storage device. The method includes:

[0066] S101. When a migration request is received to migrate data from the source volume to the target volume, obtain the number of read and write operations per second for the current service.

[0067] In scenarios such as system upgrades, data backups, and disaster recovery, it is necessary to completely migrate data from the source volume to the target volume. The source volume is the original volume of the storage object, serving as the starting point for the migration. The target volume is the destination location for the migration. Migration requests can be initiated by the administrator or automatically triggered at set intervals; this embodiment does not limit this. The current service refers to the pending business when a migration request for data to be migrated from the source volume to the target volume is received; the number of read / write operations per second for the current service's I / O is the number of input / output requests for read / write operations on the data storage device that can be processed per second.

[0068] S102. Determine the migration mode based on the number of read and write operations per second of the current service.

[0069] The migration modes include: mode 1 and mode 2. Mode 1 requires limiting the number of read and write operations per second during migration.

[0070] During data storage migration, the number of read / write operations per second (IO) directly reflects the current system load and business operation status. Assessing the current IO ...

[0071] In some embodiments, it can be determined whether the business I / O is at a high level by setting a threshold for the number of read / write operations per second or a range of thresholds for the number of read / write operations per second, so as to determine the migration mode.

[0072] It should be noted that if the user does not specify a migration mode, the migration mode can be determined by the number of read and write operations per second of the current business I / O; however, if the user specifies a migration mode, the user-specified mode will be used first.

[0073] Furthermore, the first mode needs to limit the number of read and write operations per second during migration. The specific range of the limit for the number of read and write operations per second during migration can be specified by the user or determined by a preset limit relationship. The preset limit relationship is the correspondence between the IO capacity range of multiple distributed storage devices and the number of read and write operations per second during multiple migrations. This is so that when the IO capacity attributes of the distributed storage devices are obtained, the corresponding limit for the number of read and write operations per second during migration can be determined by the preset limit relationship.

[0074] For example, the preset limiting relationships include: the IO capability range a of the distributed storage device corresponds to the number of read / write operations per second (m) of the migration limit; the IO capability range b of the distributed storage device corresponds to the number of read / write operations per second (n) of the migration limit; and the IO capability range c of the distributed storage device corresponds to the number of read / write operations per second (p) of the migration limit. If the current IO of the distributed storage device is d, which is within the IO capability range a of the distributed storage device, then the number of read / write operations per second (m) of its migration limit is determined.

[0075] S103. In migration mode, all objects from the source volume are migrated to the target volume.

[0076] In this embodiment of the invention, multiple objects from the source volume can be copied one by one to the target volume, and business I / O can be performed during the migration process.

[0077] Furthermore, the full migration process also includes: obtaining record information corresponding to multiple objects, with the record information containing information on whether each object has been accessed; and determining the accessed and unaccessed objects from among the multiple objects based on the record information corresponding to multiple objects.

[0078] In this embodiment of the invention, a bitmap object can be designed. As the record information corresponding to multiple objects, each bit corresponds to a data shard object. A bit set to 1 indicates that the object has changed after the migration process starts, while a bit set to 0 indicates that there has been no change.

[0079] In some possible scenarios, the source volume objects are fully migrated to the target volume. During this process, only objects that undergo business I / O after the migration are recorded as accessed objects. For example, the source volume includes objects 1, 2, 3, and 4; there are business I / O operations a (corresponding to object 1), b (corresponding to object 2), and c (corresponding to object 3). At the first moment, business I / O a is executed. At this time, object 1 has already migrated to the target volume, so object 1 is marked as an accessed object. At the second moment, business I / O b is executed. At this time, object 2 has not yet migrated to the target volume, and the content of object 2 may have changed. However, since object 2 is migrated at the third moment, the migrated object 2 is the changed object 2, and there is no need to mark object 2. In other possible scenarios, the source volume objects are fully migrated to the target volume. During this process, only objects that undergo business I / O are recorded, and all accessed objects are recorded as accessed objects. Continuing the example above, when business I / O b is executed at the second moment, object 2 needs to be recorded.

[0080] S104. If the full migration is completed, redirect the first service to the target volume, while keeping the second service directed to the source volume.

[0081] The objects corresponding to the first business are those that were not accessed during the migration process, while the objects corresponding to the second business are those that were accessed during the migration process.

[0082] During a full migration, since the business I / O remains unchanged and the target location is still in the source volume, the impact on business I / O performance is minimal.

[0083] After the full migration is complete, a business switchover is required to ensure uninterrupted service. To guarantee the correct identification of accessed object data, not all services will be immediately switched to the target volume. Instead, targeted redirection will be performed based on the access behavior of services during the migration process. Since some objects may have been accessed during the migration, and their data may have changed, the services for those accessed objects will remain directed to the source volume. Services containing the accessed objects (i.e., the objects corresponding to the second business) will be redirected to the source volume to ensure their normal operation and data consistency. Meanwhile, services containing the accessed objects (i.e., the objects corresponding to the second business) will continue to access the source volume to reduce unnecessary risks until the objects corresponding to the first business have also been fully migrated to the target volume, at which point they will be redirected to the target volume.

[0084] As can be seen, in this embodiment of the invention, when a data migration request is received, the migration mode is determined based on the number of read and write operations per second of the current business IO. Under high business load, by limiting the number of read and write operations per second of the migration IO, the interference of the migration activity on business operations is effectively avoided, ensuring the smooth operation of business IO. After the full migration is completed, a partial business redirection method is adopted, that is, only the business corresponding to the accessed object during the migration process is redirected to the target volume, while the business corresponding to the unaccessed object remains in the source volume, reducing the impact on business continuity during the migration process and improving the flexibility of the migration.

[0085] One possible implementation of this invention, after completing a full migration, redirects the first service to the target volume while keeping the second service directed to the source volume, further includes: incrementally synchronizing the object corresponding to the second service in the source volume to the target volume; and redirecting the second service to the target volume.

[0086] In this embodiment of the invention, during incremental migration, only the changed data objects are updated, the amount of data synchronized is small, the impact on business I / O is small, and the needs of business scenarios are better met.

[0087] Furthermore, after the incremental migration is completed, the consistency and integrity of the data can be verified. If there are no problems, the second service will be redirected to the target volume.

[0088] As can be seen, in this embodiment of the invention, the accessed objects are incrementally synchronized to the target volume to ensure that the target volume stores the latest object data, and then reclassified. This not only reduces the impact of data migration on business operations, but also ensures the continuity and availability of data. By gradually migrating and redirecting through a full and then incremental approach, the performance of the system is guaranteed.

[0089] One possible implementation of this invention is a first mode that uses a token bucket algorithm to limit the number of read and write operations per second during migration. The process of using the token bucket algorithm to limit the number of read and write operations per second during migration includes: when an object migration request for the target object of the source volume arrives, if there are tokens in the token bucket, the tokens are retrieved to facilitate the processing of the object migration request; the token bucket is set with a maximum number of tokens, and tokens are periodically added to the token bucket according to the token generation rate.

[0090] The token bucket algorithm is a commonly used flow control algorithm used to limit the data transmission rate or I / O processing rate.

[0091] In this embodiment of the invention, each object corresponds to one object migration request, and each object migration request requires one token. Specifically, a token bucket is set, along with a maximum number of tokens and a token generation rate. The maximum number of tokens represents the maximum number of migration I / O requests that can be processed simultaneously, and the token generation rate controls the speed at which tokens are generated in the token bucket, thereby determining the number of migration I / O requests that can be processed per unit time.

[0092] When a migration I / O request for the target object of the source volume arrives, the system checks if there is a token in the token bucket. If a token exists, it is retrieved from the token bucket and used to process the migration I / O request. If there is no token in the token bucket, the migration I / O request is temporarily delayed. The system periodically adds tokens to the token bucket according to the token generation rate until the number of tokens in the bucket reaches the maximum number of tokens.

[0093] As can be seen, in this embodiment of the invention, the token bucket algorithm is used to limit the number of read and write operations per second during migration IO, thereby ensuring that the data migration process does not have an excessive impact on the operation of business IO. Setting the maximum number of tokens and the token generation rate parameters allows for flexible control of the processing rate of migration IO, and the data migration process can be completed gradually while ensuring business continuity; it can not only realize object migration, but also reduce the interference and impact on business operations during the migration process.

[0094] One possible implementation of this invention further includes: acquiring in real time the maximum number of read / write operations per second of the system, the number of read / write operations per second of the current service, and a fluctuation value, wherein the fluctuation value is determined based on the historical number of read / write operations per second of the service; determining the available number of read / write operations per second based on the maximum number of read / write operations per second and the current number of read / write operations per second of the service; and determining the token generation rate based on a preset formula, using the available number of read / write operations per second, the fluctuation value, the maximum number of tokens, and the token generation cycle; wherein the preset formula is: token generation rate R = (available number of read / write operations per second - security threshold) / (token bucket size × token generation cycle).

[0095] The maximum number of read / write operations per second (bps) represents the maximum number of input / output operations that the distributed storage device can support. The current number of read / write operations per second (bps) represents the number of input / output operations the distributed storage device is currently processing, specifically the real-time state of the workload. The fluctuation value is determined based on the historical number of read / write operations per second (bps) for various services, indicating the degree to which the number of read / write operations per second for a service changes over time. Available read / write operations per second = Maximum number of read / write operations per second - Current number of read / write operations per second for the service.

[0096] In this embodiment of the invention, the fluctuation value can be calculated in several ways. Optionally, based on multiple historical read / write operation counts per second, a maximum and a minimum value are determined; a first difference between the maximum value and the mean, and a second difference between the minimum value and the mean are calculated; the average of the absolute values ​​of the first and second differences is used as the fluctuation value. Optionally, based on the changing patterns of historical read / write operation counts per second, the number of read / write operations per second in the future period is predicted, and the difference between the predicted value and the actual value is calculated as the fluctuation value of the number of read / write operations per second.

[0097] As can be seen, in this embodiment of the invention, the token generation rate is dynamically determined by comprehensively considering the number of read / write operations per second, the fluctuation value, the maximum number of tokens, and the token generation cycle. This allows the token generation rate to be reduced when the business IO load increases, thereby reducing the impact of migration on the business. When the business IO load decreases, the token generation rate is increased to accelerate the progress of data migration and effectively balance data migration and business operation.

[0098] One possible implementation of this invention involves fully migrating objects from the source volume to the target volume, including: SA1-SA4, wherein:

[0099] SA1. Obtain detailed information about the objects in the source volume, including: object type, object access frequency, and object size;

[0100] Object category indicates the type of object, such as text file, image file, or database table. Object access frequency refers to the number of times the object has been accessed during a historical period, used to assess the object's popularity. Object size indicates the amount of storage space occupied by the object.

[0101] SA2. Based on the preset correspondence of the target information in the detailed information, determine the sub-adjustment value corresponding to the target information in the detailed information. The preset correspondence is the correspondence between information and sub-adjustment value. The target information is any one of the following: object category, object access frequency, or object size.

[0102] Preset mapping relationships refer to pre-set mapping relationships, which are based on experience.

[0103] Specifically, based on the first preset correspondence of object categories, a first sub-adjustment value corresponding to the object category is determined. The first preset correspondence is a correspondence between multiple sub-adjustment values ​​and multiple object categories. Based on the second preset correspondence of object access frequency, a second sub-adjustment value corresponding to object access frequency is determined. The second preset correspondence is a correspondence between multiple sub-adjustment values ​​and multiple object access frequency ranges. Based on the third preset correspondence of object size, a third sub-adjustment value corresponding to object size is determined. The third preset correspondence is a correspondence between multiple sub-adjustment values ​​and multiple object size ranges.

[0104] SA3. For each object, if there is a sub-adjustment value that is not less than the preset highest priority, then set the object's priority to the preset highest priority; if there is a sub-adjustment value that is not greater than the preset lowest priority, and there is no sub-adjustment value that is not less than the preset highest priority, then set the object's priority to the preset lowest priority; otherwise, add up the sub-adjustment levels corresponding to each piece of information to obtain the adjustment level, and determine the object's priority based on the adjustment level and the base value.

[0105] In this embodiment of the invention, a preset highest priority, a preset lowest priority, and a base value of 0 are set.

[0106] If any of the first, second, and third sub-adjustment values ​​is greater than the preset highest priority, then the priority of the object is set to the preset highest priority. This is so that when an object is very important in a certain dimension, its priority is set to the preset highest priority to prevent interference from other dimensions.

[0107] If there is a sub-adjustment value lower than the preset minimum priority, and there is no sub-adjustment value higher than the preset maximum priority, then the priority of the object is set to the preset minimum priority. This is to ensure that if an object is not important in one dimension and is not very important in other dimensions, it is set to the preset minimum priority.

[0108] Otherwise, the final priority of the object is determined by adding the sub-adjustment levels of the multiple dimensions to the base value, where the final priority is between the preset highest priority and the preset lowest priority.

[0109] SA4. Based on the priority of each object in the source volume, migrate all objects to the target volume in descending order of priority.

[0110] As can be seen, in this embodiment of the invention, the priority of an object is determined by comparing the sub-adjustment value of the object with the preset priority threshold, so as to perform a full migration according to the priority order of the objects and ensure that the most important objects can be transferred to the target volume first during the migration process.

[0111] One possible implementation of this invention further includes:

[0112] During the migration process, when the number of migrated objects reaches a preset number, the objects that have been migrated to the target volume are clustered according to object category to obtain objects corresponding to multiple object categories.

[0113] For each object category, select at least one object from the objects corresponding to the object category, and perform a consistency check between at least one object and the corresponding object in the source volume;

[0114] When the verification fails, it is marked so that after the full migration is completed, all objects of the object category corresponding to the marked object will be migrated again.

[0115] In this embodiment of the invention, a preset number N is set. Whenever the number of migrated objects reaches N, a clustering and consistency verification process is triggered. Specifically, the migrated objects on the target volume are traversed, and clustering is performed according to object categories. For each object category, at least one object is randomly selected from the clustered object set as a verification object. The verification object on the target volume is compared with the corresponding object on the source volume to check whether the content, size, and metadata are consistent. If the verification fails, the verification object is marked, and its category and identification information are recorded. After all objects on the source volume have been migrated to the target volume, for each marked object category, all objects of that category are reread from the source volume and migrated to the target volume again. The objects migrated again are clustered and consistency verification is performed again to ensure that all objects have been correctly migrated to the target volume.

[0116] As can be seen, in this embodiment of the invention, inconsistent objects can be detected and handled in a timely manner during the migration process, ensuring the accuracy and integrity of the migration.

[0117] In one possible implementation of this invention, if a system failure occurs during the full migration process, the migration task is automatically paused, and the current migration progress and data status are saved; when the system returns to normal, the migration task is automatically resumed, and the migration continues from the last saved progress and data status.

[0118] Specifically, acquire multi-dimensional data on historical failures, and use a decision model based on this data to determine the failure status. This multi-dimensional data includes metrics such as data transfer speed, error rate, CPU utilization, memory usage, and disk I / O.

[0119] The training process of the decision model includes: acquiring training samples, which include multiple historical data and corresponding fault results; training the decision model based on the training samples to obtain a trained decision model.

[0120] Specifically, based on multi-dimensional data and corresponding fault conditions (fault conditions are represented by numerical values, with larger values ​​indicating a higher probability of failure); based on multi-dimensional data from multiple samples and corresponding fault conditions, the overall information entropy is determined; based on the data for each dimension, the information entropy corresponding to each dimension is obtained; based on the overall information entropy and the information entropy of each dimension, the gain of each dimension is calculated as the weight of each dimension, thus obtaining the decision model.

[0121] During the migration process, if a failure occurs due to network interruption, disk failure, or other reasons, it may affect the migration task. In this case, the current migration task is paused, and the actions of reading objects from the source volume and writing to the target volume are stopped. The current migration progress and data status are recorded. Once the failure is resolved, the migration task is resumed, and the migration continues based on the migration progress and data status. This ensures that even if a failure occurs during the full migration process, the migration task can be automatically paused and the current migration progress and data status can be saved. When the system returns to normal, the migration task can be automatically resumed and the migration can continue from the last saved progress and data status, thereby improving the reliability of the migration process.

[0122] Based on any of the above embodiments, this invention provides a specific data migration method, including:

[0123] The system is divided into two modes based on business load. In mode one, a token bucket algorithm is used to limit the number of read / write operations per second during I / O migration, ensuring that business I / O performance does not degrade excessively. A bitmap object is designed. Each bit corresponds to a data shard object. A bit set to 1 indicates that the object has changed after the migration process started, while a bit set to 0 indicates that there has been no change. The migration process is performed object-by-object, based on a mapping between the source and target volumes. The migration is divided into two phases: full migration and incremental migration. The overall control flow is simple and clear, resulting in high business I / O performance and minimal impact.

[0124] Specific methods are as follows Figure 2 As shown:

[0125] Step 1: Create the target volume and establish migration relationships;

[0126] Step 2: Start the thread to begin a full migration of each of the N objects, from the source volume to the target volume.

[0127] Step 3: Select the migration mode based on the business IO load. In the first mode, conventional methods (such as the token bucket algorithm) are used to limit the number of read and write operations per second during migration IO to ensure that the performance of business IO does not degrade excessively. In the second mode, there is no limit on the number of read and write operations per second during IO.

[0128] Step 4: Business I / O is located in the source volume, and creation is performed simultaneously. Objects accessed by business I / O are recorded as 1, and those not accessed are recorded as 0.

[0129] Step 5: Full migration complete, business operations are redirected to the target volume. When business I / O accesses the target volume, if the target volume object is 0, it accesses the target volume directly; if the object is 1, it accesses the source volume.

[0130] Step 6: According to Incremental synchronization is performed on objects marked as 1. Upon completion of synchronization, [the following will be displayed]. Change the corresponding object in the middle to 0.

[0131] Step 7: When all data in the middle is 0, the migration is complete.

[0132] As can be seen, in this embodiment of the invention, the number of read / write operations per second for adaptive migration is limited according to the business load, and can be flexibly switched according to the actual load to meet the needs of different scenarios. The migration process is divided into two stages using a full + incremental approach to complete the online migration task, migrating objects one by one, achieving object-level granular control of migration. The first stage's business I / O is located on the source volume, with a shorter O path and stronger performance; simultaneously, [the system] creates... Objects accessed by business I / O are recorded as 1, and those not accessed are recorded as 0. The second phase of business I / O resides on the target volume and is migrated online incrementally, avoiding the repeated migration of unmodified data blocks and significantly reducing data transfer volume. During online migration, a marker is added when object synchronization is complete. The corresponding position is 0, when When all values ​​are 0, the migration task is complete, and the service is switched to the target volume. After the service redirection, it is then... The markers enable accurate I / O routing, ensuring consistent data access; they are only cleared after incremental synchronization is complete. Tags are used to ensure the integrity of data synchronization.

[0133] The following describes an apparatus provided by an embodiment of the present invention. The apparatus described below can be referred to in correspondence with the method described above. The apparatus of this embodiment is installed in a distributed storage device. Figure 3 , Figure 3 This is a structural block diagram of a device according to one embodiment of the present invention, comprising:

[0134] The acquisition module 210 is used to acquire the number of read and write operations per second of the current business when a migration request for data to be migrated from the source volume to the target volume is received;

[0135] The determination module 220 is used to determine the migration mode based on the number of read and write operations per second of the current business. The migration mode includes a first mode and a second mode. The first mode needs to limit the number of read and write operations per second during migration.

[0136] Migration module 230 is used to migrate all objects from the source volume to the target volume in migration mode;

[0137] The redirection module 240 is used to redirect the first service to the target volume and keep the second service directed to the source volume if the full migration is completed. The object corresponding to the first service is the object that was not accessed during the migration process, and the object corresponding to the second service is the object that was accessed during the migration process.

[0138] One possible approach also includes:

[0139] The incremental synchronization module is used to incrementally synchronize the objects corresponding to the second service in the source volume to the target volume; and to redirect the second service to the target volume.

[0140] One possible approach also includes:

[0141] The record information acquisition module is used to acquire record information corresponding to multiple objects, and the record information includes whether each object has been accessed.

[0142] The object access information determination module is used to determine the accessed objects and the unaccessed objects from multiple objects based on the record information corresponding to multiple objects.

[0143] In one feasible approach, the first mode employs a token bucket algorithm to limit the number of read and write operations per second during migration;

[0144] The process of using the token bucket algorithm to limit the number of read and write operations per second during migration includes:

[0145] When an object migration request for the target object of the source volume arrives, if a token exists in the token bucket, the token is retrieved so that the migration request can be processed using the token;

[0146] The token bucket is set with a maximum number of tokens, and tokens are periodically added to the token bucket according to the token generation rate.

[0147] One possible approach also includes:

[0148] The token generation rate determination module is used to obtain in real time the maximum number of read and write operations per second of the system, the number of read and write operations per second of the current business, and the fluctuation value. The fluctuation value is determined based on the number of read and write operations per second of the historical business.

[0149] The number of read / write operations available per second is determined based on the maximum number of read / write operations per second and the current number of read / write operations per second for the business.

[0150] Based on a preset formula, the token generation rate is determined according to the number of read / write operations per second available, the fluctuation value, the maximum number of tokens, and the token generation cycle.

[0151] The preset formula is: Token generation rate R = (Number of read / write operations per second available - Security threshold) / (Token bucket size × Token generation cycle).

[0152] In one possible implementation, migration module 230 is used to obtain detailed information about objects in the source volume, including: object type, object access frequency, and object size.

[0153] Based on the preset correspondence of the target information in the detailed information, determine the sub-adjustment value corresponding to the target information in the detailed information. The preset correspondence is the correspondence between information and sub-adjustment value. The target information is any one of the following: object category, object access frequency, or object size.

[0154] For each object, if there exists a sub-adjustment value that is not less than the preset highest priority, then the object's priority is set to the preset highest priority; if there exists a sub-adjustment value that is not greater than the preset lowest priority, and there is no sub-adjustment value that is not less than the preset highest priority, then the object's priority is set to the preset lowest priority; otherwise, the sub-adjustment levels corresponding to each piece of information are added together to obtain the adjustment level, and the object's priority is determined based on the adjustment level and the base value.

[0155] Based on the priority of each object in the source volume, they are migrated to the target volume in descending order of priority.

[0156] In one possible implementation, it also includes: a verification module, used for:

[0157] During the migration process, when the number of migrated objects reaches a preset number, the objects that have been migrated to the target volume are clustered according to object category to obtain objects corresponding to multiple object categories.

[0158] For each object category, select at least one object from the objects corresponding to the object category, and perform a consistency check between at least one object and the corresponding object in the source volume;

[0159] When the verification fails, it is marked so that after the full migration is completed, all objects of the object category corresponding to the marked object will be migrated again.

[0160] This invention provides a distributed storage device, such as... Figure 4 As shown, Figure 4 The distributed storage device 300 shown includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the distributed storage device 300 may further include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one, and the structure of this distributed storage device 300 does not constitute a limitation on the embodiments of the present invention.

[0161] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0162] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0163] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory), or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0164] The memory 303 stores application code that executes the present invention and is controlled by the processor 301. The processor 301 executes the application code stored in the memory 303 to implement the content shown in the foregoing method embodiments.

[0165] Figure 4 The distributed storage device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0166] This invention provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0167] This invention provides a computer program product, including a computer program that, when executed by a processor, implements the corresponding content in the aforementioned method embodiments.

[0168] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0169] The above are only some embodiments of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data migration method, characterized in that, The data migration method, performed by a distributed storage device, includes: When a migration request is received to migrate data from the source volume to the target volume, the number of read and write operations per second for the current business is obtained. Based on the number of read / write operations per second of the current service, a migration mode is determined, wherein the migration mode includes: a first mode and a second mode. The first mode requires limiting the number of read / write operations per second during migration; the second mode is a migration-priority processing mode. During the migration process according to the migration mode, the number of read / write operations per second during the migration process is limited by the migration mode. Under the migration mode, all objects of the source volume are migrated to the target volume. If the full migration is completed, the first service will be redirected to the target volume, while the second service will remain directed to the source volume. The object corresponding to the first service is an object that was not accessed during the migration process, and the object corresponding to the second service is an object that was accessed during the migration process.

2. The data migration method according to claim 1, characterized in that, If the full migration is completed, the first service will be redirected to the target volume, while the second service will remain directed to the source volume. The process will then include: Incrementally synchronize the object corresponding to the second service in the source volume to the target volume; The second service is redirected to the target volume.

3. The data migration method according to claim 1, characterized in that, Also includes: Obtain record information corresponding to multiple objects, wherein the record information records whether each object has been accessed; Based on the record information corresponding to multiple objects, determine the accessed objects and the unaccessed objects from among the multiple objects.

4. The data migration method according to claim 1, characterized in that, The first mode uses a token bucket algorithm to limit the number of read and write operations per second during migration; The process of using the token bucket algorithm to limit the number of read and write operations per second during migration includes: When an object migration request for the target object of the source volume arrives, if a token exists in the token bucket, the token is retrieved so that the object migration request can be processed using the token; The token bucket is set with a maximum number of tokens, and tokens are periodically added to the token bucket according to the token generation rate.

5. The data migration method according to claim 4, characterized in that, Also includes: The system acquires in real time the maximum number of read / write operations per second, the number of read / write operations per second for the current business, and the fluctuation value, wherein the fluctuation value is determined based on the historical number of read / write operations per second for the business. The number of read / write operations available per second is determined based on the maximum number of read / write operations per second and the current number of read / write operations per second for the business. Based on a preset formula, the token generation rate is determined according to the number of read / write operations per second available, the fluctuation value, the maximum number of tokens, and the token generation cycle. The preset formula is: Token generation rate R = (Number of read / write operations per second available - Security threshold) / (Token bucket size × Token generation cycle).

6. The data migration method according to claim 1, characterized in that, The step of fully migrating the objects from the source volume to the target volume includes: Obtain detailed information about the objects in the source volume, including: object type, object access frequency, and object size; Based on the preset correspondence relationship corresponding to the target information in the detailed information, the sub-adjustment value corresponding to the target information in the detailed information is determined. The preset correspondence relationship is the correspondence relationship between information and sub-adjustment value. The target information is any one of object category, object access frequency, and object size. For each object, if there exists a sub-adjustment value that is not less than the preset highest priority, then the object's priority is set to the preset highest priority; if there exists a sub-adjustment value that is not greater than the preset lowest priority, and there is no sub-adjustment value that is not less than the preset highest priority, then the object's priority is set to the preset lowest priority; otherwise, the sub-adjustment levels corresponding to each piece of information are added together to obtain the adjustment level, and the object's priority is determined based on the adjustment level and the base value. Based on the priority of each object in the source volume, they are fully migrated to the target volume in descending order of priority.

7. The data migration method according to claim 6, characterized in that, Also includes: During the migration process, when the number of migrated objects reaches a preset number, the objects that have been migrated to the target volume are clustered according to object category to obtain objects corresponding to multiple object categories. For each object category, select at least one object from the objects corresponding to the object category, and perform a consistency check between the at least one object and the corresponding object of the source volume; When the verification fails, it is marked so that after the full migration is completed, all objects of the object category corresponding to the marked object will be migrated again.

8. A data migration device, characterized in that, include: The acquisition module is used to acquire the number of read and write operations per second of the current business when a migration request for data to be migrated from the source volume to the target volume is received; The determination module is used to determine the migration mode based on the number of read and write operations per second of the current service. The migration mode includes a first mode and a second mode. The first mode requires limiting the number of read and write operations per second during migration, and the second mode is a migration-priority processing mode. During the migration process according to the migration mode, the number of read and write operations per second during the migration process is limited by the migration mode. A migration module is used to migrate all objects from the source volume to the target volume in the migration mode. The redirection module is used to redirect the first service to the target volume and keep the second service directed to the source volume if the full migration is completed. The object corresponding to the first service is an object that has not been accessed during the migration process, and the object corresponding to the second service is an object that has been accessed during the migration process.

9. A distributed storage device, characterized in that, include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the steps of the data migration method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded by a processor and executed according to the steps of the data migration method according to any one of claims 1 to 7.

Images