Data migration method and device, electronic equipment and storage medium

Abstract

The invention discloses a data migration method and device, electronic equipment and a storage medium, and relates to the technical field of storage, the data migration method comprises the following steps: as the access demand of stored data changes continuously as time goes on, the original storage equipment is not matched with the access demand, so that the access demand of the storage equipment is not matched with the access demand; according to the method, the access type of the storage data in the current period can be determined after the service information corresponding to all the storage data is periodically acquired and analyzed, and then the access type of the storage data in the current period is determined on the basis of the access type and the access information of the storage data in the current period. The to-be-migrated data is migrated into the storage device corresponding to the access type of the to-be-migrated data, so that the access operation of the storage data of the corresponding access type can be realized by fully utilizing the access characteristics of the storage device, and the purpose of fully utilizing resources can be realized.

Description

Technical Field

[0001] This application relates to the field of storage technologies, and in particular, to a data migration method, apparatus, electronic device, and storage medium. Background Art

[0002] In the field of storage technologies, with the rapid development of artificial intelligence and big data technologies, the demand for storage and computing resources has increased sharply. In recent years, memory pooling and video memory pooling technologies have gradually emerged. Therefore, generally, different types of storage devices are centrally managed, and data of different service types are stored in corresponding types of storage devices.

[0003] However, under such management, the resources in the storage devices cannot be fully utilized. Summary of the Invention

[0004] This application provides a data migration method, apparatus, electronic device, storage medium, and program product, which can solve the problem of the inability to fully utilize the resources of storage devices.

[0005] This application provides a data migration method, including:

[0006] In the processing of the current cycle, obtain the service information of each piece of stored data stored in storage devices of multiple access types in the current cycle, where the service information includes access information, service weight value, and data length corresponding to the stored data;

[0007] Determine the access type corresponding to each piece of stored data according to the access information, service weight value, and data length corresponding to each piece of stored data;

[0008] When it is determined that there is at least one piece of stored data that is all data to be migrated according to the access type and access information corresponding to each piece of stored data, migrate each piece of data to be migrated to the storage device corresponding to the access type of the data to be migrated.

[0009] This application also provides a data migration apparatus, including:

[0010] An obtaining module, configured to obtain the service information of each piece of stored data stored in storage devices of multiple access types in the current cycle in the processing of the current cycle, where the service information includes access information, service weight value, and data length corresponding to the stored data;

[0011] A determining module, configured to determine the access type corresponding to each piece of stored data according to the access information, service weight value, and data length corresponding to each piece of stored data;

[0012] A migration module, configured to, when it is determined, according to the access type and access information corresponding to each piece of stored data, that there is at least one piece of stored data that is data to be migrated, migrate each piece of data to be migrated to a storage device corresponding to the access type of the data to be migrated.

[0013] This application also provides an electronic device, including: a memory for storing a computer program; a processor for implementing the steps of any of the above data migration methods when executing the computer program.

[0014] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program implements the steps of any of the above data migration methods when executed by a processor.

[0015] This application also provides a computer program product including a computer program, which implements the steps of any of the above data migration methods when executed by a processor.

[0016] Through this application, since the access requirements of stored data change continuously over time, resulting in a mismatch between the access type of the storage device where it originally resides and its access requirements, therefore, by periodically obtaining the service information corresponding to all stored data for migration operations, real-time adjustment can be achieved. Among them, the service information corresponding to the stored data may include the access information, service weight value, and data length corresponding to the stored data. Since the access information is used to indicate the level of the external access requirements for the stored data, the service weight value reflects the priority of the service to which the stored data belongs, and the data length reflects the size of the resources occupied by the migration of the stored data, therefore, it is more accurate to judge the access type of the stored data through these three factors. Further, when it is determined that there is data to be migrated according to the access type and access information corresponding to each piece of stored data, migrating the data to be migrated to a storage device corresponding to its access type can make full use of the access characteristics of the storage device to implement the access operation of the stored data of the corresponding access type, thereby achieving the purpose of making full use of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To more clearly illustrate the embodiments of this application, the following will briefly introduce the drawings required for the embodiments. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0018] Figure 1 It is a schematic diagram of the architecture of a data migration system provided by an embodiment of this application;

[0019] Figure 2A flowchart of a data migration method provided by an embodiment of the present application;

[0020] Figure 3 A schematic diagram of a data scheduling provided by an embodiment of the present application;

[0021] Figure 4 A flowchart of a data migration device provided by an embodiment of the present application;

[0022] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0023] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0024] It should be noted that in the description of the present application, the terms "including", "comprising" or any other variant thereof are intended to cover a non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0025] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0026] The data migration method provided by the present application can be implemented by a storage management system. As Figure 1 shown, the data migration system may include a management device and multiple storage devices. Among them, the management device may be a computer device, and the access types of the multiple storage devices may be different. For example, the multiple storage devices may include a memory, a video memory, a solid-state drive, etc. The memory, the hard disk, and the video memory may be interconnected with each other through a data transmission bus in pairs. For example, the data transmission bus may be a Peripheral Component Interconnect Express (PCIe). In this way, by being interconnected with each other through the data transmission bus in pairs, a high-speed data transmission network can be formed, which can reduce the data transmission delay and improve the overall performance of the data migration system.

[0027] The memory and the hard disk can be storage devices of the hot access type, and the video memory can be a storage device of the cold access type. The storage device of the hot access type can be used to store the storage data of the hot access type, and the storage device of the cold access type can be used to store the storage data of the cold access type. The storage data of the hot access type is the storage data whose classification index value is greater than the preset classification index value threshold, and the storage data of the cold access type is the storage data whose classification index value is less than or equal to the preset classification index value threshold. For example, the hard disk can be a Non-Volatile Memory Express Solid State Drive (NVMe SSD).

[0028] Embodiments of the present application provide a data migration method, which can be executed by the above-mentioned management device, such as Figure 2 shown, the specific processing steps of the data migration method may include:

[0029] Step S201, in the processing of the current cycle, obtain the service information of each piece of storage data stored in various types of storage devices in the current cycle.

[0030] Among them, the various types of storage devices may be the storage devices in the above storage management system. The service information may include access information corresponding to the storage data, service weight values, and data lengths. Or, the service information may further include the type of the storage device where the storage data is located.

[0031] The access information corresponding to the storage data may include the actual access frequency and access interval duration of the storage data in the current cycle. Or, the access information corresponding to the storage data may further include the second actual access frequency of the storage data in the previous cycle of the current cycle, the first predicted access frequency of the storage data in the previous cycle, etc.

[0032] The actual access frequency may be the number of times the storage data is accessed per unit time. For example, the actual access frequency = the number of accesses in the cycle / the cycle length, and the cycle length may be 30 days.

[0033] The access interval duration may be the time difference between the time when the storage data was last accessed and the end time of the current cycle.

[0034] The data length may be the storage space occupied by the data. Or, it may also be represented by the ratio of the storage space occupied by the storage data to the maximum storage space occupied by a single file preset in the data migration system. That is, the former is an absolute length representation, and the latter is a relative length representation.

[0035] The service weight value can be an indicator of the degree of influence of data on the service. For example, the service weight value of core service data is 2, the service weight value of ordinary service data is 1, and the service weight value of log data is 0.5. The larger the service weight value, the greater the degree of influence of the data on the service, and it can be manually marked.

[0036] Specifically, the management device can periodically obtain the service information of each piece of stored data stored in multiple types of storage devices respectively. After the current cycle ends and the management device obtains the service information of each piece of stored data in the current cycle, it can analyze the service information of each piece of stored data respectively to determine whether each piece of stored data needs to be migrated to perform the data migration operation in the current cycle.

[0037] Step S202: Determine the access type corresponding to each piece of stored data according to the access information, service weight value, and data length corresponding to each piece of stored data.

[0038] Specifically, since the access information is used to indicate the level of the access demand of the outside world for the stored data, the service weight value reflects the priority of the service to which the stored data belongs, and the data length reflects the size of the resources occupied by the migration of the stored data, therefore, it is more accurate to judge the access type of the stored data through these three factors. In addition, since the above three factors are considered when judging the access type of the stored data, in the subsequent process, the access efficiency of the stored data of the hot access type can be improved, and the occupation of the storage device of the hot access type by the stored data of the cold access type can be avoided. In addition, the resources occupied by the migrated data can be reduced and the migration efficiency can be improved.

[0039] Taking the target stored data (any piece of stored data) as an example, the management device can determine the target access type corresponding to the target stored data according to the access information, service weight value, and data length corresponding to the target stored data. Among them, the access information corresponding to the target stored data includes the first actual access frequency and the first access interval duration of the target stored data in the current cycle. Correspondingly, it can specifically include the following steps:

[0040] Step 1: Determine the target classification index value of the target stored data according to the first actual access frequency, the first access interval duration, the service weight value corresponding to the target stored data, and the data length.

[0041] Step 2: Determine the target access type corresponding to the target stored data according to the target classification index value and the preset classification index value threshold.

[0042] Among them, Step 1 can include:

[0043] Step 1, determine the first classification index value according to the first weight value, the first actual access frequency, and the second preset access frequency threshold.

[0044] Step 2, determine the second classification index value according to the second weight value and the first access interval duration.

[0045] Step 3, determine the third classification index value according to the third weight value and the service weight value corresponding to the target stored data.

[0046] Step 4, determine the fourth classification index value according to the fourth weight value, the data length corresponding to the target stored data, and the preset data length threshold.

[0047] Step 5, determine the target classification index value according to the first classification index value, the second classification index value, the third classification index value, and the fourth classification index value.

[0048] For example, Step 1 can adopt the following mathematical expression:

[0049]

[0050] Among them, C is the target classification index value, α is the first weight value, F t is the first actual access frequency, F max is the second preset access frequency threshold, is the first classification index value. β is the second weight value, T is the first access interval duration, is the second classification index value. γ is the third weight value, I is the service weight value corresponding to the target stored data, γ×I is the third classification index value. ω is the fourth weight value, S is the data length corresponding to the target stored data, S max is the preset data length threshold, is the fourth classification index value. and can achieve data normalization, eliminate the dimensional differences between different features, and make the data comparable. The first weight value, the second weight value, the third weight value, and the fourth weight value can all be preset.

[0051] After determining the target classification metric value of the target stored data through the above steps, the management device can determine whether the target classification metric value is greater than a preset classification metric value threshold (for example, it can be 0.55). If so, it means that the access demand for the target stored data is high and the migration cost is low. The management device can determine the hot access type as the target access type of the target stored data. If not, it means that the access demand for the target stored data is low and the migration cost is high. The management device can determine the cold access type as the target access type of the target stored data. Generally, the range of the classification metric value of the stored data of the cold access type is -0.2 to 0.5, and the range of the classification metric value of the stored data of the hot access type is 0.6 to 1.2.

[0052] In some alternative embodiments, the service information may further include a service type. For example, the service type may be a log, user data, backup data, etc. After obtaining the service information of each stored data, the management device can extract the service type from the service information of each stored data. When it is determined that the service type corresponding to the stored data is a log, the type of the stored data can be directly determined as the cold access type. Or, when it is determined that the service type corresponding to the stored data is not a log, step S203 is then executed. In this way, both the classification accuracy can be ensured and the migration efficiency can be improved.

[0053] Step S203, when it is determined, according to the access type and access information corresponding to each stored data, that there is at least one stored data that is data to be migrated, migrate each piece of data to be migrated to a storage device corresponding to the access type of the data to be migrated.

[0054] Specifically, if directly determining whether to migrate data only based on the access type of the stored data, it may result in a relatively high migration cost. Therefore, after determining the access type corresponding to the stored data, according to the access type corresponding to the stored data, elements corresponding to the access type can be selected from the access information corresponding to the stored data, and an analysis operation corresponding to the access type is performed, so as to more accurately determine whether the stored data is migration data, reduce unnecessary migration operations, and thus save the migration cost. Correspondingly, taking the target stored data as an example, the management device can select elements corresponding to the target access type from the range information corresponding to the target stored data, and then perform an analysis operation corresponding to the target access type to determine whether the target stored data is data to be migrated. Specifically, it can be divided into the following two cases:

[0055] Case 1: When the target access type is a hot access type, determine the second predicted access frequency of the target stored data in the current cycle according to the first actual access frequency and the first predicted access frequency (the predicted access frequency of the target stored data in the previous cycle). When it is determined that the second predicted access frequency is greater than the first preset access frequency threshold and the type of the target storage device does not match the target access type, determine that the target stored data is the data to be migrated.

[0056] Among them, the first actual access frequency and the first predicted access frequency are the access information corresponding to the hot access type. For example, the first preset access frequency threshold can be 70%.

[0057] Specifically, since the actual access frequency in the current cycle can indicate the real access demand of the stored data at present, and the predicted access frequency determined in the previous cycle can indicate the access demand of the stored data in the current cycle, therefore, by combining the actual access frequency and the predicted access frequency in the current cycle, the access demand of the stored data in the future can be predicted. Furthermore, when the predicted access frequency is greater than the first preset access frequency threshold and the type of the storage device where the stored data is located does not match its corresponding access type, the stored data is determined as the data to be migrated. By considering the future access demand, migration is only carried out when the future access demand is high and the stored data is not currently in the storage device of the hot access type, which can save costs. In addition, by pre-migrating the stored data with high future access demand to the storage device of the hot access type, the access efficiency of the stored data in the next cycle can be improved.

[0058] For example, to determine the second predicted access frequency of the target stored data in the current cycle according to the first actual access frequency and the first predicted access frequency, the following mathematical expression can be used:

[0059] P t = μ × F t +(1 - μ) × P t-1 Formula (2)

[0060] Among them, P t is the second predicted access frequency, α is the fifth weight value (which can be preset), P t-1 is the first predicted access frequency, and F t is the first actual access frequency.

[0061] In some alternative embodiments, when it is determined that the second predicted access frequency is less than or equal to the first preset access frequency threshold, it is determined that the access frequency of the target stored data in the next cycle of the current cycle is low. However, since there may be errors in the prediction and the target stored data itself is of the hot access type, therefore, the target stored data may not need to be migrated in the storage device of the hot access type. If the target stored data is in the storage device of the cold access type and meets the storage location corresponding to the second predicted access frequency, it may also not be migrated. In summary, the migration cost can be saved.

[0062] In some alternative embodiments, since the actual access frequency can indicate the real access demand of the stored data at present, and the predicted access frequency can indicate the access demand of the stored data in the future, and the actual access frequency in the current cycle and the predicted access frequency in the previous cycle can reflect the accuracy of the prediction result, and the actual access frequency in the current cycle and the actual access frequency in the previous cycle can reflect the access demand change information of the stored data, therefore, by comprehensively considering the actual access frequency and the predicted access frequency of the stored data in the previous cycle, and the actual access frequency of the stored data in the current cycle, to determine the predicted access frequency of the stored data in the next cycle can be more accurate. Furthermore, when the predicted access frequency is greater than the first preset access frequency threshold and the type of the storage device where the stored data is located does not match its corresponding access type, the stored data is determined as the data to be migrated. Correspondingly, taking the target stored data as an example, the second predicted access frequency of the target stored data in the current cycle can also be calculated by the following mathematical expression:

[0063] P t = k1 × F t + k2 × P t-1 + k3 × (F t - F t-1 ) Formula (3)

[0064] Where P t is the second predicted access frequency, k1 is the sixth weight value, k2 is the seventh weight value, P t-1 is the first predicted access frequency, k3 is the eighth weight value, F t is the first actual access frequency, and F t-1 is the second actual access frequency. The sixth weight value, the seventh weight value, and the eighth weight value can be preset, and the sum of the three can be 1.

[0065] In Case 2, when the target access type is a cold access type, determine the target frequency change value of the target stored data according to the first actual access frequency and the second actual access frequency. When it is determined that the target frequency change value is less than or equal to the first preset amplitude threshold, update the first count value. When it is determined that the first count value is equal to the first preset value and the type of the target storage device does not match the target access type, determine that the target stored data is data to be migrated.

[0066] Among them, the first count value is used to indicate the number of consecutive periods in which the frequency change value of the target stored data is less than the first preset threshold. The first preset amplitude threshold can be less than 0. For example, it can be -0.3.

[0067] Specifically, since the change value of the actual access frequency of the stored data in two adjacent periods can represent the change in access demand, and the decline amplitude of the actual access frequency in multiple consecutive periods is relatively large, it can be explained that the access demand of the stored data has been decreasing, and it can be migrated to a storage device of the cold access type to release the resources occupied by the storage device of the hot access type.

[0068] Taking the target stored data as an example, the management device can determine the target frequency change value of the target stored data according to the first actual access frequency and the second actual access frequency. For example, the following mathematical expression can be used:

[0069]

[0070] Among them, Δ is the target frequency change value, F t is the first actual access frequency, F t-1 is the second actual access frequency, F max is the second preset access frequency threshold.

[0071] For example, the access times of the target stored data in cycle 1 are 90 times, in cycle 2 are 60 times, in cycle 3 are 30 times, and in cycle 4 are 0 times. The second preset access frequency threshold is 100. Through formula (4), it can be determined that: the frequency change value from cycle 1 to cycle 2 is -0.33, the frequency change value from cycle 2 to cycle 3 is -0.33, the frequency change value from cycle 3 to cycle 4 is -0.33, and the frequency change values in 3 consecutive cycles are all less than 0.3. It can be determined that the target stored data is data to be migrated.

[0072] Or, the following mathematical expression can also be used:

[0073] Δ = F t - F t-1 Formula (5)

[0074] Among them, Δ is the target frequency change value, Ft is the first actual access frequency, F t-1 is the second actual access frequency.

[0075] In some alternative embodiments, when the target access type is a hot access type, the management device may also determine that the target frequency change value is greater than or equal to a second preset amplitude threshold, and determine that the second predicted access frequency is greater than a first preset access frequency threshold, and when the type of the target storage device does not match the target access type, then determine the target stored data as the data to be migrated.

[0076] Among them, the second preset amplitude threshold may be greater than 0. For example, it may be 0.2.

[0077] In this way, the frequency change value in one cycle increases greatly, which can indicate that its access demand suddenly increases, indicating that there is very likely to be a high access demand in the next cycle. By considering this change value, it is possible to more accurately determine whether the stored data is the data to be migrated.

[0078] In some alternative embodiments, since the setting of the preset classification index value threshold is a general value, when the utilization rate of the storage device is relatively high, it indicates that its load is relatively high. If more data is migrated to the storage device with a high utilization rate, it will cause the storage device to crash and malfunction. On the contrary, when the utilization rate of the storage device is relatively low, it indicates that its load is relatively low. If less data is migrated to the storage device with a low utilization rate, it will cause the resources to be not fully utilized. Therefore, to avoid the above problems, before determining the access type of the stored data in the current cycle, the preset classification index value threshold can be adjusted in the following way:

[0079] Obtain the first utilization rate of the storage device of the hot access type and the second utilization rate of the storage device of the cold access type. When it is determined that the first utilization rate is greater than a first preset utilization rate threshold and the second utilization rate is less than a second preset utilization rate threshold, raise the preset classification index value threshold to the value corresponding to the first utilization rate to jointly determine the target access type with the target classification index value. Or, when it is determined that the first utilization rate is less than a third preset utilization rate threshold and the second utilization rate is greater than a fourth preset utilization rate threshold, lower the preset classification index value threshold to the value corresponding to the second utilization rate to jointly determine the target access type with the target classification index value.

[0080] Specifically, when the first utilization rate of the storage device of the hot access type is relatively high and the second utilization rate of the storage device of the cold access type is relatively low, the preset classification index value threshold can be increased to reduce the proportion of storage data determined to be of the hot access type, so as to avoid further increasing the load on the storage device of the hot access type. At the same time, the proportion of storage data determined to be of the cold access type can be increased, thereby improving the resource utilization rate of the storage device of the cold access type. Or, when the second utilization rate of the storage device of the cold access type is relatively high and the first utilization rate of the storage device of the hot access type is relatively low, the preset classification index value threshold can be decreased to reduce the proportion of storage data determined to be of the cold access type, so as to avoid further increasing the load on the storage device of the cold access type. At the same time, the proportion of storage data determined to be of the hot access type can be increased, thereby improving the resource utilization rate of the storage device of the hot access type and enabling more data to be accessed efficiently.

[0081] For example, the preset classification index value threshold is 0.55. When the utilization rate of the hot access type is greater than 90% and the utilization rate of the cold access type is less than 50%, the preset classification index value threshold can be increased to 0.6. When the utilization rate of the hot access type is less than 50% and the utilization rate of the cold access type is greater than 90%, the preset classification index value threshold can be decreased to 0.5.

[0082] Finally, when it is determined that there is at least one piece of storage data to be migrated, the management device can directly migrate each piece of storage data to the storage device corresponding to its access type.

[0083] Or, the management device can also first determine the migration order of each piece of storage data to be migrated, and then perform the migration operation based on the migration order of each piece of storage data to be migrated. Specifically, it can include:

[0084] Step 1: Determine the migration order of each piece of storage data to be migrated according to the data length and service weight value corresponding to each piece of storage data to be migrated.

[0085] Step 2: Migrate each piece of storage data to be migrated to the storage device corresponding to the access type of the storage data to be migrated according to the migration order of each piece of storage data to be migrated.

[0086] Since the throughput of the storage device is limited, and the service weight value of the storage data reflects the importance of the corresponding service, and the data length reflects the migration cost and migration speed, therefore, by combining the data length, service weight value, and the number of data, the migration order of the storage data can be reasonably arranged, thereby improving the migration efficiency and orderliness. Correspondingly, determining the migration order of each piece of storage data to be migrated according to the data length and service weight value corresponding to each piece of storage data to be migrated can specifically include the following steps:

[0087] Step 1: Determine the priority of each piece of data to be migrated based on the data length and business weight value corresponding to each piece of data to be migrated.

[0088] Step 2: Count the number of data to be migrated.

[0089] Step 3: Determine the migration order of each piece of data to be migrated based on the priority of each piece of data to be migrated and the number of data to be migrated.

[0090] Specifically, taking the target migrated data (any piece of migrated data) as an example, the management device can first determine the priority corresponding to the target data to be migrated based on the data length and business weight value corresponding to the target data to be migrated. Based on a similar method, the management device can calculate the priority of each type of data to be migrated. The management device can count the number of data to be migrated, determine the ratio between the number of data to be migrated and the preset number, and determine this ratio as the migration batch. For each migration batch, the data to be migrated with a priority ranked before the order corresponding to the preset number is used as the first migration batch, and the migration order of the data to be migrated in the first migration frequency is the same. By analogy, it is possible to determine the data to be migrated included in each migration batch, that is, to determine the migration order of each piece of data to be migrated.

[0091] For example, Step 1 can adopt the following expression:

[0092]

[0093] where A is the priority, I is the business weight value, and S is the data length.

[0094] In this way, the management device can migrate each piece of data to be migrated to the storage device corresponding to its access type according to the migration order of each piece of data to be migrated, that is, migrate the data to be migrated with the hot access type to the storage device with the hot access type, and migrate the data to be migrated with the cold access type to the storage device with the cold access type.

[0095] In some alternative embodiments, taking the target stored data as an example, after determining the access type of the target stored data, a cold / hot recognition model corresponding to the target access type can be selected according to the target access type of the target stored data, and the service information of the target stored data can be input into the cold / hot recognition model corresponding to the target access type to obtain the re-recognized access type. If the two recognition results are inconsistent, the target stored data can be directly marked as data that does not need to be migrated. If the two recognition results are consistent, step S203 can be executed. For example, the cold / hot recognition mode can be a Mixture of Experts (MoE) model. The MoE model corresponding to the hot access type is a dense MoE model, and the MoE model corresponding to the cold access type is a sparse MoE model.

[0096] In some alternative embodiments, the management device can record the data volume of the data to be migrated in the current cycle, the time consumed in the migration process, and the number of migrations of each stored data, and generate an evaluation report based on the data volume of the data to be migrated, the time consumed in the migration process, and the number of migrations of each stored data, so that technicians can determine whether there are problems with the current data migration strategy and optimize it.

[0097] In some alternative embodiments, the management device can calculate the misjudgment rate according to the access type of each stored data determined in the previous cycle and the actual access frequency of each stored data in the current cycle. Specifically, it can include:

[0098] The management device determines whether the actual access frequency of each stored data is greater than the preset access frequency threshold. If so, the access type of the stored data can be determined as the hot access type. If not, the access type of the stored data can be determined as the cold access type. Further, taking the target stored data as an example, the management device can determine whether the type determined for the target stored data in the previous cycle according to steps S201 to S203 is consistent with the type determined only according to the actual access frequency in the current cycle, and record the first quantity of the consistent stored data and the second quantity of the inconsistent stored data. Finally, calculate the sum value between the first quantity and the second quantity, and determine the ratio between the second quantity and the sum value as the misjudgment rate. When it is determined that the misjudgment rate is greater than the preset misjudgment rate threshold, the above-mentioned second weight value is increased by a preset value to determine the classification index value of the stored data in the current cycle.

[0099] In the data migration method of the embodiments of the present application, since the access requirements of stored data change continuously over time, resulting in a mismatch between the access type of the storage device where it originally resides and its access requirements. Therefore, by periodically obtaining the service information corresponding to all stored data for migration operations, real-time adjustment can be achieved. Among them, the service information corresponding to the stored data may include access information corresponding to the stored data, service weight values, and data lengths. Since the access information is used to indicate the level of access requirements of the external world for the stored data, the service weight value reflects the priority of the service to which the stored data belongs, and the data length reflects the size of the resources occupied by the migration of the stored data. Therefore, judging the access type of the stored data based on these three factors is more accurate. Further, when it is determined that there is data to be migrated according to the access type and access information corresponding to each stored data, the data to be migrated is migrated to the storage device corresponding to its access type, which can make full use of the access characteristics of the storage device to implement the access operation of the stored data of the corresponding access type, thereby achieving the purpose of making full use of resources.

[0100] The following uses a specific example to illustrate the above data migration method. The data migration system may include NVMe SSD, memory, video memory, and other storage devices. Among them, NVMe SSD and memory can be storage devices of the hot access type, and video memory and storage devices can be storage devices of the cold access type. As Figure 3 shown, before storing data, the management device can first perform initialization operations on the above-mentioned various types of storage devices respectively. For example, initializing the NVMe SSD may include: detecting all NVMe SSD devices in the data migration system, loading all NVMe SSD device drivers, and configuring storage capacity, performance parameters, and dividing the NVMe SSD device into multiple logical partitions. Initializing the memory may include: detecting all available memory resources in the data migration system, configuring the memory capacity and performance parameters, and dividing the memory into multiple memory blocks of a fixed size. Initializing the video memory may include: detecting all Graphics Processing Unit (GPU) devices and their video memory resources in the data migration system, loading the GPU driver, configuring the video memory capacity and performance parameters, and dividing the video memory into multiple video memory blocks of a fixed size. To ensure that data can be quickly transferred between the NVMe SSD pool, memory pool, and video memory pool, configure the bus bandwidth and transmission protocol. Initialize the bus management module to support the fast transfer and dynamic scheduling of data.

[0101] After completing the above initialization operations, the management device can perform the operation of storing data, and after a preset duration, set the migration period and start to periodically perform the operation of migrating data.

[0102] For each piece of stored data stored in each storage device, the management device can perform cold and hot identification operations, that is, first extract the service information of each piece of stored data recorded by itself, determine the access type of each piece of stored data according to the service information of each piece of stored data, and then, according to the access type of each piece of stored data, schedule the stored data (similar to the processing of steps S201 to S203, which will not be elaborated here).

[0103] In addition, the management device can also perform relevant monitoring and feedback operations. For example, monitor the resource usage of the storage device and the performance parameters of the migration process (such as the data volume of the data to be migrated, the time taken for the migration process), adjust the preset classification metric values based on the resource usage of the storage device (such as the utilization rate), generate an evaluation report for feedback based on the performance parameters of the migration process, and optimize the data migration strategy. Among them, the resource usage can include indicators such as the number of input / output operations per second (IOPS), bandwidth utilization, latency time, and utilization rate.

[0104] In this way, through periodic scheduling operations, the advantages of the storage device can be utilized to the greatest extent to process stored data, and the overall data access efficiency can be improved.

[0105] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0106] An embodiment of the present application also provides a data migration device, as Figure 4 shown, including:

[0107] An acquisition module 410, configured to acquire the service information of each piece of stored data respectively stored in storage devices of multiple access types in the current cycle during the processing of the current cycle, where the service information includes access information corresponding to the stored data, a service weight value, and a data length;

[0108] A determination module 420, configured to determine the access type corresponding to each piece of stored data according to the access information, service weight value, and data length respectively corresponding to each piece of stored data;

[0109] A migration module 430, configured to, when it is determined according to the access type and access information corresponding to each piece of stored data that there is at least one piece of stored data that is data to be migrated, migrate each piece of data to be migrated to a storage device corresponding to the access type of the data to be migrated.

[0110] In some alternative embodiments, the access information corresponding to the target stored data includes: the first actual access frequency and the first access interval duration of the target stored data in the current period, where the target stored data is any piece of stored data;

[0111] The determining module 420 is specifically configured to:

[0112] Determine the target classification metric value of the target stored data according to the first actual access frequency, the first access interval duration, the service weight value and the data length corresponding to the target stored data;

[0113] Determine the target access type corresponding to the target stored data according to the target classification metric value and the preset classification metric value threshold.

[0114] In some alternative embodiments, the access information corresponding to the target stored data further includes: the first predicted access frequency of the target stored data in the previous period of the current period;

[0115] The migration module 430 is specifically configured to:

[0116] When the target access type is a hot access type, determine the second predicted access frequency of the target stored data in the current period according to the first actual access frequency and the first predicted access frequency;

[0117] When it is determined that the second predicted access frequency is greater than the first preset access frequency threshold and the access type of the target storage device to which the target stored data belongs does not match the target access type, determine that the target stored data is data to be migrated.

[0118] In some alternative embodiments, the access information corresponding to the target stored data further includes: the second actual access frequency of the target stored data in the previous period;

[0119] The migration module 430 is specifically configured to:

[0120] When the target access type is a cold access type, determine the target frequency change value of the target stored data according to the first actual access frequency and the second actual access frequency;

[0121] When it is determined that the target frequency change value is less than or equal to the first preset amplitude threshold, update the first count value, where the first count value is used to indicate the number of consecutive periods in which the frequency change value of the target stored data is less than the first preset threshold;

[0122] When it is determined that the first count value is equal to the first preset value and the access type of the target storage device does not match the target access type, determine that the target stored data is data to be migrated.

[0123] In some alternative embodiments, the apparatus further includes an adjustment module 440 for:

[0124] Obtain the first utilization rate of the storage device for the hot access type and the second utilization rate of the storage device for the cold access type;

[0125] When it is determined that the first utilization rate is greater than the first preset utilization rate threshold and the second utilization rate is less than the second preset utilization rate threshold, raise the preset classification index value threshold to the value corresponding to the first utilization rate, so as to jointly determine the target access type with the target classification index value.

[0126] In some alternative embodiments, the adjustment module 440 is further configured to:

[0127] When it is determined that the first utilization rate is less than the third preset utilization rate threshold and the second utilization rate is greater than the fourth preset utilization rate threshold, lower the preset classification index value threshold to the value corresponding to the second utilization rate, so as to jointly determine the target access type with the target classification index value.

[0128] In some alternative embodiments, the migration module 430 is specifically configured to:

[0129] Determine the migration order of each piece of data to be migrated according to the data length and service weight value corresponding to each piece of data to be migrated;

[0130] Migrate each piece of data to be migrated to the storage device corresponding to the access type of the data to be migrated according to the migration order of each piece of data to be migrated.

[0131] For the description of the features in the embodiments corresponding to the data migration device, reference can be made to the relevant description in the embodiments corresponding to the data migration method, which will not be elaborated here one by one.

[0132] An embodiment of the present application also provides an electronic device, as Figure 5 shown, including a memory 10 and a processor 20. A computer program is stored in the memory 10, and the processor 20 is configured to run the computer program to execute the steps in any one of the above embodiments of the data migration method. For example, the electronic device may be the above management device.

[0133] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is configured to execute the steps in any one of the above embodiments of the data migration method when running.

[0134] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to: various media that can store computer programs such as USB flash drives, read-only memory (ROM), random access memory (RAM), mobile hard disks, magnetic disks, or optical discs.

[0135] The embodiments of the present application also provide a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above-described embodiments of the data migration method.

[0136] The embodiments of the present application also provide another computer program product, including a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the steps in any of the above-described embodiments of the data migration method.

[0137] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0138] The above has introduced in detail a data migration method, device, electronic device, storage medium, and program product provided by the present application. Specific examples are used in this article to elaborate on the principles and implementation manners of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. It should be noted that for those of ordinary skill in the art in the technical field, without departing from the principle of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A data migration method, characterized in that, Including: In the processing of the current cycle, obtain the service information of each piece of stored data stored in the storage devices of multiple access types in the current cycle, where the service information includes access information corresponding to the stored data, a service weight value, and a data length; Determine the access type corresponding to each piece of stored data according to the access information, service weight value, and data length respectively corresponding to each piece of stored data; When it is determined that there is at least one piece of stored data that is all data to be migrated according to the access type and access information corresponding to each piece of stored data, migrate each piece of data to be migrated to the storage device corresponding to the access type of the data to be migrated.

2. The data migration method according to claim 1, wherein The access information corresponding to the target stored data includes: the first actual access frequency and the first access interval duration of the target stored data in the current cycle, and the target stored data is any piece of stored data; Determining the target access type corresponding to the target stored data according to the access information, service weight value, and data length corresponding to the target stored data includes: Determine the target classification index value of the target stored data according to the first actual access frequency, the first access interval duration, and the service weight value and data length corresponding to the target stored data; Determine the target access type corresponding to the target stored data according to the target classification index value and the preset classification index value threshold.

3. The data migration method according to claim 2, wherein The access information corresponding to the target stored data further includes: the first predicted access frequency of the target stored data in the previous cycle of the current cycle; Determining whether the target stored data is the data to be migrated according to the access type and access information corresponding to the target stored data includes: When the target access type is a hot access type, determine the second predicted access frequency of the target stored data in the current cycle according to the first actual access frequency and the first predicted access frequency; When it is determined that the second predicted access frequency is greater than the first preset access frequency threshold and the access type of the target storage device to which the target stored data belongs does not match the target access type, determine that the target stored data is the data to be migrated.

4. The data migration method according to claim 3, wherein The access information corresponding to the target stored data further includes: the second actual access frequency of the target stored data in the previous cycle; The method further includes: When the target access type is a cold access type, determine the target frequency change value of the target stored data according to the first actual access frequency and the second actual access frequency; When it is determined that the target frequency change value is less than or equal to the first preset amplitude threshold, update the first count value, where the first count value is used to indicate the number of consecutive cycles in which the frequency change value of the target stored data is less than the first preset threshold; When it is determined that the first count value is equal to the first preset value and the access type of the target storage device does not match the target access type, determine that the target stored data is the data to be migrated.

5. The data migration method according to any one of claims 2 to 4, characterized in that, Before determining the target access type corresponding to the target stored data according to the target classification index value and the preset classification index value threshold, the method further includes: Obtaining a first utilization rate of a storage device of a hot access type and a second utilization rate of a storage device of a cold access type; When it is determined that the first utilization rate is greater than a first preset utilization rate threshold and the second utilization rate is less than a second preset utilization rate threshold, raising the preset classification index value threshold to a value corresponding to the first utilization rate for jointly determining the target access type with the target classification index value.

6. The data migration method according to claim 5, wherein The method further includes: When it is determined that the first utilization rate is less than a third preset utilization rate threshold and the second utilization rate is greater than a fourth preset utilization rate threshold, lowering the preset classification index value threshold to a value corresponding to the second utilization rate for jointly determining the target access type with the target classification index value.

7. The data migration method according to any one of claims 2 to 4, characterized in that The migrating each piece of the data to be migrated to a storage device corresponding to the access type of the data to be migrated includes: Determining a migration order for each piece of the data to be migrated according to the data length and service weight value corresponding to each piece of the data to be migrated; Migrating each piece of the data to be migrated to a storage device corresponding to the access type of the data to be migrated according to the migration order of each piece of the data to be migrated.

8. A data migration device, characterized in that, It includes: An obtaining module, configured to obtain, in the processing of the current cycle, service information of each piece of stored data respectively stored in storage devices of multiple access types in the current cycle, where the service information includes access information, service weight value, and data length corresponding to the stored data; A determining module, configured to determine an access type corresponding to each piece of the stored data according to the access information, service weight value, and data length respectively corresponding to each piece of the stored data; A migrating module, configured to, when it is determined that there is at least one piece of stored data that is all data to be migrated according to the access type and access information corresponding to each piece of the stored data, migrate each piece of the data to be migrated to a storage device corresponding to the access type of the data to be migrated.

9. An electronic device, characterized in that, It includes: A memory, configured to store a computer program; A processor, configured to implement the steps of the data migration method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, where the computer program implements the steps of the data migration method according to any one of claims 1 to 7 when executed by a processor.

Images