Data migration scheduling method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data migration scheduling method and device, equipment, a storage medium and a program product. The data migration scheduling method comprises the following steps: acquiring metadata information of target data stored on a warm storage device, a data application weight, device information of the warm storage device and a device information weight; the data application weight is used for indicating the feature expression of the access behavior of the user to access the data; the equipment information weight is used for indicating the influence degree of the hardware environment of the temperature storage equipment on the data migration decision; analyzing and processing the metadata information and the equipment information to obtain a data application score and an equipment information score; performing weighted summation on the weights and the scores to obtain a total score; and under the condition that the total score meets a preset score condition, migrating the target data from the warm storage device to the cold storage device. The data migration scheduling method can effectively distinguish whether the data is warm or cold, and the technical effect of reducing the data storage cost is achieved.

Description

Technical Field

[0001] The present application belongs to the field of information technology, and in particular, relates to a data migration scheduling method, device, equipment, storage medium and program product. Background Art

[0002] In the existing technical solutions, the migration strategy configuration of file data relies on static configuration strategies. The operation and maintenance personnel or operation personnel formulate file warm and cold rules based on experience and determine the file migration strategy. The storage device completes the data migration action according to the established strategy. The migration strategy is executed once or periodically.

[0003] Existing solutions and static scheduling algorithms are feasible in traditional information technology management scenarios, because each enterprise is familiar with its own business, and operation and maintenance personnel can formulate migration strategies and select the optimal static migration algorithm based on business characteristics. Existing static scheduling algorithms often consider scheduling time, system throughput, and job type, and as the amount of data increases, it is difficult for operation and maintenance personnel to have enough experience to understand the business data of all tenants. Therefore, existing migration scheduling algorithms may have technical problems such as difficulty in effectively judging the warmth and coldness of data and migrating data to more appropriate storage devices. Summary of the invention

[0004] The embodiments of the present application provide a data migration scheduling method, apparatus, device, storage medium and program product, which can solve the technical problem of how to more accurately determine the warmth and coldness of data and migrate the data to a more suitable storage device.

[0005] In a first aspect, an embodiment of the present application provides a data migration scheduling method, the method comprising:

[0006] Obtain metadata information of target data stored on the warm storage device, data application weight, device information of the warm storage device, and device information weight; the data application weight is used to indicate the characteristic expression of the access behavior of the user accessing the data; the device information weight is used to indicate the degree of influence of the hardware environment of the warm storage device on the data migration decision;

[0007] Analyze and process metadata information to obtain data application scores;

[0008] Analyze and process the device information to obtain a device information score;

[0009] The data application weight, device information weight, data application score and device information score are weighted and summed to obtain a total score;

[0010] When the total score meets the preset score condition, the target data is migrated from the warm storage device to the cold storage device.

[0011] In one possible implementation, the method further includes: the metadata information includes: access frequency and data type;

[0012] Analyze and process metadata information to obtain data application scores, including:

[0013] According to the access frequency and the data type, an access frequency score and a data type score are determined respectively;

[0014] The data application score is determined based on the access frequency score and the data type score.

[0015] In an achievable implementation, the method further includes: the device information includes occupied device capacity and device load pressure; and the device information is analyzed and processed to obtain a device information score, including:

[0016] According to the occupied equipment capacity and the equipment load pressure, a capacity score and a load pressure score are determined respectively;

[0017] The equipment information score is determined based on the capacity score and the load pressure score.

[0018] In one possible implementation, the method further includes: after migrating the target data from the warm storage device to the cold storage device, the method further includes:

[0019] Mark the warm-cold period of the target data. The warm-cold period is the time period during which the target data will not be used after being migrated to the cold storage device.

[0020] When a recall request for target data is received within the warm-down period, the data application weight is updated to reduce negative incentives for the data application weight.

[0021] In an achievable implementation, the method further includes: before updating the data application weight, the method further includes:

[0022] The recall request is determined to be an unexpected recall request.

[0023] In an achievable implementation, the method further includes: updating the data application weight, including:

[0024] The data application weight is updated by applying a weight regulator to reduce negative incentives for data application weight.

[0025] In one possible implementation, the method further includes: migrating the target data from the warm storage device to the cold storage device, including:

[0026] Determine the migration time and migration rate of data migration based on metadata information and device information;

[0027] At the migration time, the target data is migrated from the warm storage device to the cold storage device according to the migration rate.

[0028] In a second aspect, an embodiment of the present application provides a data migration scheduling device, the device comprising: a data monitoring module, configured to obtain metadata information of target data stored on a warm storage device;

[0029] Device monitoring module, used to obtain device information of warm storage devices;

[0030] A migration strategy module, for recording data application weights and device information weights, and for analyzing and processing metadata information to obtain a data application score; analyzing and processing device information to obtain a device information score; and performing weighted summation of the data application weights, device information weights, data application scores, and device information scores to obtain a total score;

[0031] Among them, the data application weight is used to indicate the characteristic expression of the user's access behavior to access data; the device information weight is used to indicate the degree of influence of the hardware environment of the warm storage device on the data migration decision;

[0032] The data scheduling module is used to migrate the target data from the warm storage device to the cold storage device when the total score meets the preset score condition.

[0033] In a third aspect, an embodiment of the present application provides a data migration scheduling device, the device comprising: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the data migration scheduling method provided in any one of the above embodiments.

[0034] In a fourth aspect, an embodiment of the present application provides a computer storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the data migration scheduling method provided in any one of the above embodiments is implemented.

[0035] In a fifth aspect, an embodiment of the present application provides a computer program product. When instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data migration scheduling method provided by any one of the above embodiments.

[0036] The data migration scheduling method provided in the embodiment of the present application takes into account the characteristics of the target data itself and the operating status of the storage device at the same time, and realizes a technical means of judging the data warmth and coldness by comprehensively considering multiple factors, thereby increasing the accuracy of the data warmth and coldness judgment and realizing a more reasonable data migration strategy; and the characteristics of the data application weight itself determine that the data migration scheduling method also has the characteristics of self-optimization and self-adaptation; thus, the data migration scheduling method provided in the embodiment of the present application not only takes into account the characteristics of the target data, but also takes into account the changes in the user calling behavior of the target data over time, thereby making self-optimization and providing a more reasonable data migration strategy; in the comprehensive consideration After considering the data application weight, device information weight, data application score and device information score, the total score is compared with the preset score conditions to determine whether the data is warm or cold, and a reliable basis for determining whether cold data is migrated from warm storage devices is provided; the data migration scheduling method provided in the embodiment of the present application can effectively transfer cold data in warm storage devices to other storage devices, and retain warm data in warm storage devices, so that the user end can call warm data from warm storage devices; furthermore, the data migration scheduling method provided in the embodiment of the present application can migrate data to a more suitable storage device, and can achieve the technical effect of reducing the storage cost of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 It is a schematic diagram of a scenario structure of a data migration scheduling method in the prior art;

[0039] Figure 2 It is a schematic diagram of a scenario structure of a data migration scheduling method provided in an embodiment of the present application;

[0040] Figure 3 It is a flowchart of a data migration scheduling method provided in an embodiment of the present application;

[0041] Figure 4 It is a structural diagram of an intelligent scheduling system provided in an embodiment of the present application;

[0042] Figure 5 yes Figure 4 A schematic diagram of the principle of the intelligent scheduling algorithm in the intelligent scheduling system shown;

[0043] Figure 6 yes Figure 3A flowchart of a method for analyzing and processing metadata information in the data migration scheduling method shown.

[0044] Figure 7 yes Figure 3 A schematic flow chart of a device information analysis and processing method in the data migration scheduling method shown.

[0045] Figure 8 It is a flow chart of a scheduling algorithm in a data migration scheduling method provided in an embodiment of the present application.

[0046] Fig. 9 It is a self-optimization principle diagram of the scheduling algorithm provided in the embodiment of the present application.

[0047] Fig.10 It is a flowchart of a method for migrating the target data from the warm storage device to the cold storage device provided in an embodiment of the present application.

[0048] Fig.11 It is a structural schematic diagram of a data migration device provided in an embodiment of the present application.

[0049] Fig.12 It is a structural schematic diagram of a data migration device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0051] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0052] The warm and cold data mentioned in the embodiments of the present application are specifically divided by the access frequency of the data. It can be understood that when the data is frequently accessed, it tends to be judged as warm data, and when the access frequency of the data decreases, it tends to be judged as cold data. Among them, the specific values ​​of the access frequency corresponding to the division of data into warm and cold may be different in different scenarios. In the embodiments of the present application, examples will be given for the warm and cold division of data. Metadata refers to data about data or data used to describe data. Metadata may include attributes of the target data (name, size, data type, etc.), and / or its structure (length, field, data column), and / or its related data (where it is located, how to contact, owner).

[0053] In existing technical solutions, the migration strategy configuration of file data relies on static configuration strategies. The operation and maintenance personnel or operation personnel formulate file warm and cold rules based on IT experience and determine the file migration strategy. For example, static migration strategies are formulated according to established rules such as file size, access time, and file name. The storage device completes the data migration action according to the established static strategy, and the migration strategy is executed once or periodically. For example Figure 1 An application scenario of a migration policy configuration is shown. The user side 110 accesses the data in the file storage device 120 through a file access storage protocol such as NFS (Network File System) / CIFS (Common Internet File System). The static migration policy is used to migrate the cold data in the file storage device 120 to the file storage device 130.

[0054] This static configuration strategy is feasible in traditional information technology management scenarios. For example, in enterprise scenarios, each enterprise is familiar with its own business, and operation and maintenance personnel can formulate migration strategies and optimal static migration algorithms based on business characteristics.

[0055] However, under the influence of the cloud trend, due to the changing business types and huge data volume, it is difficult for operation and maintenance personnel to have enough experience to understand the business data of all tenants. This will cause technical problems such as inaccurate judgment of data warmth and coldness and unreasonable data migration.

[0056] After noticing the above technical problems, the inventors of this application found through observation and analysis that in cloud scenarios, the criteria for determining data migration have evolved from a single device to more peripheral factors, such as migration network factors, device capacity and other factors. In addition, through analysis of existing scheduling algorithms, the inventors found that existing scheduling strategies focus on "short time", "system throughput" and "job type", but do not take into account the characteristics of the scheduling object, the surrounding environment and other related factors, nor do they consider how to match the scheduling algorithm with different scenarios. Therefore, the traditional data migration solution strategy configuration is driven by human experience, which will lead to inaccurate identification of warm and cold data and incorrect strategy configuration.

[0057] Based on the above, see Figure 2 , an embodiment of the present application provides an intelligent scheduling system 240. Figure 2 An application scenario of the intelligent scheduling system 240 is shown. The intelligent scheduling system 240 includes a data migration scheduling method to realize data flow driven by intelligent algorithm configuration policy. The user terminal 210 accesses the data in the file storage 220 through NFS / CIFS. The intelligent scheduling system 240 manages and configures the file storage device 220 and the file storage device 230, and determines whether the data is warm or cold based on the above-mentioned data migration scheduling method, and migrates the cold data from the file storage device 220 to the file storage device 230. The data migration scheduling method replaces the artificial policy configuration to realize the identification of warm and cold data.

[0058] A data migration scheduling method provided in an embodiment of the present application includes:

[0059] S301, obtaining metadata information of target data stored on a warm storage device, data application weight, device information of the warm storage device, and device information weight; the data application weight is used to indicate the characteristic expression of the access behavior of the user accessing the data; the device information weight is used to indicate the degree of influence of the hardware environment of the warm storage device on the data migration decision;

[0060] S302, analyzing and processing the metadata information to obtain a data application score; analyzing and processing the device information to obtain a device information score;

[0061] S303, performing weighted summation on the data application weight, the device information weight, the data application score, and the device information score to obtain a total score;

[0062] S304: When the total score meets the preset score condition, the target data is migrated from the warm storage device to the cold storage device.

[0063] The data migration scheduling method provided in the embodiment of the present application takes into account the characteristics of the target data itself and the operating status of the storage device at the same time, and realizes a technical means of judging the temperature of the data by comprehensive consideration of multiple factors, thereby increasing the accuracy of the judgment of the temperature of the data and realizing a more reasonable data migration strategy. And the characteristics of the data application weight itself determine that the data migration scheduling method also has the characteristics of self-optimization and self-adaptation. Therefore, the data migration scheduling method provided in the embodiment of the present application not only takes into account the characteristics of the target data, but also takes into account the changes in the user call behavior of the target data over time, so as to make self-optimization and provide a more reasonable data migration strategy. After comprehensively considering the data application weight, the device information weight, the data application score and the device information score, the temperature of the data is judged by comparing the total score with the preset score conditions, and a reliable judgment basis is provided for the migration of cold data from the warm storage device. The data migration scheduling method provided in the embodiment of the present application can effectively transfer the cold data in the warm storage device to other storage devices in a cloud scenario, and keep the warm data in the warm storage device, so that the user end can call the warm data from the warm storage device. Furthermore, the data migration scheduling method provided in the embodiment of the present application can migrate data to a more suitable storage device and can achieve the technical effect of reducing the storage cost of data.

[0064] In addition, the data migration scheduling method provided in the embodiment of the present application can be applied in cloud scenarios, and can also be applied to existing specific types of services, such as call record services, and an algorithm is formed by analyzing data access behavior to discover data access rules.

[0065] Specifically, in the process of implementing the data migration scheduling method provided in the embodiment of the present application, the scheduling system first obtains the metadata information of the target data and the device information of the warm storage device, and thereby obtains the data application score and device information score of the target data in the warm storage device.

[0066] The data application score reflects the impact of the target data's own attributes on the data warm or cold judgment. For example, some types of data are accessed more frequently. Therefore, the data application score can reflect the access characteristics of the data type, thereby judging such frequently accessed data as warm data and retaining it in the storage device.

[0067] The device information score also reflects the impact of the device status on the data migration strategy. For example, for the same target data, as the remaining capacity of the warm storage device increases, the target data originally determined to be warm data will be adjusted to cold data and migrated out of the warm storage device, thereby achieving the technical effect of freeing up space for the warm storage device.

[0068] Data application weights are used to indicate the characteristic expression of user access behavior. The user access behavior will change over time, and as the cumulative use time of the data migration scheduling method increases, the method can obtain more user access behaviors and give corresponding data application weights based on these user access behaviors. Therefore, the existence of data application weights can make the data migration scheduling method intelligent and self-correcting, and gradually make the judgment of data warmth and coldness more accurate over time.

[0069] The device information weight indicates the degree of influence of the hardware environment of the warm storage device on the data migration decision. For example, the higher the device information weight, the more the data migration strategy considers the device information score. The device information weight can be obtained based on practical experience analysis. The change of the device information weight depends on the different device environments and can be obtained by laboratory testing, so there is no need for online optimization.

[0070] join Figure 4 , showing an intelligent scheduling system 400 provided by an embodiment of the present application. The intelligent scheduling system 400 includes a scheduling management front desk 410, a data flow management strategy module 420, a metadata management center 430 and a data flow and status monitoring module 440.

[0071] The scheduling management front desk 410 is used to provide a scheduling management system interface, a metadata display interface, and a storage management status page, for example, to facilitate centralized management and display of the system. The core functions of the scheduling management front desk 410 may include:

[0072] (1) Unified interface display: including cross-domain data temperature distribution and global device health status;

[0073] (2) Basic configuration entry: including device access configuration, storage alarm threshold configuration, and automatic migration water level configuration. After the above configurations are successfully accessed, the data flow and status monitoring module 440 takes over each storage device.

[0074] The data flow and status monitoring module 440 manages the underlying storage devices based on the heterogeneous storage management interface, and includes the health status monitoring of the storage devices and the data flow management between different storage devices. The health data of the storage devices is also provided as parameter data to the data flow management strategy module 420. For example, in the data migration scheduling method, the device information of the warm storage device can be obtained through the data flow and status monitoring module and sent to the data flow management strategy module 420.

[0075] The storage device management structure of the data flow and status monitoring module 440 may be defined as:

[0076] (1) File system authentication;

[0077] (2) File directory level migration;

[0078] (3) Data file level migration;

[0079] (4) Data file migration;

[0080] (5) Query the data file migration results.

[0081] In the data flow and status monitoring module 440, the data flow management only executes the migration instruction initiated by the data flow management strategy module 420. The migration instruction contains a list of files that need to be migrated. The migration flow management steps are as follows:

[0082] (1) receiving a data migration instruction;

[0083] (2) Create data migration subtasks;

[0084] (3) Analyze the file list contained in the data migration instruction and find the device information to which the file belongs;

[0085] (4) Initiate a file migration instruction to the device to which the file belongs;

[0086] (5) Query the underlying device for file migration progress and results, and synchronously update the subtask status.

[0087] The metadata management center 430 can collect metadata information based on the metadata management interface to provide parameters for the algorithm of the data flow management strategy module 420. For example, in the data migration scheduling method, the metadata information of the target data can be obtained through the metadata management center 430. The metadata collection interface may include:

[0088] (1) Metadata notification;

[0089] (2) Metadata reconciliation;

[0090] (3) Metadata repair.

[0091] See also Figure 4 and Figure 5 The data flow management strategy module 420 can, for example, use the metadata information obtained by the metadata management center 430 and the device information obtained by the data flow and status monitoring module 440 as algorithm inputs, perform algorithm training and self-optimization online, and intelligently provide data flow management strategies. The data flow management strategy uses algorithm analysis to form data migration instructions.

[0092] The data flow management strategy module 420 may include a scheduling algorithm. The scheduling algorithm may be, for example, the data migration scheduling method provided in the embodiment of the present application, and include a data application score, a device information score, a data application weight, and a device information weight. The data application score, the device information weight, and the device information score of the above four factors (the factor is a multidimensional vector within the algorithm, representing multiple dimensional evaluation values) may be, for example, static factors, and a fixed value may be obtained according to scenario testing. The data application weight may be set as a dynamic factor, and has self-regulation, self-adaptation, and learning capabilities. The design of the data application weight is intended to target complex application scenarios, where the representation of application access data is different, that is, the access behavior of users accessing data is different. The algorithm forms an initial weight value through the analysis of metadata, continuously corrects the algorithm efficiency in subsequent applications, adjusts the data application warrant, and finally obtains the optimal solution to achieve accurate data warm and cold judgment and data migration.

[0093] The numerical ranges of the above four factors are, for example, within a closed interval of 1 to 10 points. Figure 5 As shown, the total score can be obtained by weighted summing the scores of the above four factors. The preset score condition can be set to a threshold value, for example. When the total score exceeds the threshold value, it is judged that the target data should be migrated from the warm storage device. Therefore, if the target data is judged to be cold data, the total score of the target data should exceed the threshold value, and the score size of the above four factors is proportional to the possibility of the target data being migrated. According to the above content, it can be imagined that when the total score is less than the threshold value, it is judged that the target data should be migrated from the storage device. In this case, the score size of the above four factors is inversely proportional to the possibility of the target data being migrated. Alternatively, it can also be thought that the preset score condition is set to a score interval, and when the total score falls into the score interval, or does not fall into the branch interval, it is judged that the target data should be migrated. The setting of the above various preset scores can be realized by a variety of combinations based on elementary functions, which cannot be exhausted here. For example, the data application score and the device information score may also have negative weights. In this case, the data application weight, the device information weight, the data application score and the device information score are still weighted sums, but the total score can be negative. Accordingly, the preset score condition can also be set in a negative range. Through different function combinations, the total score and the preset score condition can be adjusted to positive numbers under this condition.

[0094] The above-mentioned different methods for obtaining the total score and the preset score conditions can reflect the consideration of additional unforeseen factors in the data migration scheduling method, but what can still be determined is the weight relationship between the data application weight, the device information weight, the data application score and the device information score, and the independent impact of the data application score and the device information score on the total score. The following is a specific description of the acquisition of the data application weight, the device information weight, the data application score and the device information score in the embodiment of the present application, and takes the preset score condition as the threshold value. When the total score exceeds the threshold value, it is determined that the target data should be migrated from the warm storage device as an example for explanation.

[0095] In some embodiments of the present application, the metadata information includes: access frequency and data type. The metadata information is analyzed and processed to obtain a data application score, such as Figure 6 This may include:

[0096] S601, determining an access frequency score and a data type score respectively according to the access frequency and the data type;

[0097] S602: Determine a data application score according to the access frequency score and the data type score.

[0098] Therefore, the data application score takes into account the current access frequency of the target data on the one hand, and the different access behaviors of users on different types of data on the other hand. After comprehensively considering the access frequency and data type, the data application score is obtained, and thus the data application score can more accurately express the actual temperature of the target data.

[0099] Specifically, data types can express data types such as videos, pictures, and documents through different data formats. The access frequency will be adjusted as the use time of the migration scheduling method increases, and reflects the access frequency of the current target data. The data application score can be, for example, the average of the access frequency score and the data type score. For example, in the access frequency, no access for three months will get 10 points, no access for two months will get 5 points, and no access for one month will get 3 points; in the data type dimension: video type gets 10 points, picture type gets 5 points, and document type gets 3 points. Total score = (access frequency score + data type score) / 2.

[0100] In some embodiments of the present application, the device information may include occupied device capacity and device load pressure. Figure 7 , analyze and process the device information to obtain the device information score, including:

[0101] S701, determining a capacity score and a load pressure score respectively according to the occupied equipment capacity and the equipment load pressure;

[0102] S701. Determine a device information score according to a capacity score and a load pressure score.

[0103] During the operation of the warm storage device, the user side will store data in the warm storage device, and the warm storage device will obtain data from the cold storage device due to the user's access behavior. In addition, the data migration scheduling method itself will make warm and cold judgments on the data. Therefore, the available storage space of the warm storage device and the load pressure of the processor such as the CPU during operation will change over time. Therefore, the consideration of the capacity score and the load pressure score will dynamically adjust the assessment of the warmth and coldness of the target data according to the current state of the warm storage device. Thereby achieving the technical effect of freeing up space for the warm storage device and reducing the load pressure during operation.

[0104] Specifically, the calculation formula for the device information score can be, for example, the sum of the capacity score and the load pressure score and then take the average value. The capacity score can be divided, for example: 10 points for the existing capacity reaching 90% of the total capacity, 5 points for the existing capacity reaching 80% of the total capacity, 3 points for the existing capacity reaching 70% of the total capacity, and 0 points for other cases (the ratio of the existing capacity to the total capacity <70%). From this analysis, the higher the occupied capacity, the more inclined to make a decision to migrate. The device pressure score can be divided, for example: 10 points for the storage controller CPU occupancy rate less than or equal to 30%, 5 points for the storage controller CPU occupancy rate less than or equal to 60% and greater than 30%, 3 points for the storage controller CPU occupancy rate less than or equal to 70% and greater than 60%, and 0 points for the storage controller CPU occupancy rate greater than 70%. From this, it can be seen that the busier the system is, the more inclined it is not to migrate. The calculation of the device information score can be adjusted according to the actual test situation. For example, the scores of storage devices from different manufacturers are different.

[0105] In the process of determining the data application weight, it can be assumed, for example, that each file system (namespace) corresponds to an application, and the weight of the data corresponding to all applications is initially set to 10 points. Since the application weight represents the characteristic expression of the user's data access behavior, the weight score will be adjusted through a self-optimization algorithm.

[0106] Assume that the threshold value is set to 120 points, and the total score meets the preset score condition only when it is greater than the threshold value. Then, after obtaining the data application score, device information score, data application weight, and device information weight, the total score can be obtained, for example, by weighted summation, and then compared with the threshold value of 120 points. This provides a basis for data warm or cold judgment and data migration.

[0107] Since the behavior of applications (user side) accessing data is unpredictable, in the data migration scheduling method provided in the embodiment of the present application, it is necessary to finally obtain the rules of application accessing data through online analysis and optimization, so the data application weight needs to be self-optimized through an algorithm.

[0108] For example, see Figure 8 In some embodiments of the present application, after migrating the target data from the warm storage device to the cold storage device, the data migration scheduling method further includes:

[0109] S801, marking the warm-cold period of the target data, where the warm-cold period is the time period during which the target data will not be used after being migrated to the cold storage device;

[0110] S802: When a recall request for target data is received within the warm-down period, update the data application weight to reduce negative incentives for the data application weight.

[0111] The judgment of whether the target data is warm or cold is most directly reflected in the access period or access frequency of the target data. The marked warm-cold period means that when using the data migration scheduling method provided in the embodiment of the present application, the intelligent scheduling system will calibrate the access period of the target data, that is, the warm-cold period, based on past experience. If the warm-cold period is large, it means that the target data has not been accessed for a long time and should be classified as cold data and migrated out of the warm storage device. In order to correct possible errors in past experience, step S801 and step S802 can correct the warm-cold judgment of the target data by adjusting the data application weight according to the access situation of the target data.

[0112] Specifically, after analyzing the metadata information, the intelligent scheduling system determines whether the target data is warm or cold. For example, if there is no read or write operation on the target data within 3 months, it is judged as cold data, otherwise it is warm data. Then, if the target data is judged as cold data, the target data is migrated from the warm storage device to the cold storage device, and the data warm and cold period is marked as 3 months. A simple subsequent judgment method is that once the target data is in the cold storage device and is accessed and called, the intelligent scheduling system determines that the previous warm and cold judgment of the target data is wrong, and the data application weight should be negatively incentivized, and the target data is inclined to the warm data judgment. The intelligent scheduling system automatically schedules the weight ratio optimization algorithm, and obtains more positive incentives and less negative incentives through its own trial optimization. Thereby updating the data application weight, and then reducing the negative incentives for the data application weight, until the target data is repeated in the above process, and the intelligent scheduling system's judgment on the target data becomes correct.

[0113] The weight ratio optimization algorithm can be implemented, for example, through a machine learning algorithm such as a neural network. The adjustment value of the data application weight can, for example, start from an initial amplitude of 0.1, and self-adjust the amplitude of each data application weight value change through a continuous algorithm self-optimization process. In addition, different user sides or applications have different data application rules. In the data migration scheduling method provided in the embodiment of the present application, the scheduling algorithm isolates the application according to the file system namespace, that is, each namespace forms an independent weight ratio optimization algorithm by continuously verifying and adjusting the data application weight.

[0114] In some embodiments of the present application, before updating the data application weight, the data migration scheduling method further includes: determining that the recall request is an unexpected recall request.

[0115] In the process of migrating target data from cold storage devices back to warm storage devices based on recall requests, the recall request itself may be a periodic behavior from users or applications, or a recall request that has been scheduled in advance and issued at a specific time point. Therefore, this predictable recall can be defined as an expected recall request. This expected recall request can be considered as an access behavior that does not affect the judgment of data warmth or coldness. Because the target data will not be used outside of this periodic recall, it is more reasonable to judge it as cold data. Of course, this periodicity needs to be long enough, such as a periodic recall once a year. Therefore, this expected recall request will not trigger the adjustment of the data application weight, so that the data application weight reflects the data warmth or coldness more reasonably.

[0116] On the contrary, a recall request may also be an unexpected recall request, that is, for example, a user may request to recall the target data from time to time through an application. Unexpected recall requests are difficult to predict, so they need to be considered through a weight ratio optimization algorithm. Therefore, this expected judgment of recall requests can better distinguish the access behaviors of users or applications that should be considered for data warmness judgment, so that the final total score can more accurately judge the warmness of the target data.

[0117] In some embodiments of the present application, updating the data application weight includes:

[0118] The data application weight is updated by applying a weight regulator to reduce negative incentives for data application weight.

[0119] See also Fig. 9, the intelligent migration scheduling system will periodically analyze the metadata information, data application weight, device information of the warm storage device and the device information weight of the target data. Then calculate the total score and determine whether to migrate. In any of the above cycles, since the data is recalled, the data application weight of the target data needs to be re-evaluated and adjusted when necessary. And this adjustment is achieved through a weight regulator. The weight regulator can be understood as a weight regulator used for a specific type of target data obtained in advance through algorithm training. The specific type of target data can be divided, for example, according to the file system namespace. As a result, different types of target data correspond to different weight regulators, so there will be different numerical adjustments for data application weights. Therefore, the weight ratio optimization algorithm will not generalize all data, but will be trained and optimized for different types of data, so that the adjustment of data application weights is more targeted and more accurate. In addition, according to the above content, the training of the weight ratio optimization algorithm and the logical steps of the above-mentioned data migration scheduling method can be carried out separately. For example, the weight ratio optimization algorithm can be run in other devices with computing functions, and after training to a certain degree, it can be introduced into the intelligent migration scheduling system for application.

[0120] In some embodiments of the present application, see Fig.10 , migrate the target data from the warm storage device to the cold storage device, including:

[0121] S1001. Determine a migration time and a migration rate of data migration according to metadata information and device information;

[0122] S1002: At the migration time, migrate the target data from the warm storage device to the cold storage device according to the migration rate.

[0123] On the one hand, due to the update of the attributes of the target data itself (such as the update of the access frequency), as well as the change in the capacity and load pressure of the warm storage device, the warm and cold judgment of the target data at different times will be affected accordingly. On the other hand, when the target data is judged as cold data and needs to be migrated from the warm data storage device, the warm storage device may be connected to multiple cold storage devices, so the warm storage device can be migrated to the most suitable cold storage device according to the migration rate. This judgment based on the migration rate can be specifically to select the line with the fastest migration rate, or the line with the lowest cost, etc.

[0124] The data migration scheduling method provided in the embodiment of the present application can be applied to Figure 2In the scenario shown. For example, the construction cost of file storage device 220 is 1.2288 million per PB. File storage device 230 is about 45% lower than file storage device 220 due to land, electricity and labor costs. The cost per PB can be saved by 553,000 yuan, and the cost of migration dedicated line is 200,000 yuan, so migrating 100PB per year will save 35.3 million yuan. By applying this data migration scheduling method, the storage operation and management costs of cloud service providers are reduced, and it is directly reflected in the pricing of smart storage products, reducing the procurement costs of tenants who use smart storage products. The data migration scheduling method provided in the embodiment of the present application can support the reasonable layout of storage resources and can support storage management above 100P.

[0125] In summary, the data migration scheduling method provided in the embodiment of the present application can realize the intelligent identification of warm and cold data, and judge whether the data is warm or cold based on the data reading and writing rules; it enables the intelligent migration scheduling system to regularly analyze the data usage rules and intelligently mobilize data across domains, which not only meets the needs of application-free use, but also meets the needs of cross-domain low-carbon storage of data; and it enables the intelligent migration scheduling system to select the optimal scheduling time and network transmission path based on dedicated line fees, network bandwidth utilization, etc., to maximize the economic benefits of cross-domain data scheduling.

[0126] Fig.11 1 is a schematic diagram of a data migration scheduling device provided in an embodiment of the present application. The data migration device includes:

[0127] The data monitoring module 1110 is used to obtain metadata information of target data stored on the warm storage device;

[0128] A device monitoring module 1120, used to obtain device information of a warm storage device;

[0129] Migration strategy module 1130, used to record data application weight and device information weight, and to analyze and process metadata information to obtain data application score; analyze and process device information to obtain device information score; and perform weighted summation of data application weight, device information weight, data application score and device information score to obtain a total score;

[0130] Among them, the data application weight is used to indicate the characteristic expression of the user's access behavior to access data; the device information weight is used to indicate the degree of influence of the hardware environment of the warm storage device on the data migration decision;

[0131] The data scheduling module 1140 is used to migrate the target data from the warm storage device to the cold storage device when the total score meets the preset score condition.

[0132] The data migration device provided in the embodiment of the present application is a device corresponding to the above-mentioned data migration scheduling method. All implementations in the above-mentioned method embodiment are applicable to the embodiment of the device and have the same technical effects.

[0133] Fig.12 A schematic diagram of the hardware structure of a data migration scheduling device provided in an embodiment of the present application is shown.

[0134] The data migration scheduling device may include a processor 1201 and a memory 1202 storing computer program instructions.

[0135] Specifically, the processor 1201 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0136] The memory 1202 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 1202 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In one example, the memory 1202 may include a removable or non-removable (or fixed) medium, or the memory 1202 is a non-volatile solid-state memory. The memory 1202 may be inside or outside the integrated gateway disaster recovery device.

[0137] In one example, the memory 1202 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0138] The memory 1202 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0139] The processor 1201 implements the data migration scheduling method in the above embodiment by reading and executing the computer program instructions stored in the memory 1202 .

[0140] In one example, the data migration scheduling device may further include a communication interface 1203 and a bus 1204. Figure 3 As shown, the processor 1201, the memory 1202, and the communication interface 1203 are connected via a bus 1204 and communicate with each other.

[0141] The communication interface 1203 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0142] Bus 1204 includes hardware, software or both, and the components of online data flow billing equipment are coupled to each other. For example, but not limitation, the bus may include Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), Hyper Transport (HT) interconnection, Industry Standard Architecture (ISA) bus, InfiniBand interconnection, Low Pin Count (LPC) bus, Memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 1204 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnection.

[0143] In addition, in combination with the data migration scheduling method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the data migration scheduling methods in the above embodiment is implemented.

[0144] An embodiment of the present application also provides a computer program product, including a computer program, which implements any one of the data migration scheduling methods in the above embodiments when the computer program is processed and executed.

[0145] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0146] The functional blocks shown in the above structural block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or communication link by a data signal carried in a carrier. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (Read-Only Memory, ROM), flash memory, erasable read-only memory (Erasable ReadOnly Memory, EROM), floppy disks, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical discs, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0147] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0148] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0149] The above are only specific implementation methods of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. A data migration scheduling method, characterized in that: include: Obtain metadata information of target data stored on a warm storage device, data application weight, device information of the warm storage device, and device information weight; the data application weight is used to indicate a characteristic expression of an access behavior of a user accessing data; The device information weight is used to indicate the degree of influence of the hardware environment of the warm storage device on the data migration decision; Analyzing and processing the metadata information to obtain a data application score; Analyzing and processing the device information to obtain a device information score; Performing a weighted summation on the data application weight, the device information weight, the data application score and the device information score to obtain a total score; When the total score satisfies a preset score condition, the target data is migrated from the warm storage device to the cold storage device.

2. The data migration scheduling method according to claim 1, characterized in that: The metadata information includes: access frequency and data type, The step of analyzing and processing the metadata information to obtain a data application score includes: Determine an access frequency score and a data type score respectively according to the access frequency and the data type; A data application score is determined according to the access frequency score and the data type score.

3. The data migration scheduling method according to claim 1, characterized in that: The equipment information includes occupied equipment capacity and equipment load pressure; the equipment information is analyzed and processed to obtain an equipment information score, including: Determining a capacity score and a load pressure score, respectively, according to the occupied equipment capacity and the equipment load pressure; An equipment information score is determined based on the capacity score and the load pressure score.

4. The data migration scheduling method according to any one of claims 1 to 3, characterized in that: After migrating the target data from the warm storage device to the cold storage device, the method further includes: Marking the warm-cold period of the target data, where the warm-cold period is a time period during which it is determined that the target data will not be used after being migrated to the cold storage device; When a recall request for the target data is received within the warm-down period, the data application weight is updated to reduce negative incentives for the data application weight.

5. The data migration scheduling method according to claim 4, characterized in that: Before updating the data application weight, the method further includes: The recall request is determined to be an unexpected recall request.

6. The data migration scheduling method according to claim 4, characterized in that: The updating of the data application weight comprises: The data application weight is updated by applying a weight adjuster to reduce negative incentives for the data application weight.

7. The data migration scheduling method according to any one of claims 1 to 3, characterized in that: Migrating the target data from the warm storage device to the cold storage device includes: Determine a migration time and a migration rate of the data migration according to the metadata information and the device information; At the migration time, the target data is migrated from the warm storage device to the cold storage device according to the migration rate.

8. A data migration scheduling device, characterized in that: The device comprises: A data monitoring module, used to obtain metadata information of target data stored on the warm storage device; Device monitoring module, used to obtain device information of warm storage devices; a migration strategy module, for recording data application weights and device information weights, and for analyzing and processing the metadata information to obtain a data application score; and analyzing and processing the device information to obtain a device information score; and performing weighted summation of the data application weights, device information weights, data application scores and device information scores to obtain a total score; The data application weight is used to indicate the characteristic expression of the user's access behavior to data; the device information weight is used to indicate the degree of influence of the hardware environment of the warm storage device on the data migration decision; The data scheduling module is used to migrate the target data from the warm storage device to the cold storage device when the total score meets a preset score condition.

9. A data migration scheduling device, characterized in that: The device includes: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the data migration scheduling method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer storage medium stores computer program instructions, and when the computer program instructions are executed by the processor, the data migration scheduling method according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data migration scheduling method as described in any one of claims 1 to 7.

Images