A method and device for managing data hot and cold lifecycle

Abstract

The present invention relates to the field of computer technology, and provides a method and device for managing the hot and cold lifecycle of data, wherein the method comprises: constructing a temperature prediction model; obtaining the predicted temperature of each group of data according to the temperature prediction model; making decisions on the temperature of the data according to the predicted temperature of each group of data, and constructing a scheduling strategy model with the goal of optimizing the comprehensive performance of the system; solving the scheduling strategy model to obtain the optimal scheduling strategy, scheduling data according to the optimal scheduling strategy, and calculating the optimal temperature of each group of data; reversely optimizing the temperature prediction model according to the optimal temperature of each group of data to optimize the results of subsequent temperature prediction. The present invention reversely optimizes the temperature prediction model through scheduling, thereby reducing the impact of poor quality data on the temperature prediction model, ensuring the accuracy of data hot and cold separation, and making the final temperature prediction result provide a higher reference value for the scheduling of the system.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for managing data hot and cold lifecycles. Background Art

[0002] The data stored in the database is usually divided into hot and cold data. We believe that frequently accessed data is called hot data, while in contrast, data that is infrequently accessed or even not accessed for a period of time is called cold data. Therefore, data can be stored on different media according to the hotness or coldness of the data, thereby minimizing storage costs or achieving higher overall performance while ensuring system performance. How to evaluate the hotness or coldness of data (determine the temperature of the data) has always been a hot topic in the field of database research.

[0003] The most direct measure of the hotness or coldness of data is the access probability of the data in the future. That is, data with a high probability of being accessed in the future is hot data, while data with a low probability of being accessed or no longer accessed is cold data. In practical applications, it is impossible to predict the access situation of data in the future, so other measurement methods are usually developed in the database in order to "predict" the access situation in the future, and then express the hotness or coldness of the data.

[0004] In the prior art, the separation of hot and cold data is usually achieved by using large quantities of labeled data for machine learning. Specifically, experts manually label the hot and cold data in advance, and then use the labeled data to train the corresponding machine learning algorithm, and use the training results to separate the hot and cold data of the actual data. In this implementation method, the accuracy of the labeled data will directly affect the effect of the hot and cold data separation. When the labeled data is of poor quality, the final effect of the hot and cold data separation is not satisfactory. In addition, the classification of hot and cold data in the prior art only considers the access frequency of the data, and the type of data storage area, cost, and even the performance characteristics of the storage medium are not taken into consideration. As a result, the hot and cold data obtained by the final separation cannot be combined with the actual storage environment, and cannot accurately reflect the reference value for data storage scheduling based on the hot and cold data.

[0005] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the field of this technology. Summary of the invention

[0006] The technical problem to be solved by the present invention is that the quality of the labeled data directly affects the effect of data hot and cold separation. When the labeled data is of poor quality, the effect of data hot and cold separation is poor.

[0007] The present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a method for managing hot and cold data lifecycles, comprising:

[0009] Using the labeled hot and cold data training set, feature learning is performed to build a temperature prediction model; based on the temperature prediction model, temperature prediction is performed on each group of data in the system to obtain the predicted temperature of each group of data;

[0010] According to the predicted temperature of each group of data, the temperature variation range corresponding to each group of data is determined, and within the corresponding temperature variation range, a decision is made on the temperature of the data, and a scheduling strategy model with the goal of optimizing the comprehensive performance of the system is constructed; the scheduling strategy model is solved to obtain an optimal scheduling strategy, and data scheduling is performed according to the optimal scheduling strategy;

[0011] Wherein, in each simulation scheduling of solving the scheduling strategy model, a temperature decision is made for each group of data to obtain the decision temperature corresponding to each group of data, and the current storage medium of each group of data is used as the storage medium before scheduling, and according to the temperature range where the decision temperature of each group of data is located, the storage medium after scheduling of each group of data is determined, so as to perform simulation scheduling; the temperature range is obtained by dividing according to the decision temperature of each group of data in the last simulation scheduling, and each temperature range corresponds to a storage medium; in the first simulation scheduling, the temperature range is obtained by dividing according to the predicted temperature of each group of data;

[0012] According to the comprehensive performance changes of the system before and after data scheduling, the optimal temperature of each group of data is calculated; according to the optimal temperature of each group of data, the temperature prediction model is reversely optimized to optimize the results of subsequent temperature prediction.

[0013] Preferably, the use of the labeled hot and cold data training set to perform feature learning to build a temperature prediction model specifically includes:

[0014] Each feature vector in the cold and hot data training set is normalized, and the normalized feature vector is input into a Bert-based bidirectional transformer network for feature learning to construct a temperature prediction model.

[0015] Preferably, the method of using the labeled hot and cold data training set to perform feature learning to construct a temperature prediction model also includes optimizing the temperature prediction model using a labeled validation data set, specifically including:

[0016] Inputting the labeled data in the labeled verification data set into the temperature prediction model to obtain the predicted temperature of the labeled data;

[0017] According to the labeled temperature of the labeled data and the predicted temperature of the labeled data, the classifier is optimized using the cross entropy loss, and the cross entropy loss is calculated;

[0018] According to the cross entropy loss, the temperature prediction model is optimized using a gradient descent algorithm.

[0019] Preferably, the construction of a scheduling strategy model with the goal of optimizing the overall performance of the system specifically includes:

[0020] Based on the temperature of each group of data, the size of the space occupied by the data, the length of time the data is stored, the value of the data, the storage cost of the storage medium where each group of data is located, and the access rate of each group of data in the corresponding storage medium, an objective function is generated with the goal of maximizing the overall performance of the system;

[0021] Generate a constraint function based on the size of the space occupied by each group of data, the storage space capacity of each storage medium, the storage cost of the system before scheduling, the storage cost of the system after scheduling, and the total scheduling cost;

[0022] The scheduling strategy model is established by combining the objective function and the constraint function.

[0023] Preferably, the constraint function includes a storage space constraint function and a scheduling cost constraint function;

[0024] The storage space constraint function is specifically:

[0025] For each storage medium, the space occupied by the data group stored in the storage medium is not greater than the actual capacity of the storage medium;

[0026] The scheduling cost constraint function is specifically:

[0027] The sum of the total storage cost of the system after scheduling and the scheduling cost is less than the total storage cost of the system before scheduling.

[0028] Preferably, the objective function is specifically:

[0029] The total storage cost of the system plus the total scheduling cost of the system is the total cost of the system, which is obtained by multiplying the system performance score by the target coefficient k and then dividing it by the total cost of the system;

[0030] The total storage cost of the system is specifically obtained by multiplying the actual occupied space of the i-th group of data by the storage cost of the storage medium where the i-th group of data is located to obtain a first value of the i-th group of data;

[0031] Subtract the storage value of the i-th group of data from the first value of the i-th group of data to obtain the remaining storage value of the i-th group of data, and perform a logarithmic operation with the remaining storage value of the i-th group of data as a true number and the natural number e as the base to obtain the second value of the i-th group of data;

[0032] The storage cost of the i-th group of data is obtained by multiplying the second value of the i-th group of data by the storage duration of the i-th group of data, and the total storage cost of the system is obtained by adding the storage cost of each group of data;

[0033] The performance score of the system is specifically:

[0034] The temperature of the i-th group of data is multiplied by the actual access rate of the i-th group of data, and the resultant is divided by the actual occupied space of the i-th group of data to obtain a third value of the i-th group of data;

[0035] Adding the third values ​​of each set of data to obtain a fourth value, and dividing the storage space capacity by the fourth value to obtain a performance score of the system;

[0036] The total dispatch cost is specifically:

[0037] The scheduling cost of each set of data is added up to get the total scheduling cost of the system.

[0038] Preferably, the actual access rate of the i-th group of data and the storage cost of the i-th group of data are determined according to the temperature of the i-th group of data, specifically including:

[0039] Determining a storage medium for the i-th group of data according to a temperature range corresponding to the temperature of the i-th group of data;

[0040] The storage cost of the i-th group of data is obtained by multiplying the unit value of the storage medium of the i-th group of data, the storage duration of the i-th group of data, and the actual occupied space of the i-th group of data;

[0041] The actual access rate of the storage medium of the i-th group of data is used as the actual access rate of the i-th group of data.

[0042] Preferably, the step of calculating the optimal temperature of each set of data according to the comprehensive performance change of the system before and after the data scheduling specifically includes:

[0043] The total storage cost of the system plus the total scheduling cost of the system is used to obtain the total cost of the system. The total cost of the system is divided by the coefficient k and then multiplied by the comprehensive performance difference Δtarget of the system before and after scheduling to obtain the performance score difference before and after the system scheduling.

[0044] Multiply the system storage space capacity by the actual access rate of the i-th group of data, and then divide it by the actual occupied space of the i-th group of data to obtain the fifth value of the i-th group of data, calculate the partial derivative of the performance score difference with respect to the fifth value, and add the partial derivative to the pre-scheduling temperature of the i-th group of data to obtain the optimal temperature of the i-th group of data.

[0045] Preferably, the reverse optimization of the temperature prediction model according to the optimal temperature of each set of data specifically includes:

[0046] According to the optimal temperature, the classifier is optimized using the cross entropy loss to calculate the cross entropy loss;

[0047] According to the cross entropy loss, the temperature prediction model is optimized using a gradient descent algorithm.

[0048] In a second aspect, the present invention further provides a device for managing data hot and cold lifecycles, which is used to implement the method for managing data hot and cold lifecycles in the first aspect, and the device includes:

[0049] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the method for data hot and cold lifecycle management described in the first aspect.

[0050] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the method for data hot and cold lifecycle management described in the first aspect.

[0051] After obtaining the optimal scheduling strategy through the scheduling strategy model, the present invention reversely calculates the optimal temperature of the data according to the scheduling strategy, and optimizes the temperature prediction model according to the optimal temperature, so that the temperature prediction effect of the optimized temperature prediction model is closer to the optimal temperature, thereby making the temperature prediction model strongly correlated with the data scheduling of the system. Even if the labeled data in the cold and hot data training set used when the temperature prediction model was initially established is poor quality data, in the present invention, it can still be reversely optimized through scheduling to keep the temperature prediction model accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 It is a flowchart of a method for managing hot and cold data lifecycles provided by an embodiment of the present invention;

[0054] Figure 2 It is a schematic diagram of the architecture of a bidirectional transformer network in a method for managing hot and cold data lifecycles provided by an embodiment of the present invention;

[0055] Figure 3 It is a flowchart of a method for managing hot and cold data lifecycles provided by an embodiment of the present invention;

[0056] Figure 4 It is a flowchart of a method for managing hot and cold data lifecycles provided by an embodiment of the present invention;

[0057] Figure 5 It is a schematic diagram of the architecture of a PSO algorithm in a method for managing hot and cold data lifecycles provided by an embodiment of the present invention;

[0058] Figure 6 It is a schematic diagram of the overall architecture of a method for managing hot and cold data lifecycles provided by an embodiment of the present invention;

[0059] Figure 7 It is a schematic diagram of the architecture of a device for managing hot and cold data lifecycles provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0061] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0062] Embodiment 1:

[0063] In the prior art, the quality of the labeled data directly affects the effect of data hot and cold separation. When the labeled data is of poor quality, the effect of data hot and cold separation is poor, resulting in unsatisfactory effect of data scheduling based on data hot and cold separation. In order to solve this problem, embodiment 1 of the present invention provides a method for data hot and cold lifecycle management, such as Figure 1 As shown, including:

[0064] In step 201, feature learning is performed using the labeled hot and cold data training set to construct a temperature prediction model; temperature prediction is performed on each group of data in the system according to the temperature prediction model to obtain a predicted temperature of each group of data.

[0065] Among them, multiple data that are inseparable and stored in the system are divided into the same group of data, thereby forming the groups of data, and the feature learning and temperature prediction are mainly performed based on data access conditions, data producer status, data consumer status and other data.

[0066] The quality of the labeled data in the labeled hot and cold data training set directly depends on the level of the expert.

[0067] In step 202, the temperature variation range corresponding to each group of data is determined based on the predicted temperature of each group of data, a decision is made on the temperature of the data within the corresponding temperature variation range, and a scheduling strategy model is constructed with the goal of optimizing the overall performance of the system.

[0068] In step 203, the scheduling strategy model is solved to obtain an optimal scheduling strategy, and data scheduling is performed according to the optimal scheduling strategy.

[0069] Among them, in each simulation scheduling of solving the scheduling strategy model, a temperature decision is made for each group of data to obtain the decision temperature corresponding to each group of data, and the current storage medium of each group of data is used as the storage medium before scheduling, and according to the temperature range where the decision temperature of each group of data is located, the storage medium after scheduling of each group of data is determined, thereby performing simulation scheduling; the temperature range is obtained according to the decision temperature of each group of data in the last simulation scheduling, and each temperature range corresponds to a storage medium; in the first simulation scheduling, the temperature range is obtained according to the predicted temperature of each group of data.

[0070] The scheduling strategy model is mainly established based on the temperature of each group of data, the data value of each group of data, the storage duration, the size of the occupied space, and the storage characteristics of each storage medium in the system.

[0071] The process of solving the scheduling strategy model can be regarded as a process of performing multiple simulation scheduling, that is, in one simulation scheduling, it is assumed that the temperature of the data is a certain temperature, and the data is simulated and scheduled according to the temperature to obtain the simulated comprehensive performance of the system after the simulation scheduling, and the results of multiple simulation scheduling are compared to obtain the optimal system simulation comprehensive performance. The scheduling strategy corresponding to the optimal system simulation comprehensive performance is the optimal scheduling strategy. The process of multiple simulation scheduling can also be regarded as a continuous iteration process, that is, according to the decision temperature in the previous simulation scheduling, the decision temperature in the next simulation scheduling is determined. When the simulation scheduling is performed for the first time, the corresponding decision temperature is obtained according to the predicted temperature. In multiple simulation scheduling, the decision temperature does not exceed the temperature variation range, so that when the optimized temperature is obtained, a large deviation from the temperature prediction model is avoided. That is, the obtained optimized temperature is affected by both the temperature prediction model and the scheduling strategy model.

[0072] And the temperature intervals are divided according to the decision temperature in the last simulation scheduling. The division of each temperature interval is determined by the size of the storage medium and the size of the data at each temperature. First, the initial range of each temperature interval is divided proportionally according to the size of each storage medium. Then, according to the decision temperature of each group of data, the size of the temperature interval is dynamically adjusted so that the storage medium corresponding to the temperature interval can store all the data with the decision temperature within the temperature interval as completely as possible.

[0073] It should be noted here that the “last simulation scheduling” and “next simulation scheduling” described in this embodiment are relative to two adjacent comparison processes. For example, three simulation schedulings have been carried out by a certain time. For the convenience of description, these three simulation schedulings are called in chronological order: the first simulation scheduling, the second simulation scheduling and the third simulation scheduling. The first simulation scheduling is the “previous simulation scheduling” of the second simulation scheduling, the second simulation scheduling is the “next simulation scheduling” of the first simulation scheduling, the second simulation scheduling is the “previous simulation scheduling” of the third simulation scheduling, and the third simulation scheduling is the “next simulation scheduling” of the second simulation scheduling.

[0074] The temperature of each group of data determines in which storage medium the data is stored. For example, multiple temperature intervals are divided, each temperature interval corresponds to a storage medium, and the data with a temperature within the corresponding temperature interval is stored in the corresponding storage medium.

[0075] In step 204, the optimal temperature of each group of data is calculated according to the comprehensive performance change of the system before and after the data scheduling; based on the optimal temperature of each group of data, the temperature prediction model is reversely optimized to optimize the subsequent temperature prediction results.

[0076] The above-mentioned calculation of the optimal temperature of each group of data based on the comprehensive performance change of the system before and after data scheduling is actually to obtain the optimal temperature of the data under the optimal scheduling strategy after the optimal scheduling strategy is obtained in step 203, and then inversely calculate.

[0077] The result of optimizing the subsequent temperature prediction can be iteratively executing the above steps 201-203 until the number of iterations is greater than the preset number, or the iterative stop condition is set by a technician in this field, and the iterative stop condition is reached. It is considered that after the iteration, an accurate temperature prediction model is obtained, and the predicted temperature obtained by using the temperature prediction model is the most accurate data temperature under the current system, so that data scheduling can be performed directly according to the predicted temperature without formulating a scheduling strategy through a scheduling decision model. The preset number of times is obtained by a technician in this field based on empirical analysis. Among them, the optimal temperature is the optimal temperature obtained in one iteration process. As the number of iterations increases, the optimal temperature obtained will continue to approach the accurate temperature of the data in the current system.

[0078] In this embodiment, a temperature prediction model and a scheduling strategy model are set, and after obtaining the optimal scheduling strategy through the scheduling strategy model, the optimal temperature of the data is obtained by reverse calculation according to the scheduling strategy, and the temperature prediction model is optimized according to the optimal temperature, so that the temperature prediction effect of the optimized temperature prediction model is closer to the optimal temperature, thereby making the temperature prediction model strongly related to the data scheduling of the system. Even if the labeled data in the cold and hot data training set used when the temperature prediction model was initially established is poor quality data, in this embodiment, it can still be reverse optimized through scheduling to keep the temperature prediction model accurate. In this embodiment, since scheduling is used to reversely optimize the temperature prediction model, the data temperature predicted by the optimized temperature prediction model has a higher reference value for data scheduling, so that it can be applied to a variety of systems without the need to formulate a separate scheduling method for each system.

[0079] This embodiment also provides an optional implementation method, that is, using the labeled cold and hot data training set to perform feature learning to build a temperature prediction model, specifically including:

[0080] Each feature vector in the cold and hot data training set is normalized, and the normalized feature vector is input into a Bert-based bidirectional transformer network for feature learning to construct a temperature prediction model.

[0081] like Figure 2 Schematic diagram of the architecture of the bidirectional transformer network, where I0, I1, and I n are the feature vectors input into the network.

[0082] It should be noted here that the above implementation is only an optional implementation. In actual use, it is also feasible to use the neural network algorithm model LSTM or a custom model of technicians in this field to build a temperature prediction model.

[0083] In actual use, the effect of feature learning may vary depending on the amount of data in the cold and hot data training set, which may ultimately lead to an inaccurate temperature prediction model. To address this problem, this embodiment also provides the following preferred implementation, namely, using the labeled cold and hot data training set to perform feature learning to construct a temperature prediction model, and also includes using the labeled verification data set to optimize the temperature prediction model, such as Figure 3 As shown, specifically including:

[0084] In step 301, the labeled data in the labeled verification data set is input into the temperature prediction model to obtain the predicted temperature of the labeled data.

[0085] In step 302, a cross entropy loss is calculated by optimizing a classifier using the cross entropy loss according to the labeled temperature of the labeled data and the predicted temperature of the labeled data.

[0086] In step 303, the temperature prediction model is optimized using a gradient descent algorithm according to the cross entropy loss.

[0087] The cross entropy loss is used to optimize the classifier, and the cross entropy loss is calculated to be expressed in the form of a mathematical function:

[0088]

[0089] Among them, Loss is the cross entropy loss, i is the number of data groups, c represents the temperature of a type of data, m is the number of temperature categories, N is the number of feature vectors used in the input feature learning, and Figure 2 For example, if n eigenvectors are input, the value of N is n. ic The temperature prediction model predicts the probability that the predicted temperature of the i-th group of data is c, and its value range is [0, 1], y ic Indicates whether the marked temperature of the i-th group of data is temperature c. When the marked temperature of the i-th group of data is c, y ic The value of y is 1, otherwise, ic The value of is 0.

[0090] The use of the gradient descent algorithm to optimize the temperature prediction model can be understood as changing and optimizing the model parameters in the temperature prediction model so that the data temperature predicted by the optimized temperature prediction model is more accurate. The accuracy here is still relative to the labeled data, even if the temperature predicted by the temperature prediction model is closer to the labeled temperature.

[0091] This embodiment also provides an optional implementation method, that is, constructing a scheduling strategy model with the goal of optimizing the overall performance of the system, such as Figure 4 As shown, specifically including:

[0092] In step 401, an objective function aiming at maximizing the overall performance of the system is generated based on the temperature of each group of data, the size of the space occupied by the data, the data storage duration, the data value, the storage cost of the storage medium where each group of data is located, and the access rate of the storage medium where each group of data is located.

[0093] In step 402, a constraint function is generated according to the size of the space occupied by each group of data, the storage space capacity of each storage medium, the storage cost of the system before scheduling, the storage cost of the system after scheduling, and the total scheduling cost.

[0094] In step 403, the scheduling strategy model is established by combining the objective function and the constraint function.

[0095] The comprehensive performance of the system is partially represented by the system storage cost and data scheduling cost, and partially represented by the system data access performance. For example, when data with high data value and frequent calls are stored in a storage medium with relatively good performance, and data with low data value and few calls are stored in a storage medium with relatively poor performance, the comprehensive performance of the system is relatively high.

[0096] There are a variety of optional algorithms for establishing the scheduling strategy model, including genetic algorithm GA, particle swarm algorithm PSO and differential evolution algorithm DE.

[0097] In this embodiment, the use of the particle swarm algorithm PSO to establish the scheduling strategy model is taken as an example to specifically explain the objective function and constraint function in the model.

[0098] When using the particle swarm algorithm PSO, the objective function is specifically:

[0099] The total storage cost storage of the system plus the total dispatching cost dispatch of the system get the total cost of the system, which is obtained by multiplying the system's performance score performance by the target coefficient k and then dividing it by the total cost of the system. Among them, the total storage cost storage of the system refers to the total storage cost of the system after the simulated scheduling, and the performance score performance of the system refers to the performance score of the system after the simulated scheduling. That is, both of the above are calculated based on the state of the system after the simulated scheduling. The storage medium where each group of data involved refers to the corresponding post-scheduling medium in this simulated scheduling, and the temperature of each group of data refers to the corresponding decision temperature in this simulated scheduling. The total dispatching cost dispatch of the system refers to the scheduling cost consumed in the simulated scheduling process, which is usually manifested in the time consumed by data migration, memory occupancy, and CPU occupancy.

[0100] The objective function is expressed in the form of a mathematical formula:

[0101]

[0102] Among them, target can be regarded as the comprehensive performance of the system, performance is the performance score of the system, storage is the total storage cost of the system, dispatch is the total scheduling cost of the system, k is the target coefficient, and the value of k is obtained by technical personnel in this field according to the system demand analysis. The larger the value of k, the more the scheduling strategy model focuses on improving the performance score of the system, and the smaller the value of k, the more the scheduling strategy model focuses on balancing the storage cost of the system.

[0103] In the process of solving the scheduling strategy model, each simulated scheduling requires the calculation of the objective function, and the targets of multiple simulated scheduling are used as the basis for judging whether the simulated scheduling is optimal. Storage is the total storage cost of the system after the simulated scheduling, performance is the performance score of the system after the simulated scheduling, and dispatch is the total scheduling cost required for the simulated scheduling.

[0104] The total storage cost storage of the system is specifically: a first value of the i-th group of data is obtained by multiplying the actual occupied space of the i-th group of data by the storage cost of the storage medium where the i-th group of data is located.

[0105] Use the first value of the i-th group of data to subtract the storage value of the i-th group of data to obtain the remaining storage value of the i-th group of data. Take the remaining storage value of the i-th group of data as the true number and the natural number e as the base to perform a logarithmic operation to obtain the second value.

[0106] The storage cost of the i-th group of data is obtained by multiplying the second value of the i-th group of data by the storage duration of the i-th group of data, and the total storage cost storage of the system is obtained by adding the storage cost of each group of data.

[0107] The total storage cost of the system is expressed in the form of a mathematical formula:

[0108]

[0109] Among them, storage is the total storage cost of the system, i is the number of data groups, n is the total number of data groups in the system, and duration i is the storage duration of the i-th group of data, footprint i is the actual space occupied by the i-th group of data, cost i is the storage cost of the storage medium where the i-th group of data is located, worth i is the data value of the i-th group of data.

[0110] The performance score of the system is as follows:

[0111] The temperature of the i-th group of data and the actual access rate of the i-th group of data are multiplied, and the resultant value is divided by the actual occupied space of the i-th group of data to obtain a third value of the i-th group of data.

[0112] The third values ​​of each set of data are added together to obtain a fourth value, and the storage space capacity is divided by the fourth value to obtain the system performance score performance.

[0113] The performance score of the system is expressed in the form of a mathematical formula:

[0114]

[0115] Among them, performance is the performance score of the system, capacity is the total storage capacity of the system, i is the number of data groups, n is the total number of data groups in the system, temperature i is the temperature of the i-th group of data, rate i is the actual access rate of the data, footprint i is the actual space occupied by the i-th group of data.

[0116] It should be noted that the first value, the second value, the third value and the fourth value are used to clearly show the calculation process of the objective function, and are used to represent different intermediate values ​​used in the calculation process. Among them, the first value, the second value and the third value are all relative to a set of data, that is, for a set of data, in a simulation scheduling process, the corresponding first value, second value and third value are calculated. The fourth value is relative to a simulation scheduling process as a whole, that is, in a simulation scheduling process, a fourth value corresponds.

[0117] The total dispatch cost dispatch is specifically:

[0118] Add up the scheduling costs of each set of data to get the total scheduling cost of the system dispatch.

[0119] The total dispatch cost dispatch is expressed in the form of a mathematical formula:

[0120]

[0121] Where i is the count value of the data to be scheduled, n is the total number of data groups in the system, and the disfootprint i is the scheduling cost of the i-th group of data to be scheduled. If the i-th group of data needs to be scheduled, its scheduling cost disfootprint i The value is 1. If the i-th group of data does not need to be scheduled, then disfootprint i The value is 0.

[0122] In the process of solving the scheduling strategy model, the temperature of the data is used as a decision variable, that is, multiple simulation scheduling is performed by using different temperatures. In each scheduling, the storage medium of the corresponding data after scheduling may be different according to the different temperatures of the data, so there is the following correlation relationship. The actual access rate of the i-th group of data and the storage cost of the i-th group of data are determined according to the temperature of the i-th group of data, specifically including:

[0123] The storage medium of the i-th group of data is determined according to the temperature range corresponding to the temperature of the i-th group of data.

[0124] The storage cost of the i-th group of data is obtained by multiplying the unit value of the storage medium of the i-th group of data, the storage duration of the i-th group of data and the actual occupied space of the i-th group of data; the actual access rate of the storage medium of the i-th group of data is used as the actual access rate of the i-th group of data.

[0125] Assuming that the temperature intervals are divided according to the data sets in the last simulation scheduling, the temperature interval corresponding to the kth storage medium is obtained as follows:

[0126] [temperature min (k), temperature max (k)]

[0127] When the temperature of the i-th group of data meets the following conditions in a certain simulation scheduling:

[0128] temperature i ∈[temperature min (k),temperature max (k)]

[0129] In this simulation scheduling, the storage medium after scheduling of the i-th group of data is the k-th storage medium. In this simulation scheduling, that is, after the data simulation scheduling, the i-th group of data will be stored in the k-th storage medium. Then in this simulation scheduling, the storage cost of the storage medium where the i-th group of data is located is i for:

[0130] cost i =footprint i ×price k ×time

[0131] Among them, footprint i is the actual space occupied by the i-th group of data, price k is the unit value of the kth storage medium, which is obtained by those skilled in the art based on the characteristics of the storage medium. time is the storage duration of the i-th group of data, which can also be understood as duration in the above implementation. i .

[0132] The actual access rate of the i-th group of data i is the actual access rate of the kth storage medium.

[0133] The constraint functions include a storage space constraint function and a scheduling cost constraint function.

[0134] The storage space constraint function is specifically:

[0135] For each storage medium, the space occupied by the data group stored in the storage medium is not greater than the actual capacity of the storage medium.

[0136] The space occupied by the data group stored in the kth storage medium is finalfootprint k for:

[0137]

[0138] Among them, temperature i is the temperature of the i-th group of data, footprint i is the actual space occupied by the i-th group of data, temperature i ∈[temperature min (k),temperature max (k)] represents the temperature interval corresponding to the kth storage medium. The actual occupied space of all data with temperatures within this interval is added to obtain the final footprint k , if the actual capacity of the kth storage medium is truefootprint k , then for any k, the following must be satisfied:

[0139] finalfootprint k ≤truefootprint k

[0140] The scheduling cost constraint function is specifically:

[0141] The sum of the total storage cost of the system after scheduling and the scheduling cost is less than the total storage cost of the system before scheduling.

[0142] The total storage cost of the system before scheduling is storage bef , the total storage cost of the system after scheduling is storage aft , the dispatch cost is dispatch, see the above formula for details, and the estimated data storage time is time. When optimizing, set the optimization goal as:

[0143] max(|storage aft ×time-(storage bef ×time+dispatch)|)

[0144] Among them, max() represents the maximization function.

[0145] When the particle swarm algorithm PSO is used to construct the scheduling strategy model and obtain the above objective function, the following optional implementation methods exist correspondingly, that is, the optimal temperature of each group of data is calculated according to the comprehensive performance change of the system before and after data scheduling, specifically including:

[0146] The total storage cost of the system storage plus the total dispatching cost of the system dispatch is used to get the total cost of the system. The total cost of the system is divided by the coefficient k and then multiplied by the comprehensive performance difference Δtarget of the system before and after scheduling to get the performance score difference Δpreformance before and after the system scheduling.

[0147] The performance score difference Δpreformance is expressed in the form of a mathematical formula:

[0148]

[0149] Among them, Δpreformance is the performance score difference, storage is the total storage cost of the system after scheduling, dispatch is the total scheduling cost required for scheduling, k is the target coefficient, and Δtarget is the comprehensive performance difference of the system before and after scheduling.

[0150] The comprehensive performance difference Δtarget of the system before and after the scheduling is expressed in the form of a mathematical formula:

[0151] Δtarget=|target-target′|

[0152] Among them, target is the comprehensive performance of the system before scheduling, and target′ is the comprehensive performance of the system after scheduling.

[0153] Multiply the system storage space capacity by the actual access rate of the i-th group of data, and then divide it by the actual occupied space of the i-th group of data to obtain the fifth value of the i-th group of data, calculate the partial derivative of the performance score difference Δpreformance with respect to the fifth value, and add the partial derivative to the pre-scheduling temperature of the i-th group of data to obtain the optimal temperature of the i-th group of data i ′.

[0154] The optimal temperature i ′ is expressed in the form of mathematical formula:

[0155]

[0156] Among them, temperature i is the predicted temperature of the i-th group of data, Δpreformance is the difference in system performance score after scheduling, capacity is the total storage capacity of the system, rate i is the read and write rate of the i-th group of data in the storage medium after scheduling, footprint i is the actual space occupied by the i-th group of data.

[0157] It should be noted here that the fifth numerical value is used to clearly express the temperature value calculation process and is used to represent the intermediate value used in the calculation process. The fifth numerical value is relative to a set of data, that is, for a set of data, there is one fifth numerical value.

[0158] This implementation is only applicable to the case where the above-mentioned objective function is obtained by the PSO algorithm. When the objective function is different, the corresponding implementation method for calculating the optimal temperature is also different. For example, when the genetic algorithm GA is used to construct the objective function, the corresponding calculation method of the optimal temperature is different from this implementation.

[0159] As an optional implementation, this embodiment uses a cross entropy loss optimization classifier to optimize the temperature prediction model, that is, according to the optimal temperature of each group of data, the temperature prediction model is reversely optimized, specifically including:

[0160] According to the optimal temperature, the classifier is optimized using the cross entropy loss, and the cross entropy loss is calculated.

[0161] According to the cross entropy loss, the temperature prediction model is optimized using a gradient descent algorithm.

[0162] The cross entropy loss is used to optimize the classifier, and the cross entropy loss is calculated to be expressed in the form of a mathematical function:

[0163]

[0164] Among them, Loss is the cross entropy loss, i is the number of data groups, temperature i ′ is the optimized temperature of the i-th group of data, temperature i is the predicted temperature of the i-th group of data. N is the number of feature vectors used in the input feature learning.

[0165] This embodiment also proposes a preferred implementation mode based on the above implementation modes, taking an actual usage scenario as an example, specifically including:

[0166] Each preset period is used to collect characteristic information of corresponding data from the system. The preset period is obtained by those skilled in the art based on empirical analysis, such as collecting characteristic information of IoT data in the storage area at intervals of one to two days between 1:00 and 3:00 a.m. every day.

[0167] The characteristic information is divided into three categories, including characteristic information related to data access conditions, characteristic information related to the cost of storing data, and characteristic information related to data producers and consumers.

[0168] The characteristic information related to the data access situation includes:

[0169] (1) Data recent access frequency data_frequency

[0170] (2) Data access speed data_rate

[0171] The characteristic information related to the storage data cost includes:

[0172] (1) Data storage space size (i.e., the actual occupied space in the above embodiment) storage_footprint

[0173] (2) Data storage duration storage_duration

[0174] (3) Data value data_worth

[0175] (4) Storage area cost storage_cost

[0176] The characteristic information related to data producers and consumers includes:

[0177] (1) Producer: device online status device_status, device recent online duration device_duration

[0178] (2) Consumer: user online status user_status, user recent access duration user_duration.

[0179] The performance-related features are represented by I1, I2, …, I m Indicates that the cost-related features are expressed as I m+1 ,I m+2 ,…,I m+n express.

[0180] The characteristic information related to data access and the characteristic information related to data producers and consumers (i.e., I1, I2, ..., I m ) is used as a feature vector and input into a Bert-based bidirectional transformer network for feature learning to construct a temperature prediction model.

[0181] Then, according to the characteristic information related to data cost (i.e., I m+1 ,I m+2 ,…,I m+n ) and storage capacity, scheduling cost, etc., and the temperature predicted by the temperature prediction model, the particle swarm algorithm PSO is used to establish the scheduling strategy model. When establishing the scheduling strategy model, the k value can be determined according to the system requirements, so as to balance between minimizing system cost and maximizing system performance according to system requirements. Figure 5 Finally, the optimization of Temperature by the scheduling strategy model is back-propagated to the Sort-Bert network to achieve end-to-end integrated training, thus forming a complete temperature prediction and scheduling model, as shown in Figure 6After the temperature prediction model is optimized using the temperature prediction and scheduling model, the optimized temperature prediction model can be used to obtain the accurate temperature of the data, so that the data can be scheduled to the corresponding storage medium according to the accurate temperature.

[0182] Embodiment 2:

[0183] like Figure 7 , which is a schematic diagram of the architecture of a device for managing data hot and cold lifecycles according to an embodiment of the present invention. The device for managing data hot and cold lifecycles according to this embodiment includes one or more processors 21 and a memory 22. Figure 7 A processor 21 is taken as an example.

[0184] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0185] The memory 22 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs and non-volatile computer executable programs, such as the method for managing the hot and cold lifecycles of data in Example 1. The processor 21 executes the method for managing the hot and cold lifecycles of data by running the non-volatile software programs and instructions stored in the memory 22.

[0186] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0187] The program instructions / modules are stored in the memory 22 , and when executed by the one or more processors 21 , the method for managing the hot and cold lifecycles of data in the above-mentioned embodiment 1 is executed.

[0188] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0189] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0190] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for managing hot and cold data lifecycles, characterized in that: include: Use the labeled hot and cold data training set to perform feature learning to build a temperature prediction model; According to the temperature prediction model, temperature prediction is performed on each group of data in the system to obtain the predicted temperature of each group of data; According to the predicted temperature of each group of data, the temperature variation range corresponding to each group of data is determined, and within the corresponding temperature variation range, a decision is made on the temperature of the data, and a scheduling strategy model with the goal of optimizing the comprehensive performance of the system is constructed; the scheduling strategy model is solved to obtain an optimal scheduling strategy, and data scheduling is performed according to the optimal scheduling strategy; Wherein, in each simulation scheduling of solving the scheduling strategy model, a temperature decision is made for each group of data to obtain the decision temperature corresponding to each group of data, and the current storage medium of each group of data is used as the storage medium before scheduling, and according to the temperature range where the decision temperature of each group of data is located, the storage medium after scheduling of each group of data is determined, so as to perform simulation scheduling; the temperature range is obtained by dividing according to the decision temperature of each group of data in the last simulation scheduling, and each temperature range corresponds to a storage medium; in the first simulation scheduling, the temperature range is obtained by dividing according to the predicted temperature of each group of data; According to the comprehensive performance changes of the system before and after data scheduling, the optimal temperature of each group of data is calculated; according to the optimal temperature of each group of data, the temperature prediction model is reversely optimized to optimize the results of subsequent temperature prediction.

2. The method for managing the hot and cold lifecycle of data according to claim 1, characterized in that: The method uses the labeled hot and cold data training set to perform feature learning to build a temperature prediction model, specifically including: Each feature vector in the cold and hot data training set is normalized, and the normalized feature vector is input into a Bert-based bidirectional transformer network for feature learning to construct a temperature prediction model.

3. The method for managing the hot and cold lifecycle of data according to claim 2, characterized in that: The method of using the labeled hot and cold data training set to perform feature learning to build a temperature prediction model also includes optimizing the temperature prediction model using the labeled validation data set, specifically including: Inputting the labeled data in the labeled verification data set into the temperature prediction model to obtain the predicted temperature of the labeled data; According to the labeled temperature of the labeled data and the predicted temperature of the labeled data, the classifier is optimized using the cross entropy loss, and the cross entropy loss is calculated; According to the cross entropy loss, the temperature prediction model is optimized using a gradient descent algorithm.

4. The method for managing the hot and cold lifecycle of data according to claim 1, characterized in that: The construction of a scheduling strategy model with the goal of optimizing the overall performance of the system specifically includes: Based on the temperature of each group of data, the size of the space occupied by the data, the length of time the data is stored, the value of the data, the storage cost of the storage medium where each group of data is located, and the access rate of each group of data in the corresponding storage medium, an objective function is generated with the goal of maximizing the overall performance of the system; Generate a constraint function based on the size of the space occupied by each group of data, the storage space capacity of each storage medium, the storage cost of the system before scheduling, the storage cost of the system after scheduling, and the total scheduling cost; The scheduling strategy model is established by combining the objective function and the constraint function.

5. The method for managing the hot and cold lifecycle of data according to claim 4, characterized in that: The constraint functions include a storage space constraint function and a scheduling cost constraint function; The storage space constraint function is specifically: For each storage medium, the space occupied by the data group stored in the storage medium is not greater than the actual capacity of the storage medium; The scheduling cost constraint function is specifically: The sum of the total storage cost of the system after scheduling and the scheduling cost is less than the total storage cost of the system before scheduling.

6. The method for managing the hot and cold lifecycle of data according to claim 4, characterized in that: The objective function is specifically: The total storage cost of the system plus the total scheduling cost of the system is the total cost of the system, which is obtained by multiplying the system performance score by the target coefficient k and then dividing it by the total cost of the system; The total storage cost of the system is specifically obtained by multiplying the actual occupied space of the i-th group of data by the storage cost of the storage medium where the i-th group of data is located to obtain a first value of the i-th group of data; Subtract the storage value of the i-th group of data from the first value of the i-th group of data to obtain the remaining storage value of the i-th group of data, and perform a logarithmic operation with the remaining storage value of the i-th group of data as a true number and the natural number e as the base to obtain the second value of the i-th group of data; The storage cost of the i-th group of data is obtained by multiplying the second value of the i-th group of data by the storage duration of the i-th group of data, and the total storage cost of the system is obtained by adding the storage cost of each group of data; The performance score of the system is specifically: The temperature of the i-th group of data is multiplied by the actual access rate of the i-th group of data, and the resultant is divided by the actual occupied space of the i-th group of data to obtain a third value of the i-th group of data; Adding the third values ​​of each set of data to obtain a fourth value, and dividing the storage space capacity by the fourth value to obtain a performance score of the system; The total dispatch cost is specifically: The scheduling cost of each set of data is added up to get the total scheduling cost of the system.

7. The method for managing the hot and cold lifecycle of data according to claim 6, characterized in that: The actual access rate of the i-th group of data and the storage cost of the i-th group of data are determined according to the temperature of the i-th group of data, specifically including: Determining a storage medium for the i-th group of data according to a temperature range corresponding to the temperature of the i-th group of data; The storage cost of the i-th group of data is obtained by multiplying the unit value of the storage medium of the i-th group of data, the storage duration of the i-th group of data, and the actual occupied space of the i-th group of data; The actual access rate of the storage medium of the i-th group of data is used as the actual access rate of the i-th group of data.

8. The method for managing the hot and cold lifecycle of data according to claim 6, characterized in that: The method of calculating the optimal temperature of each group of data according to the comprehensive performance change of the system before and after the data scheduling specifically includes: The total storage cost of the system plus the total scheduling cost of the system is used to obtain the total cost of the system. The total cost of the system is divided by the coefficient k and then multiplied by the comprehensive performance difference Δtarget of the system before and after scheduling to obtain the performance score difference before and after the system scheduling. Multiply the system storage space capacity by the actual access rate of the i-th group of data, and then divide it by the actual occupied space of the i-th group of data to obtain the fifth value of the i-th group of data, calculate the partial derivative of the performance score difference with respect to the fifth value, and add the partial derivative to the pre-scheduling temperature of the i-th group of data to obtain the optimal temperature of the i-th group of data.

9. The method for managing the hot and cold lifecycle of data according to claim 1, characterized in that: The reverse optimization of the temperature prediction model according to the optimal temperature of each set of data specifically includes: According to the optimal temperature, the classifier is optimized using the cross entropy loss to calculate the cross entropy loss; According to the cross entropy loss, the temperature prediction model is optimized using a gradient descent algorithm.

10. A device for managing the hot and cold lifecycle of data, characterized in that: The device comprises: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the method for data hot and cold lifecycle management as described in any one of claims 1-9.

Images