A method for calculating data temperature and performing hierarchical storage management

Abstract

This invention relates to a method for calculating data temperature and performing hierarchical storage management, belonging to the field of computer data processing. Based on data access time, data access frequency, data attributes, and the attributes of the user initiating the access, a data temperature calculation model is constructed. This model manages the dynamic migration of data between a local hot database, a local cold database, and a remote cold database. Newly generated data and cold data with rising local temperatures are stored in the hot database. Periodic time-driven migrations are performed, storing data predicted to be migrated to the hot database in the next cycle and data with currently high temperatures in the hot database. Based on the benefit of storing unit data remotely compared to storing it locally, a decision is made regarding whether cold data needs to be migrated remotely. This invention can significantly reduce costs while ensuring data access performance, predict new data volume, balance hot storage space utilization and data migration overhead, and pre-migrate data by identifying periodic data migration patterns, avoiding time lag issues in migration.

Description

Technical Field

[0001] This invention belongs to the field of computer data processing, and specifically relates to a method for calculating data temperature and performing hierarchical storage management. Background Technology

[0002] One of the primary functions of a data center is data storage. In recent years, the amount of data in data centers has increased exponentially, and the demand for data management and processing capabilities has grown significantly, placing higher demands on data center storage capacity. To cope with the dramatic increase in data volume, it is necessary to consider identifying hot and cold data and allocating data to different storage media to achieve a balance between data access response speed and data center operating costs.

[0003] HDDs (Hard Disk Drives) are the most commonly used storage devices. While they are relatively inexpensive, their slow read speeds make them unsuitable for high-speed access, making them ideal for storing cold databases. Conversely, SSDs (Solid State Drives) offer faster read speeds, but their higher price limits their widespread adoption, making them suitable for storing hot databases.

[0004] Currently, land development in most developed cities is nearing saturation, and the supply of basic materials such as water and electricity is also in a state of tension. Meanwhile, some inland areas have achieved remarkable results in the development of new energy sources, with lower electricity prices and relatively lower temperatures, making them suitable for building data centers. The concept of "data in the east and storage in the west" has received widespread attention and development. How to manage data under the new data management needs and situation is an issue that urgently needs to be studied and solved.

[0005] Over a period of time, data access follows the "Pareto Principle" (80 / 20 rule), meaning that most access is concentrated on a small portion of the data. The "hotness" or "coldness" of data describes how frequently it is used; hot data typically refers to frequently accessed data, while cold data refers to data accessed less frequently but requiring long-term retention. If we can accurately identify the hotness or coldness of data, we can store frequently used data on high-performance local SSDs, moderately used data on lower-cost local HDDs, and rarely used data on larger, remote HDDs, thus achieving tiered data storage.

[0006] In research on hot and cold data identification, the most common solutions are the two classic cache replacement algorithms, LRU (Least Recently Used) and LFU (Least Frequently Used). However, LRU performs poorly when faced with occasional or periodic data access; LFU performs poorly when access patterns frequently change. Later researchers proposed LRU-K and LFU-Aging based on LRU and LFU. LRU-K adds a counter to LRU to count the number of data accesses. Only when the number of data accesses reaches K times will the data enter the hot storage space.

[0007] LFU-Aging adds constraints on access time to LFU to prevent data with high historical access frequency from residing in hot storage space. However, using simple data structures to identify hot and cold data cannot maintain good performance under all access modes. In recent years, methods have emerged that use data temperature values ​​to measure data popularity, but these still only evaluate basic access characteristics such as access time, access frequency, and access correlation, without considering the impact of the importance of users and data in data access on data popularity. Summary of the Invention

[0008] The technical problem to be solved by this invention is:

[0009] To address the shortcomings of existing methods for identifying hot and cold data, which only evaluate basic access characteristics such as access time, access frequency, and access correlation without considering the impact of the importance of users and data on data popularity, and which only consider local storage management of different media without comprehensively considering local and remote storage management, this invention provides a method for calculating data temperature and performing hierarchical storage management.

[0010] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0011] A method for calculating data temperature and performing hierarchical storage management, characterized in that it includes:

[0012] A data temperature calculation model is constructed based on data access time, data access frequency, data attributes, and user attributes that initiate the access.

[0013] Based on the time it takes for data to migrate from a cold database to a hot database, we can uncover patterns in the periodic migration of data.

[0014] Local cold and hot data storage management: Based on data temperature and periodic data migration patterns, manage the migration of local data between local cold and hot databases. Migration is divided into event-driven migration and time-driven migration. Event-driven migration includes the generation of new data and the increase of local cold data temperature; time-driven migration occurs periodically.

[0015] Local and remote data storage management: Based on data temperature, manage the migration of data between the local database and the remote database. The migration is event-driven, and the events include the increase in remote cold data temperature and the insufficient storage space of the local cold database.

[0016] A further technical solution of the present invention: The data temperature calculation model is described as follows:

[0017] To set an initial temperature for the new data, the average temperature of the thermal data at the time the new data was generated is set as the initial temperature of the new data; the formula for calculating the initial temperature is:

[0018]

[0019] Where hotDataCount represents the number of hot data points, T hotData(i) This represents the temperature of the i-th thermal data point;

[0020] The formula for increasing the temperature of accessed data is as follows:

[0021] T new =T pre +α+β×I+γ×Q

[0022] Where I and Q represent the user attribute and data attribute that issued the access, and T pre The temperature of the data before access is given by α, β, and γ, which represent the weights of the basic access, user attributes, and data attributes in a single access, respectively.

[0023] The data temperature is periodically reduced; the calculation formula is as follows:

[0024] T new =δ*T pre

[0025] Among them, T pre To reduce the temperature of the previous data, δ is the temperature attenuation coefficient.

[0026] A further technical solution of the present invention: The specific method for mining the periodic migration pattern of data is as follows:

[0027] A two-dimensional array `coldToHotPeriod[d,y]` records the period in which cold data `d` is migrated to the hot database for the yth time; when cold data `d` is migrated to the hot database for the yth time, and y≥4, calculate:

[0028] gap i =coldToHot[d,yi]-coldToHot[d,yi-1],i=0,1,2

[0029] If the following conditions are met:

[0030] gap0 = gap1 = gap2

[0031] It is then assumed that the migration of cold data d from the cold database to the hot database follows a periodic pattern, with the period gap being:

[0032] gap = gap0 = gap1 = gap2

[0033] The next migration is predicted to occur in the nth period, where n is calculated using the following formula:

[0034] n = coldToHot[d,y] + gap

[0035] At the end of the (n-1)th period, the data d is pre-migrated, and the validity period of each rule is m periods gap.

[0036] A further technical solution of the present invention: In local cold and hot data storage management, the migration of new data is described as follows:

[0037] New data is stored in the local hot database. If the hot database does not have enough available storage space, some hot data needs to be eliminated and moved to the local cold database in order of increasing temperature.

[0038] A further technical solution of the present invention: In local cold and hot data storage management, the migration that occurs when the temperature of the local cold data rises is described as follows:

[0039] After cold data is accessed and its temperature is updated, it is determined whether its temperature has reached the standard value for hot data temperature. If so, it is migrated to the hot database to meet the data access requirements.

[0040] If the available storage space in the hot database is insufficient, some hot data needs to be evicted to the local cold database in order of increasing temperature.

[0041] A further technical solution of the present invention: In local cold and hot data storage management, the specific method of time-driven migration is as follows:

[0042] Set a migration cycle, and perform data migration between the local cold and hot databases at the end of each cycle. Store the data that is predicted to be migrated to the hot database in the next cycle and the data with the highest current temperature in the hot database, while ensuring that the used storage space of the hot database does not exceed the upper limit threshold after the migration is completed.

[0043] Step 1-1: Calculate the upper limit threshold;

[0044] The upper limit threshold represents the upper limit of the used storage space of the hot database after the time-driven migration is completed; based on the LSTM prediction of the amount of new data e generated in the next cycle, and c representing the total storage capacity of the hot database, the formula for calculating the upper limit threshold ζ is:

[0045] ζ=c-0.5×e

[0046] Step 1-2: Select the data set that needs to be pre-migrated based on the periodic migration pattern;

[0047] Let the upper threshold of this migration be ζ. Traverse the cold data in descending order of temperature and select the data that need to be pre-migrated to form a set D1 = {d1, d2, ..., dn}. n}, the corresponding data size set V1={v1,v2,…,v n Set D1 includes all pre-migrated data, or satisfies and

[0048] Steps 1-3: Filter the data set that needs to be migrated based on data temperature;

[0049] Calculate the remaining available space v of the thermal database r The calculation formula is:

[0050]

[0051] Let the universal set U represent all data. From the set U-D1, select data in descending order of temperature to form a set D2 = {d...} n+1 d n+2 , ...,d m}, the corresponding data size set V2={v n+1 v n+2 , ..., v m},satisfy and

[0052] Steps 1-4: Migrate data;

[0053] Let set D = D1 + D2. First, migrate the data in the hot database that does not belong to set D to the cold database. Then, migrate the data in set D that is located in the cold database to the hot database.

[0054] A further technical solution of the present invention: The specific method for predicting the amount of new data generated in the next cycle based on LSTM is as follows:

[0055] The steps for training an LSTM model are as follows:

[0056] Step 2-1: Collect 1680 data points from the past 240 weeks as a dataset, and standardize all data in the dataset;

[0057] Step 2-2: Construct input and output samples. Each input sample contains 84 data points, i.e., I(i) = {x} i x i+1 , ..., x i+83 Each output sample contains 7 data points, i.e., O(i) = {x} i+84 x i+85 , ..., x i+90}, and use 80% of the sample set as the training set and 20% as the test set;

[0058] Steps 2-3: Construct an LSTM network. Use the Keras deep learning framework to build a Sequential model, adding 3 LSTM layers, 1 fully connected layer, and 1 output layer. In the first LSTM layer, set the_units, input_shape, activation, and return_sequences. Remove the input_shape in the second LSTM layer, and remove the input_shape and return_sequences in the third LSTM layer. Set the Dropout value to 0.1, and use ReLU as the activation function.

[0059] Steps 2-4: Compile the LSTM network, using Adam as the optimizer, MSE as the loss function, and MapE as the evaluation function;

[0060] When using a trained LSTM model for prediction, the 84 new data points from each day of the most recent 12 weeks are standardized as input, and the output sequence is destandardized to obtain the predicted sequence of new data points for each day of the next week (7 days).

[0061] A further technical solution of the present invention: In local and remote data storage management, the migration that occurs when the temperature of remote cold data rises is described as follows:

[0062] After cold data is accessed and its temperature is updated, it is determined whether its temperature has reached the standard value for hot data temperature. If so, it is migrated to the hot database to meet the data access requirements.

[0063] If the available storage space in the hot database is insufficient, some hot data needs to be evicted to the local cold database in order of increasing temperature.

[0064] A further technical solution of the present invention: The specific method for discarding hot data when the hot database storage space is insufficient is as follows:

[0065] Step 3-1: Determine whether a hot data list has been built in this cycle. If not, build the hot data list. The list nodes store the data name, data temperature, and data size.

[0066] Step 3-2: Sort the thermal data list in descending order of temperature values;

[0067] Step 3-3: Delete the data with the lowest current temperature to the cold database;

[0068] Step 3-4: Determine if the hot database storage space is sufficient. If yes, proceed to step 3-5; otherwise, proceed to step 3-3.

[0069] Steps 3-5: Migrate the cold data to be migrated to the hot database, and insert the data information into the hot data linked list.

[0070] A further technical solution of the present invention: In local and remote data storage management, the migration that occurs when the local cold database storage space is insufficient is described as follows:

[0071] Set the upper limit threshold of the local cold database storage space to 90% of the total storage capacity of the local cold database. When the upper limit threshold is reached, it indicates that the storage space of the local cold database is insufficient, triggering the migration of local cold data to a remote location.

[0072] The cost-benefit analysis for storing data remotely versus locally is based on the following method:

[0073] Storage cost includes monetary cost (expCost) and latency cost (dlyCost). L The local storage cost for cold data is calculated using the following formula:

[0074] cost L =expCost L +θ*dlyCost L

[0075] cost R The cost of storing cold data remotely is calculated using the following formula:

[0076] cost R =expCost R +θ*dlyCost R

[0077] The formula for calculating expCost is:

[0078] expCost = s * e * ep

[0079] Where s represents the size of the data file, e represents the power of storing a unit of data, and ep represents the electricity price per unit.

[0080] The formula for calculating dlyCost is:

[0081]

[0082] Where bw represents bandwidth, pd represents propagation delay, and dlyCost R pd in R dlyCost is obtained by measuring the latency of short messages. L pd in L Negligible; T' normalizes the data temperature T, and the calculation formula is:

[0083]

[0084] Among them, T min and T max These represent the lowest and highest temperatures in the local cold data, respectively.

[0085] Let udy represent the benefit of storing unit data remotely compared to storing it locally, calculated using the following formula:

[0086]

[0087] Here, θ represents the weight of latency cost relative to monetary cost, and epdiff represents the difference in unit electricity price between local and remote locations; the amount of data migrated to the remote location is changed by adjusting the value of θ.

[0088] The beneficial effects of this invention are as follows:

[0089] The present invention provides a method for calculating data temperature and performing hierarchical storage management, which has the following advantages compared with the prior art:

[0090] (1) Based on different user attributes and data attributes, the present invention classifies data access into categories. Different categories of access have different contribution values ​​to data temperature, which can prioritize the satisfaction of important users when resources are limited.

[0091] (2) This invention uses LSTM to predict the amount of new data in the next cycle, and makes a decision on the upper limit threshold of time-driven migration based on the predicted value of the new data amount, which can balance the hot storage space utilization rate and data migration overhead.

[0092] (3) By analyzing the periodic access patterns and sequential access patterns of data, this invention can migrate the data that is about to be migrated in advance, thus avoiding the problem of time lag in data migration after the temperature of the data changes.

[0093] (4) This invention quantifies the real-time hot and cold status of data in the data center. Layered storage can maintain a high hot database hit rate and a fast access response time when storing massive amounts of data.

[0094] (5) This invention proposes a local and remote data storage management scheme. Attached Figure Description

[0095] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0096] Figure 1 This is a diagram illustrating the overall architecture of the method of the present invention. Detailed Implementation

[0097] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0098] This invention provides a method for calculating data temperature and performing hierarchical storage management. Based on data access time, data access frequency, data attributes, and the attributes of the user issuing the access, a data temperature calculation model is constructed. This model manages the dynamic migration of data between a local hot database, a local cold database, and a remote cold database. Newly generated data and cold data with rising local temperatures are stored in the hot database. Periodic time-driven migrations are performed, storing data predicted to be migrated to the hot database in the next cycle and data with currently high temperatures in the hot database. Based on the benefit of storing unit data remotely compared to storing it locally, the method determines whether cold data needs to be migrated remotely. This invention can significantly reduce costs while ensuring data access performance. It uses LSTM to predict the amount of new data, balances hot storage space utilization and data migration overhead, and mines periodic data migration patterns to pre-migrate data, avoiding time lag issues in migration.

[0099] like Figure 1 As shown in the figure, an embodiment of the present invention provides a method for calculating data temperature and performing hierarchical storage management, which includes the following steps:

[0100] Step 1: Classify data access types;

[0101] Based on user attributes and data attributes, data access is divided into four different types: ordinary users accessing ordinary data, ordinary users accessing key data, important users accessing ordinary data, and important users accessing key data.

[0102] Step 2: Establish a data temperature calculation model;

[0103] A data temperature calculation model is established based on data access time, data access frequency, data attributes, and user attributes that initiate the access.

[0104] Step 2-1: Set the initial temperature for the newly generated data;

[0105] The new data is data that users will access in the near future and needs to be stored in the thermal database. Therefore, its initial temperature is set to the average temperature of the thermal data when the new data is generated.

[0106] Step 2-2: Increase temperature after data is accessed;

[0107] Different types of access contribute differently to data temperature;

[0108] Steps 2-3: Periodically reduce the data temperature;

[0109] To avoid the lasting impact of historically high access frequency on data temperature, it is necessary to periodically reduce the data temperature.

[0110] Step 3: Tiered data storage management;

[0111] Step 3-1: Local cold and hot data storage management;

[0112] It includes two types of event-driven migrations and one type of time-driven migration: one type of event-driven migration occurs when new data is generated, and the other type of event-driven migration occurs when the temperature of local cold data rises; the time-driven migration occurs periodically.

[0113] Step 3-2: Local and remote data storage management;

[0114] There are two types of event-driven migrations: one occurs when the temperature of remote cold data rises, and the other occurs when the storage space of the local cold database is insufficient.

[0115] Furthermore, the data temperature calculation model is described as follows:

[0116] For the newly generated data, the initial temperature T newData The calculation formula is:

[0117]

[0118] Where hotDataCount represents the number of hot data points, T hotData(i) This represents the temperature of the i-th thermal data point;

[0119] Let I and Q represent the user attribute and data attribute that initiate the access. If the user is an important user, I = 1; otherwise, I = 0. If the data is important data, Q = 1; otherwise, Q = 0. Let T be the temperature of the data before the access.pre The temperature T of the data after being accessed new The calculation formula is:

[0120] T new =T pre +α+β×I+γ×Q

[0121] Where α, β, and γ represent the weights of the basic access, user attributes, and data attributes on the temperature contribution in a single access session, respectively.

[0122] Periodically reduce the data temperature, denoted as T. pre To reduce the previous data temperature, where δ is the temperature attenuation coefficient, the reduced data temperature T is... new It can be represented as:

[0123] T new =δ*T pre

[0124] Furthermore, in local cold and hot data storage management, the migration that occurs when new data is generated is described as follows:

[0125] Newly generated data is stored in the local hot database. If the available storage space in the hot database is insufficient, some hot data needs to be eliminated and moved to the local cold database in order of increasing temperature.

[0126] Furthermore, in local cold and hot data storage management, the migration that occurs when the local cold data temperature rises is described as follows:

[0127] After cold data is accessed and its temperature is updated, determine whether its temperature has reached the standard value T for hot data temperature. std If so, then migrate it to the hot database to meet data access needs; T std The calculation formula is:

[0128]

[0129] Where hotdataCount represents the number of hot data points, T hotData(i) This represents the temperature of the i-th thermal data point, where ε is a constant; the larger the value, the higher the standard value T. std The lower;

[0130] If the available storage space in the hot database is insufficient, some hot data needs to be evicted to the local cold database in order of increasing temperature.

[0131] Furthermore, in local cold and hot data storage management, the specific method of time-driven migration is as follows:

[0132] Let the migration period be p. At the end of each period, a large-scale data migration between the local cold and hot databases is carried out. The data that is predicted to be migrated to the hot database in the next period and the data with the highest current temperature are stored in the hot database. At the same time, it is ensured that the used storage space of the hot database does not exceed the upper limit threshold after the migration is completed.

[0133] Step a-1: Calculate the upper limit threshold;

[0134] The upper limit threshold represents the upper limit of the used storage space of the hot database after the time-driven migration is completed; based on the LSTM prediction of the amount of new data e generated in the next cycle, and c representing the total storage capacity of the hot database, the formula for calculating the upper limit threshold ζ is:

[0135] ζ=c-0.5×e

[0136] Step a-2: Select the data set that needs to be pre-migrated based on the periodic migration pattern;

[0137] Let the upper threshold of this migration be ζ. Traverse the cold data in descending order of temperature and select the data that need to be pre-migrated to form a set D1 = {d1, d2, ..., dn}. n}, the corresponding data size set V1={v1,v2,…,v n Set D1 includes all pre-migrated data, or satisfies and

[0138] Step a-3: Filter the data set that needs to be migrated based on data temperature;

[0139] Calculate the remaining available space v of the thermal database r The calculation formula is:

[0140]

[0141] Let the universal set U represent all data. From the set U-D1, select data in descending order of temperature to form a set D2 = {d...} n+1 d n+2 , ...,d m}, the corresponding data size set V2={v n+1 v n+2 , ..., v m},satisfy and

[0142] Step a-4: Migrate data;

[0143] Let set D = D1 + D2. First, migrate the data in the hot database that does not belong to set D to the cold database. Then, migrate the data in set D that is located in the cold database to the hot database.

[0144] Furthermore, the specific method for predicting the amount of new data generated in the next cycle based on LSTM is as follows:

[0145] The steps for training an LSTM model are as follows:

[0146] Step b-1: Collect 1680 data points from the past 240 weeks as a dataset, and standardize all data in the dataset;

[0147] Step b-2: Construct input and output samples. Each input sample contains 84 data points, i.e., I(i) = {x i x i+1 , ..., x i+83 Each output sample contains 7 data points, i.e., O(i) = {x} i+84 x i+85 , ..., x i+90}, and use 80% of the sample set as the training set and 20% as the test set;

[0148] Step b-3: Construct an LSTM network. Use the Keras deep learning framework to build a Sequential model, adding 3 LSTM layers, 1 fully connected layer, and 1 output layer. In the first LSTM layer, set the_units, input_shape, activation, and return_sequences. Remove the input_shape in the second LSTM layer, and remove the input_shape and return_sequences in the third LSTM layer. Set the Dropout value to 0.1 and use ReLU as the activation function.

[0149] Step b-4: Compile the LSTM network, using Adam as the optimizer, MSE as the loss function, and MapE as the evaluation function;

[0150] When using a trained LSTM model for prediction, the 84 new data points from each day of the most recent 12 weeks are standardized as input, and the output sequence is destandardized to obtain the predicted sequence of new data points for each day of the next week (7 days).

[0151] Furthermore, the specific methods for uncovering the periodic migration patterns of data are as follows:

[0152] A two-dimensional array `coldToHotPeriod[d,y]` records the period in which cold data `d` is migrated to the hot database for the yth time. At least four records are needed to uncover periodic patterns. When cold data `d` is migrated to the hot database for the yth time (y≥4), calculate:

[0153] gap i=coldToHot[d,yi]-coldToHot[d,yi-1],i=0,1,2

[0154] If the following conditions are met:

[0155] gap0 = gap1 = gap2

[0156] It is then assumed that the migration of cold data d from the cold database to the hot database follows a periodic pattern, with the period gap being:

[0157] gap = gap0 = gap1 = gap2

[0158] The next migration is predicted to occur in the nth period, where n is calculated using the following formula:

[0159] n = coldToHot[d,y] + gap

[0160] At the end of the (n-1)th period, the data d is pre-migrated, and the validity period of each rule is m periods gap.

[0161] Furthermore, in local and remote data storage management, the migration that occurs when the temperature of the remote cold data increases is described as follows:

[0162] After cold data is accessed and its temperature is updated, determine whether its temperature has reached the standard value T for hot data temperature. std If so, then migrate it to the hot database to meet data access needs;

[0163] If the available storage space in the hot database is insufficient, some hot data needs to be evicted to the local cold database in order of increasing temperature.

[0164] Furthermore, the specific method for discarding hot data when the hot database storage space is insufficient is as follows:

[0165] Step c-1: Determine whether a hot data list has been built in this cycle. If not, build the hot data list. The list nodes store information such as data name, data temperature, and data size.

[0166] Step c-2: Sort the thermal data list in descending order of temperature values;

[0167] Step c-3: Remove the data with the lowest current temperature from the cold database;

[0168] Step c-4: Determine if the hot database storage space is sufficient. If yes, proceed to step 3-5; otherwise, proceed to step 3-3.

[0169] Step c-5: Migrate the cold data to be migrated to the hot database, and insert the data information into the hot data linked list.

[0170] Furthermore, in local and remote data storage management, the migration that occurs when the local cold database storage space is insufficient is described as follows:

[0171] Set the upper limit threshold of the local cold database storage space to 90% of the total storage capacity of the local cold database. When the upper limit threshold is reached, it is considered that the storage space of the local cold database is insufficient, triggering the migration of local cold data to the remote database.

[0172] The cost-benefit analysis for storing data remotely versus locally is based on the following method:

[0173] Storage cost includes monetary cost (expCost) and latency cost (dlyCost). L The local storage cost for cold data is calculated using the following formula:

[0174] cost L =expCost L +θ*dlyCost L

[0175] cost R The cost of storing cold data remotely is calculated using the following formula:

[0176] cost R =expCost R +θ*dlyCost R

[0177] The formula for calculating expCost is:

[0178] expCost = s * e * ep

[0179] Where s represents the size of the data file, e represents the power of storing a unit of data, and ep represents the electricity price per unit.

[0180] The formula for calculating dlyCost is:

[0181]

[0182] Where bw represents bandwidth, pd represents propagation delay, and dlyCost R pd in R dlyCost is obtained by measuring the latency of short messages. L pd in L Negligible; T' normalizes the data temperature T, and the calculation formula is:

[0183]

[0184] Among them, Tmin and T max These represent the lowest and highest temperatures in the local cold data, respectively.

[0185] Let udy represent the benefit of storing unit data remotely compared to storing it locally, calculated using the following formula:

[0186]

[0187] Here, θ represents the weight of latency cost relative to monetary cost, and epdiff represents the difference in unit electricity price between local and remote locations; the amount of data migrated to the remote location is changed by adjusting the value of θ.

[0188] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in the present invention, and such modifications or substitutions should all be covered within the scope of protection of the present invention.

Claims

1. A method for calculating data temperature and performing hierarchical storage management, characterized in that, include: A data temperature calculation model is constructed based on data access time, data access frequency, data attributes, and user attributes that initiate the access. Based on the time it takes for data to migrate from a cold database to a hot database, we can uncover patterns in the periodic migration of data. Local hot and cold data storage management: Based on data temperature and periodic data migration patterns, manage the migration of local data between local hot and cold databases. Migration is divided into event-driven migration and time-driven migration. Event-driven migration includes the generation of new data and the increase of local cold data temperature; time-driven migration occurs periodically. The specific method for time-driven migration in local hot and cold data storage management is as follows: Set a migration cycle, and perform data migration between the local cold and hot databases at the end of each cycle. Store the data that is predicted to be migrated to the hot database in the next cycle and the data with the highest current temperature in the hot database, while ensuring that the used storage space of the hot database does not exceed the upper limit threshold after the migration is completed. Step 1-1: Calculate the upper limit threshold; The upper limit threshold represents the upper limit of the used storage space of the hot database after the time-driven migration is completed; Predicting the amount of new data generated in the next cycle based on LSTM. e ,use c This represents the total storage capacity of the hot database, with an upper limit threshold. ζ The formula for calculating the value is: Step 1-2: Select the data set that needs to be pre-migrated based on the periodic migration pattern; Let the upper limit threshold of this migration be... The cold data is traversed in descending order of temperature, and the data that needs to be pre-migrated is selected to form a set D1={d1, d2, ..., d...}. n }, the corresponding data size set V1={ v 1, v 2, ..., v n Set D1 includes all pre-migrated data and satisfies and ; Steps 1-3: Filter the data set that needs to be migrated based on data temperature; Calculate the remaining available space in the thermal database. v r The calculation formula is: Let the universal set U represent all data. From the set U-D1, select data in descending order of temperature to form a set D2 = {d...} n+1 d n+2 , ..., d m }, the corresponding data size set V2={ v n+1 , v n+2 , ..., v m },satisfy and ; Steps 1-4: Migrate data; Let set D = D1 + D2. First, migrate the data in the hot database that does not belong to set D to the cold database. Then, migrate the data in set D that is located in the cold database to the hot database. Local and remote data storage management: Based on data temperature, manage the migration of data between the local database and the remote database. The migration is event-driven, and the events include the increase in remote cold data temperature and the insufficient storage space of the local cold database.

2. The method for calculating data temperature and performing hierarchical storage management according to claim 1, characterized in that, The data temperature calculation model is described as follows: To set an initial temperature for the new data, the average temperature of the thermal data at the time the new data was generated is set as the initial temperature of the new data; the formula for calculating the initial temperature is: in, hotDataCount Indicates the number of hot data points. T hotData(i) Indicates the first i Temperature of each thermal data point; The formula for increasing the temperature of accessed data is as follows: in, I and Q This indicates the user attributes and data attributes that issued the access request. T pre Temperature of the data before access. α , β and γ These represent the weights of the basic visit, user attributes, and data attributes in terms of their contribution to temperature during a single visit. The data temperature is periodically reduced; the calculation formula is as follows: in, To reduce the temperature of the previous data, δ This is the temperature attenuation coefficient.

3. The method for calculating data temperature and performing hierarchical storage management according to claim 1, characterized in that, The specific methods for uncovering the periodic migration patterns in data are as follows: Use a two-dimensional array coldToHot[d,y] to record cold data. d No. y The number of cycles in which the data is migrated to the hot database; When cold data d No. y This time it was migrated to the hot database, and y When ≥4, calculate: If the following conditions are met: It is then assumed that the migration of cold data d from the cold database to the hot database follows a periodic pattern, with the period gap being: The next migration is predicted to occur in the nth period, where n is calculated using the following formula: In the n -1 period end data d Pre-migrating is performed, and the validity period of each rule is [duration missing]. m One cycle gap.

4. The method for calculating data temperature and performing hierarchical storage management according to claim 1, characterized in that, In local cold and hot data storage management, the migration that occurs when new data is generated is described as follows: New data is stored in the local hot database. If the hot database does not have enough available storage space, some hot data needs to be eliminated and moved to the local cold database in order of increasing temperature.

5. The method for calculating data temperature and performing hierarchical storage management according to claim 1, characterized in that, In local cold and hot data storage management, the migration that occurs when the temperature of local cold data rises is described as follows: After cold data is accessed and its temperature is updated, it is determined whether its temperature has reached the standard value for hot data temperature. If so, it is migrated to the hot database to meet the data access requirements. If the available storage space in the hot database is insufficient, some hot data needs to be evicted to the local cold database in order of increasing temperature.

6. The method for calculating data temperature and performing hierarchical storage management according to claim 1, characterized in that, The specific method for predicting the amount of new data generated in the next period based on LSTM is as follows: The steps for training an LSTM model are as follows: Step 2-1: Collect 1680 data points from the past 240 weeks as a dataset, and standardize all data in the dataset; Step 2-2: Construct input and output samples. Each input sample contains 84 data points, i.e., I(i) = {x i x i+1 , ..., x i+83 Each output sample contains 7 data points, i.e., O(i) = {x} i+84 x i+85 , ..., x i+90 }, and use 80% of the sample set as the training set and 20% as the test set; Steps 2-3: Construct an LSTM network. Use the Keras deep learning framework to build a Sequential model, adding 3 LSTM layers, 1 fully connected layer, and 1 output layer. In the first LSTM layer, set the_units, input_shape, activation, and return_sequences. Remove the input_shape in the second LSTM layer, and remove the input_shape and return_sequences in the third LSTM layer. Set the Dropout value to 0.1, and use ReLU as the activation function. Steps 2-4: Compile the LSTM network, using Adam as the optimizer, MSE as the loss function, and MapE as the evaluation function; When using a trained LSTM model for prediction, the 84 new data points from each day of the most recent 12 weeks are standardized as input, and the output sequence is destandardized to obtain the predicted sequence of new data points for each day of the next week (7 days).

7. The method for calculating data temperature and performing hierarchical storage management according to claim 1, characterized in that, In local and remote data storage management, the migration that occurs when the temperature of remote cold data rises is described as follows: After cold data is accessed and its temperature is updated, it is determined whether its temperature has reached the standard value for hot data temperature. If so, it is migrated to the hot database to meet the data access requirements. If the available storage space in the hot database is insufficient, some hot data needs to be evicted to the local cold database in order of increasing temperature.

8. A method for calculating data temperature and performing hierarchical storage management according to claim 3, 4, or 7, characterized in that, The specific methods for discarding hot data when the hot database storage space is insufficient are as follows: Step 3-1: Determine whether a hot data list has been built in this cycle. If not, build the hot data list. The list nodes store the data name, data temperature, and data size. Step 3-2: Sort the thermal data list in descending order of temperature values; Step 3-3: Delete the data with the lowest current temperature to the cold database; Step 3-4: Determine if the hot database storage space is sufficient. If yes, proceed to step 3-5; otherwise, proceed to step 3-3. Steps 3-5: Migrate the cold data to be migrated to the hot database, and insert the data information into the hot data linked list.

9. The method for calculating data temperature and performing hierarchical storage management according to claim 1, characterized in that, In local and remote data storage management, the migration that occurs when the local cold database storage space is insufficient is described as follows: Set the upper limit threshold of local cold database storage space to 90% of the total storage capacity of local cold database. When the upper limit threshold is reached, it indicates that the storage space of local cold database is insufficient, triggering the migration of local cold data to remote storage. The cost-benefit analysis for storing data remotely versus locally is based on the following method: Storage costs cost Including monetary costs expCost and latency costs dlyCost ,use cost L The local storage cost for cold data is calculated using the following formula: use cost R The cost of storing cold data remotely is calculated using the following formula: expCost The calculation formula is: in, s Indicates the size of the data file. e Indicates the power of storing a unit of data. ep This indicates the unit price of electricity; dlyCost The calculation formula is: in, bw Indicates bandwidth. pd Indicates the propagation delay. dlyCost R In pd R Obtained by measuring the latency of short messages. dlyCost L In pd L Negligible; T' Data temperature T Normalization was performed, and the calculation formula is as follows: in, T min and T max These represent the lowest and highest temperatures in the local cold data, respectively. use udy The formula representing the benefit of storing unit data remotely compared to storing it locally is: in, θ Assigning the weight of latency costs relative to monetary costs. epdiff This represents the difference between local and remote electricity unit prices; it is adjusted... θ The value is used to change the amount of data migrated to the remote location.

Images