Frequency access data storage migration method and device for big data

Abstract

The invention belongs to the field of data storage, and discloses a frequency access data storage migration method and device for big data, and the method comprises the steps: obtaining a data access log, carrying out the preprocessing of the data access log, and obtaining a time window statistical matrix; carrying out timeliness attenuation weighting calculation on a plurality of data access times in the time window statistical matrix to obtain a weighted access frequency vector; obtaining a multi-dimensional feature matrix based on the weighted access frequency vector; clustering the feature vector of each time window slice in the multi-dimensional feature matrix through a Gaussian mixture model to obtain a clustering tag vector; inputting the clustering label vector into an access frequency prediction model to obtain an access frequency prediction value of the data in a future preset time step; the migration decision matrix is determined based on the access frequency predicted value and the data storage cost parameter, data migration is carried out based on the migration decision matrix, the data migration complexity can be reduced, and the access speed and the storage cost are balanced.

Description

Technical Field

[0001] The present invention belongs to the field of data storage technology, and in particular relates to a method and device for migrating frequently accessed data for big data. Background Art

[0002] With the increasing popularity of large-scale big data storage systems like data lakes and data centers, and the continuous expansion of enterprise data, the requirements for physical data storage and access are also increasing, leading to an exponential increase in hardware costs. Currently, data storage solutions in related technologies typically store data in fixed locations on big data storage media like data lakes and data centers, meaning that data storage locations cannot be dynamically adjusted.

[0003] In practical applications, some data is frequently accessed, often referred to as frequently accessed data. To achieve fast access to frequently accessed data, conventional data storage solutions require manual migration of frequently accessed data to high-speed storage areas, increasing labor costs and complexity. Summary of the Invention

[0004] The purpose of the present invention is to reduce the labor cost consumed by data migration and reduce the complexity of data migration, and to ensure low data storage costs while meeting data access speed.

[0005] In a first aspect, an embodiment of the present invention provides a method for migrating frequently accessed data for big data storage, the method comprising:

[0006] Obtaining data access logs, preprocessing the data access logs to obtain a time window statistical matrix, where the time window statistical matrix includes multiple time window slices, each time window slice includes a window start time, a window end time, and the number of data accesses within the time window;

[0007] Performing a time-attenuation weighted calculation on multiple data access times in the time window statistical matrix to obtain a weighted access frequency vector, wherein the weighting coefficient corresponding to the data access times of each time window slice is inversely proportional to the time difference between the time window slice and the current time;

[0008] Based on the weighted access frequency vector, the statistical features of multiple data access times are extracted to obtain a multi-dimensional feature matrix, where each row of the multi-dimensional feature matrix corresponds to the feature vector of a time window slice, and each column corresponds to a feature type;

[0009] Clustering the feature vectors of each time window slice in the multidimensional feature matrix using a Gaussian mixture model to obtain a cluster label vector;

[0010] Input the cluster label vector into the access frequency prediction model to obtain the access frequency prediction value of the data within a preset time step in the future, and each access frequency prediction value corresponds to a confidence interval;

[0011] Based on the access frequency prediction value and the data storage cost parameter, a migration decision matrix is ​​determined, and data migration is performed based on the migration decision matrix, wherein the migration decision matrix balances access speed and storage cost by dynamically adjusting the data storage location.

[0012] Optionally, preprocessing the data access log to obtain a time window statistical matrix includes:

[0013] Collecting data request details from the data access log, and extracting data query items from the data request details;

[0014] Cleaning, merging, and deduplication operations are performed on the data query items to obtain a continuously readable data request standard detail;

[0015] A plurality of time window slices are extracted from the continuously readable data request standard details, and the plurality of time window slices are combined into a time window statistical matrix.

[0016] Optionally, the statistical features of the number of data accesses are extracted based on the weighted access frequency vector to obtain a multi-dimensional feature matrix, including:

[0017] Perform first-order difference calculation on the weighted access frequency vector to eliminate the trend component in the weighted access frequency vector and transform the weighted access frequency vector from a non-stationary sequence to a stationary sequence;

[0018] Calculating statistics of the weighted access frequency vector; the statistics include mean, variance and autocorrelation coefficient;

[0019] For each time window, a multidimensional feature vector of the time window is composed based on the original value corresponding to the time window in the weighted access frequency vector, the first-order difference result, mean, variance and autocorrelation coefficient of the time window;

[0020] The multidimensional feature vectors corresponding to multiple time windows are combined into a multidimensional feature matrix.

[0021] Optionally, clustering the feature vectors of each time window slice in the multidimensional feature matrix using a Gaussian mixture model to obtain a cluster label vector includes:

[0022] The probability of the feature vector of each time window being assigned to multiple clusters is calculated through the soft clustering method of Gaussian mixture model;

[0023] Through the mixed Gaussian distribution of the Gaussian mixture model, each cluster is described by a Gaussian distribution, parameterized by mean, covariance and mixing weight to form a probability density function, identify similar groups of access patterns, and cluster the feature vectors of each time window slice based on the identified similar groups. A cluster label is assigned to each time window to obtain a cluster label vector.

[0024] Optionally, inputting the cluster label vector into an access frequency prediction model to obtain a predicted access frequency value and a confidence interval for the data within a future preset time step includes:

[0025] Eliminate trends and periodicity through non-seasonal and seasonal differences, so that the cluster label vector meets the stationary requirements;

[0026] Use autocorrelation function and partial autocorrelation function to determine seasonal and non-seasonal parameters, and fit model parameters by least squares or maximum likelihood estimation;

[0027] Based on the stabilized cluster label vector and model parameters, the access frequency prediction value and confidence interval of the data within the preset time step in the future are predicted.

[0028] Optionally, determining a migration decision matrix based on the access frequency prediction value and the data storage cost parameter, and performing data migration based on the migration decision matrix includes:

[0029] Calculate the storage cost of different storage solutions based on the access frequency prediction value;

[0030] Solving the lowest-cost storage solution that satisfies the constraints is done by using an objective function of a linear programming model with constraints. The objective function considers the unit storage cost of storing data on high-speed media or low-cost media, the impact of migration frequency on total storage cost, and the cost of migrating frequently accessed data from low-cost media to high-speed media. The constraints include ensuring that the usage of the high-speed media does not exceed its total capacity and storing frequently accessed data on the high-speed media when the predicted access frequency of the frequently accessed data exceeds a preset threshold.

[0031] Based on the solved minimum cost storage solution, a migration decision matrix is ​​determined, and the storage gateway is triggered to perform data migration operations according to the migration decision matrix.

[0032] In a second aspect, an embodiment of the present invention provides a device for migrating frequently accessed data of big data, the device comprising:

[0033] A data access log processing module is used to obtain data access logs, pre-process the data access logs, and obtain a time window statistical matrix. The time window statistical matrix includes multiple time window slices, each time window slice includes a window start time, a window end time, and the number of data accesses within the time window;

[0034] A timeliness decay weighted calculation module is used to perform timeliness decay weighted calculation on the multiple data access times in the time window statistical matrix to obtain a weighted access frequency vector, wherein the weighting coefficient corresponding to the data access times of each time window slice is inversely proportional to the time difference between the time window slice and the current time;

[0035] A multi-dimensional feature matrix extraction module is used to extract the statistical features of multiple data access times based on the weighted access frequency vector to obtain a multi-dimensional feature matrix, where each row of the multi-dimensional feature matrix corresponds to the feature vector of a time window slice, and each column corresponds to a feature type;

[0036] A clustering label vector acquisition module is used to cluster the feature vectors of each time window slice in the multidimensional feature matrix using a Gaussian mixture model to obtain a clustering label vector;

[0037] An access frequency prediction module is used to input the cluster label vector into an access frequency prediction model to obtain access frequency prediction values ​​of the data within a preset time step in the future, where each access frequency prediction value corresponds to a confidence interval;

[0038] The data migration module is used to determine a migration decision matrix based on the access frequency prediction value and the data storage cost parameter, and perform data migration based on the migration decision matrix, wherein the migration decision matrix balances access speed and storage cost by dynamically adjusting the data storage location.

[0039] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0040] at least one processor;

[0041] a memory for storing the at least one processor-executable instruction;

[0042] The at least one processor is configured to execute the instructions to implement the method described in the first aspect.

[0043] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the method described in the first aspect.

[0044] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0045] The technical solution provided by the embodiment of the present invention can obtain a time window statistical matrix, that is, time series data, by analyzing the data access log; then, combined with timeliness attenuation weighted calculation, extraction of multidimensional feature matrix and time series clustering calculation, the access frequency prediction model is used to predict the access frequency prediction value of the data within a preset time step in the future, and a storage medium migration strategy is defined. The storage location of the data is dynamically adjusted according to the access frequency prediction value and the storage medium migration strategy, thereby reducing the complexity of data migration and migrating data with high access frequency to high-speed media to achieve high-speed access to data.

[0046] Furthermore, the access frequency prediction value within a preset future time step typically changes dynamically. If the access frequency prediction value of frequently accessed data decreases, the frequently accessed data stored on high-speed media can be migrated back to low-cost media. This invention not only meets data access speed requirements but also ensures low data storage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of an overall technical architecture provided by an embodiment of the present invention;

[0048] Figure 2 A flowchart of a frequently accessed data storage and migration method for big data provided by an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of a frequently accessed data storage and migration device for big data provided by an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be described in detail below through examples.

[0052] With the widespread adoption of large-scale big data storage systems like data lakes and data centers, and the continuous expansion of enterprise data, the requirements for physical data storage and access are increasing, leading to an exponential increase in hardware costs. Currently, data storage solutions in related technologies typically store data in fixed locations on big data storage media like data lakes and data centers. For example, data is stored in fixed locations on solid-state drives (SSDs) or mechanical hard drives (HDDs). This means that these data storage methods cannot dynamically adjust the data storage location.

[0053] In practical applications, some data is frequently accessed, often referred to as frequently accessed data. To achieve fast access to frequently accessed data, conventional data storage solutions require manual migration of frequently accessed data to high-speed storage areas, increasing labor costs and complexity.

[0054] In order to solve the above-mentioned technical problems existing in the related art, the present invention predicts and calculates the access frequency of data through computational analysis of data access logs, combined with dynamic time window slicing, timeliness attenuation weighted calculation, and time series clustering calculations, and defines a storage medium migration strategy. According to the predicted access frequency and storage medium migration strategy, frequently accessed data is dynamically and intelligently migrated to high-performance storage media (high-speed media, such as SSD) to achieve high-speed access to data.

[0055] The present invention is a big data storage solution based on data access frequency. Its purpose is to effectively identify the access frequency of data stored in the platform and store it in corresponding high-speed storage areas (high-speed media) according to its access frequency to improve space utilization efficiency and computing efficiency.

[0056] The present invention involves two concepts: the original library and the cache library. The cache library is used to store frequently accessed data, that is, to store frequently accessed data in high-speed media. The original library is used to store all data. The execution entity of the embodiment of the present invention can be a high-frequency access data system. For frequently accessed data, the high-frequency access data system will request the original library to migrate data to the cache library. When the data access frequency decreases, the high-frequency access data system will clear the less frequently accessed data in the cache library.

[0057] In order to describe the solution clearly, the overall technical solution provided by the embodiment of the present invention will be described in detail below in combination with the overall system architecture.

[0058] like Figure 1 As shown, the overall technical solution of the embodiment of the present invention may include the following steps:

[0059] 1. The business data used by the enterprise on a daily basis is uniformly collected in the enterprise business service data center, that is, the data lake. Figure 1 It can be seen that the data stored in the data storage system may include data in the financial field, data in the procurement field, data in the trade field, data in the marketing field, etc.

[0060] 2. Each business department requests the required business data from the data lake through the enterprise data service bus based on data needs. The enterprise data service bus records the data access log of each request to facilitate the subsequent calculation of the data request frequency. The data access log can include the request data cycle, request data conditions, etc.

[0061] 3. Data preprocessing is to collect data request details from data request log records, extract query items in the log, and clean the log data. Finally, by merging or deduplication, a continuous and readable data request standard detail is obtained. Then, multiple time window slices are extracted from the data request standard detail. Each time window slice includes: window start time, window end time and data access count. The final output window statistics matrix M∈R N×3 ,The window statistics matrix includes N rows, each row represents a time window slice, and each row contains the window start time, the window end time and the number of data accesses.

[0062] 4. Perform a time-attenuation weighted calculation on the window statistical matrix. Specifically, data is processed based on the time attenuation factor. Recent access records are more valuable for predicting future access trends. Therefore, their influence is enhanced through weighted calculation. Finally, a weighted access frequency vector is output to facilitate subsequent extraction of time series features.

[0063] That is, if the window start time and window end time in a time window slice are close to the current time, then the time decay factor of the data access count of the time window slice is low, and when performing the time-efficiency decay weighted calculation, the corresponding weighting coefficient of the data access count of the time window slice is large. Similarly, if the window start time and window end time in a time window slice are far from the current time, then the time decay factor of the data access count of the time window slice is high, and when performing the time-efficiency decay weighted calculation, the corresponding weighting coefficient of the data access count of the time window slice is small.

[0064] 5. By combining the weighted access frequency vector with methods such as first-order differences, mean, and variance, we extract the statistical features of the time series data, construct a multidimensional feature space, and ultimately output the feature matrix X. Each row of the feature matrix X corresponds to the feature vector of a time window slice, and each column corresponds to a feature type. To ensure a complete and clear description of the solution, the specific implementation method for obtaining the feature matrix X will be detailed in the following examples.

[0065] 6. Input the feature matrix X into the Gaussian mixture model GMM to identify similar groups of data access patterns to pre-divide storage categories. This process mainly includes: feature extraction, parameter estimation, cluster assignment, and finally obtain the cluster label vector The cluster label vector It can be called the time series data after clustering. In order to describe the scheme completely and clearly, the cluster label vector will be The specific implementation method is elaborated in detail.

[0066] 7. Cluster label vector Input the access frequency prediction model, analyze the time series characteristics of historical data through the access frequency prediction model, predict the future access frequency trend of the data, predict hot data (if the predicted access frequency is high, then the data is hot data), and finally output the predicted value sequence And confidence interval. Where t is the current time, is the predicted data access frequency at time t+1, is the predicted data access frequency at time t+h.

[0067] 8. After predicting the data access frequency in the future preset time h steps, perform data migration in combination with the storage medium migration strategy. The core goal is to balance performance (such as access speed) and storage cost (such as storage fee) by dynamically adjusting the data storage location. For example, frequently accessed data is migrated to high-speed media (such as SSD), and data with lower access frequency is migrated to low-cost media (such as HDD). That is, the storage level of the data is dynamically adjusted based on the predicted data access frequency. And when the access frequency of the frequently accessed data drops, the data with lower access frequency in the high-speed medium will be cleared. In order to make the solution description complete and clear, the specific implementation method of data migration based on the predicted data access frequency and the storage medium migration strategy will be described in detail in the following embodiments.

[0068] The technical solution provided by the embodiment of the present invention predicts and calculates the access frequency of data through computational analysis of data access logs, combined with dynamic time window slicing, time-attenuation weighted calculation, time series clustering and other calculations, and defines a "storage medium migration strategy". The storage location of the data is intelligently migrated according to the predicted access frequency and the storage medium migration strategy, thereby reducing the complexity of data migration and migrating data with high access frequency to high-speed media to achieve high-speed access to data.

[0069] After describing in detail the overall technical solution of an embodiment of the present invention, a frequently accessed data storage and migration method for big data provided by an embodiment of the present invention will be described in detail below.

[0070] like Figure 2 As shown, an embodiment of the present invention provides a method for migrating frequently accessed data storage for big data, which may include the following steps:

[0071] S210: Obtain data access logs, pre-process the data access logs, and obtain a time window statistical matrix.

[0072] The time window statistics matrix includes multiple time window slices, and each time window slice includes the window start time, the window end time, and the number of data accesses within the time window.

[0073] As an implementation of the embodiment of the present invention, preprocessing the data access log to obtain a time window statistical matrix may include the following steps, namely steps a1 to a3:

[0074] Step a1: collect data request details from the data access log, and extract data query items from the data request details.

[0075] Step a2: clean, merge, and deduplicate the data query items to obtain a continuously readable data request standard detail.

[0076] Step a3: extract multiple time window slices from the continuously readable data request standard details, and form a time window statistical matrix with the multiple time window slices.

[0077] Data collection is the process of obtaining data access logs from systems or devices to generate raw data. Time window slicing involves dividing data access logs into analyzable time segments to establish basic statistics. Specifically, by accumulating the number of accesses within a window, we quantify the access frequency within a specific time period, providing basic statistics for subsequent analyses (such as time decay and clustering).

[0078] Specifically, the data access log records the data access timestamp sequence {t1, t2, ..., t n}, Use sliding time window to calculate the number of visits: Where Δt is the window length, which can be 1 hour, 1 day, or a week, W is the window range (e.g., 2025.5.6-2025.5.8), II(accesS k ∈W) is the indicator function, if the access time access k If it is within the window W, the value is 1, otherwise it is 0. After obtaining the number of data accesses within each time window slice, the window statistics matrix M∈R can be output N×3 , each row contains [window start time, window end time, data access times].

[0079] S220 , performing time-attenuation weighted calculation on the number of accesses to multiple data in the time window statistical matrix to obtain a weighted access frequency vector.

[0080] The weighting coefficient corresponding to the number of data accesses for each time window slice is inversely proportional to the time difference between the time window slice and the current time.

[0081] After obtaining the time window statistical matrix, we input it into the time decay model, processing the data based on the time decay factor, which means that the value of data decreases over time. Recent access records are more valuable for predicting future access trends, so their influence needs to be strengthened through weighted calculations.

[0082] Specifically, the exponential decay model formula is: Where λ is the decay coefficient. The larger the λ value, the faster the weight decays (high timeliness requirement); the smaller the λ value, the more weight the historical data retains. λ can usually be set to 0.1-0.3, t now is the current timestamp, i.e. the current moment; t is the timestamp of historical access data, which is between the window start time and the window end time, e -λ is an exponential function that controls the weight decay rate. (t now -t) is larger, the older the data is, and the smaller the weight w(t). For example, if the time difference between the current time and the data access time is 5 days, when λ = 0.2, the weight is e -0.2×5 =e -1 ≈0.368 If the time difference is 10 days, the weight drops to e -2 ≈0.135

[0083] The final output weighted access frequency vector Among them, w(t1) is the weighted coefficient corresponding to C1, w(t2) is the weighted coefficient corresponding to C2, and w(t n ) is C n The corresponding weighting coefficient. The weighted access frequency vector reflects the actual access intensity of the current window and is the basis for subsequent feature calculations.

[0084] S230 , extracting statistical features of multiple data access times based on the weighted access frequency vector to obtain a multi-dimensional feature matrix.

[0085] In the multidimensional feature matrix, each row corresponds to the feature vector of a time window slice, and each column corresponds to a feature type.

[0086] The purpose of this step is to extract the statistical features of time series data and construct a multidimensional feature space. The algorithms used are differential transformation and statistical calculation.

[0087] As an implementation of the embodiment of the present invention, S230, based on the weighted access frequency vector, extracting statistical features of the number of data accesses to obtain a multi-dimensional feature matrix, may include the following steps, namely, steps b1 to b4:

[0088] Step b1, performing first-order difference calculation on the weighted access frequency vector, eliminating the trend component in the weighted access frequency vector, and transforming the weighted access frequency vector from a non-stationary sequence to a stationary sequence.

[0089] Step b2, calculating the statistics of the weighted access frequency vector; the statistics include mean, variance and autocorrelation coefficient;

[0090] Step b3: For each time window, a multidimensional feature vector of the time window is constructed based on the original value corresponding to the time window in the weighted access frequency vector, the first-order difference result, mean, variance and autocorrelation coefficient of the time window;

[0091] Step b4: compose a multidimensional feature matrix from the multidimensional feature vectors corresponding to the multiple time windows.

[0092] Specifically, first-order difference is a basic preprocessing method in time series analysis, which is used to eliminate the trend component in the data and transform the non-stationary series into a stationary series. The formula is: in, It is the result of the first-order difference, which represents the change between two adjacent time points; F t is the observed value of the time series at time t; F t-1 is the observed value of the time series at time t-1.

[0093] Statistical calculations include calculating mean μ, variance σ 2 And autocorrelation coefficient ACF(k). The final output feature matrix X∈R N ×5 , where N is the number of time window slices, 5 is the 5 feature dimensions extracted from each time window, and these 5 feature dimensions include [original value, first-order difference, μ, σ 2 , ACF(k)]. Each row of X corresponds to the feature vector of a time window slice, and each column corresponds to a feature type.

[0094] S240, clustering the feature vectors of each time window slice in the multidimensional feature matrix using a Gaussian mixture model to obtain a cluster label vector.

[0095] Specifically, after obtaining the multidimensional feature matrix X, the Gaussian mixture model is used to cluster the feature vectors of each time window slice in the multidimensional feature matrix to obtain the cluster label vector

[0096] As an implementation of an embodiment of the present invention, S240, clustering the feature vectors of each time window slice in the multidimensional feature matrix using a Gaussian mixture model to obtain a cluster label vector, can include the following steps, namely step c1 and step c2:

[0097] In step c1, the probability of the feature vector of each time window being assigned to multiple clusters is calculated by using the soft clustering method of the Gaussian mixture model.

[0098] In step c2, each cluster is described by a Gaussian distribution through the mixed Gaussian distribution of the Gaussian mixture model, parameterized by mean, covariance and mixing weight to form a probability density function, identify similar groups of access patterns, and cluster the feature vectors of each time window slice based on the identified similar groups, and assign a cluster label to each time window to obtain a cluster label vector.

[0099] This step uses the Gaussian mixture model (GMM), assuming that the data consists of a mixture of multiple Gaussian distributions (normal distributions). In time series clustering, GMM works through soft clustering and mixed Gaussian distributions, as follows:

[0100] 1. Soft clustering: Calculate the probability that the feature vector (such as mean, variance, autocorrelation coefficient, etc.) of each time window slice is assigned to multiple clusters, rather than hard-classifying it into a single cluster.

[0101] 2. Mixture Gaussian distribution: Each cluster is described by a Gaussian distribution, parameterized by mean (μ), covariance (Σ) and mixing weight (π) to form a probability density function.

[0102] Its role is to identify groups with similar access patterns and pre-divide storage categories.

[0103] The above probability density function can be expressed as Where K is the number of clusters, determined by the BIC criterion; π k is the mixture weight of the kth Gaussian distribution Indicates the probability ratio of the cluster in the overall data; is the mean vector of the kth Gaussian distribution, describing the center position of the cluster; ∑ k is the covariance matrix of the kth Gaussian distribution, reflecting the correlation between features, such as the correlation between access frequency and difference.

[0104] The specific implementation process is as follows: After obtaining the multi-dimensional feature matrix X, parameter estimation is performed. Specifically, the expectation maximization (EM) algorithm is used to optimize the GMM parameters. The E step is used to calculate each eigenvector The probability of belonging to each cluster (posterior probability). Step M: Update π based on the posterior probability k 、 and ∑ k , maximize the log-likelihood of the data. Next, perform cluster assignment. Specifically, assign cluster labels to each time window based on the posterior probability. Realize the recognition of time series patterns. Finally output the cluster label vector

[0105] S250: Input the cluster label vector into the access frequency prediction model to obtain the access frequency prediction value of the data within a future preset time step.

[0106] Among them, each access frequency prediction value corresponds to a confidence interval.

[0107] Specifically, the cluster label vector is input into the access frequency prediction model, and the time series characteristics of the historical data are analyzed based on the access frequency prediction model to predict the future trend of access volume changes and predict hot data.

[0108] As an implementation of an embodiment of the present invention, inputting the cluster label vector into the access frequency prediction model to obtain the access frequency prediction value and confidence interval of the data within a future preset time step may include the following steps, namely steps d1 to d3:

[0109] In step d1, the trend and periodicity are eliminated by non-seasonal and seasonal differences so that the cluster label vector meets the stationarity requirement.

[0110] Step d2: seasonal parameters and non-seasonal parameters are determined using the autocorrelation function and the partial autocorrelation function, and the model parameters are fitted using the least squares method or the maximum likelihood estimation method.

[0111] Step d3: Based on the stabilized cluster label vector and model parameters, the access frequency prediction value and confidence interval of the data within the future preset time step are predicted.

[0112] Specifically, using the seasonal ARIMA model, the model equation is:

[0113]

[0114] Where B is the lag operator; for the non-seasonal part, d is the difference order to eliminate trend; p is the order of the autoregressive (AR) term, which represents the linear relationship between the current value and the historical value; q is the order of the moving average (MA) term, which represents the linear relationship between the current value and the historical error; φ i is the autoregressive parameter, which represents the coefficient of the non-seasonal autoregressive term; θ j is the moving average parameter, which represents the coefficient of the non-seasonal moving average term. For the seasonal part, P and Q are the orders of the seasonal AR and MA terms, respectively; D is the seasonal difference order; s is the seasonal period. For example, if it is a daily period, s = 24, if it is a weekly period, s = 7; ∈ t is the error term, which obeys the white noise distribution. I is the autoregressive parameter, which represents the coefficient of the seasonal autoregressive term; θ J is the moving average parameter, which represents the coefficient of the seasonal moving average term.

[0115] The specific implementation steps can be as follows:

[0116] 1. Data stabilization: through non-seasonal differencing (1-B) d and seasonal differences (1-B S ) D Eliminate trends and periodicity to make the series meet the requirements of stationarity.

[0117] 2. Parameter estimation: Use the autocorrelation function (ACF) and partial autocorrelation function (PACF) to determine p, d, q and seasonal parameters P, D, Q, S; fit the model parameter φ by least squares or maximum likelihood estimation i ,θ j 、φ I ,θ J .

[0118] 3. Forecast generation: Based on historical data and model parameters, calculate the forecast value for the next h steps and confidence intervals, which are used for storage medium adjustment decisions. h can be determined based on actual conditions and is not limited here. The final output is the predicted value sequence and confidence intervals.

[0119] S260 , determining a migration decision matrix based on the access frequency prediction value and the data storage cost parameter, and performing data migration based on the migration decision matrix.

[0120] Among them, the migration decision matrix balances access speed and storage cost by dynamically adjusting the data storage location.

[0121] Specifically, after obtaining the access frequency prediction value of the data within the future preset time step through the access frequency prediction model, the storage medium migration strategy determines the data migration strategy based on the access frequency prediction value and the data storage cost parameter. This process is an optimization decision-making process for migrating data between different storage media (such as SSD, HDD, cloud storage). Its core goal is to balance performance (such as access speed) and cost (such as storage fee) by dynamically adjusting the data storage location. For example, frequently accessed data (data with a higher access frequency) is migrated to high-speed media (such as SSD), and data with a lower access frequency is migrated to low-cost media (such as HDD). And the access frequency prediction value within the future preset time step is usually dynamically changing. If the access frequency prediction value of the frequently accessed data becomes lower, then the frequently accessed data stored in the high-speed medium can be migrated back to the low-cost medium. In short, the present invention is based on the access frequency prediction value within the preset time step to dynamically adjust the storage location of the data, which can not only meet the data access speed, but also ensure that the data storage cost is low.

[0122] As an implementation of an embodiment of the present invention, determining a migration decision matrix based on the access frequency prediction value and the data storage cost parameter, and performing data migration based on the migration decision matrix may include the following steps, namely steps e1 to e3:

[0123] Step e1: Calculate the cost of different storage solutions based on the access frequency prediction value.

[0124] In step e2, the objective function of the linear programming model with constraints is used to solve the minimum cost storage solution that meets the constraints.

[0125] The objective function considers the unit storage cost of storing data on high-speed media or low-cost media, the impact of migration frequency on the total storage cost, and the cost of migrating frequently accessed data from low-cost media to high-speed media; the constraints include ensuring that the usage of high-speed media does not exceed its total capacity, and storing frequently accessed data on high-speed media when the predicted access frequency value of frequently accessed data is greater than a preset threshold.

[0126] The objective function of the linear programming model is represents the unit storage cost of the i-th type of data on SSD and HDD respectively; x i 、y i is a decision variable indicating whether to store data in SSD or HDD (values ​​are 0 or 1); γ is a weight coefficient that controls the impact of migration frequency on the total cost; To predict the frequency of future visits, high-frequency data ( Large) Penalty cost is reduced when migrating to SSD; is the migration flag (value is 0 or 1).

[0127] The constraints are:

[0128] Storage capacity limit: ∑x i ≤C ssd , ensure that the usage of the SSD does not exceed its total capacity C ssd .

[0129] Access latency requirements: When accessing the frequency prediction value When the threshold τ is exceeded, the data is forced to remain in the SSD to meet the low latency requirement.

[0130] Step e3: Based on the solved minimum cost storage solution, determine the migration decision matrix and trigger the storage gateway to perform data migration operations.

[0131] Specifically, after solving the lowest cost storage solution that meets the constraints, the migration decision matrix D∈{0,1} is determined based on the lowest cost storage solution solved.N×2 ,where the two columns represent SSD or HDD, respectively, marking the data to be migrated to SSD or HDD.

[0132] The technical solution provided by the embodiment of the present invention can obtain a time window statistical matrix, that is, time series data, by analyzing the data access log; then, combined with timeliness attenuation weighted calculation, extraction of multidimensional feature matrix and time series clustering calculation, the access frequency prediction model is used to predict the access frequency prediction value of the data within a preset time step in the future, and a storage medium migration strategy is defined. The storage location of the data is dynamically adjusted according to the access frequency prediction value and the storage medium migration strategy, thereby reducing the complexity of data migration and migrating data with high access frequency to high-speed media to achieve high-speed access to data.

[0133] Furthermore, the access frequency prediction value within a preset future time step typically changes dynamically. If the access frequency prediction value of frequently accessed data decreases, the frequently accessed data stored on high-speed media can be migrated back to low-cost media. This invention not only meets data access speed requirements but also ensures low data storage costs.

[0134] In a second aspect, an embodiment of the present invention provides a frequently accessed data storage and migration device 30 for big data, such as Figure 3 As shown, the device includes:

[0135] A data access log processing module 310 is configured to obtain data access logs, preprocess the data access logs, and obtain a time window statistical matrix. The time window statistical matrix includes multiple time window slices, each of which includes a window start time, a window end time, and the number of data accesses within the time window.

[0136] A timeliness decay weighted calculation module 320 is configured to perform a timeliness decay weighted calculation on the multiple data access counts in the time window statistical matrix to obtain a weighted access frequency vector, wherein the weighting coefficient corresponding to the data access count of each time window slice is inversely proportional to the time difference between the time window slice and the current time;

[0137] A multi-dimensional feature matrix extraction module 330 is used to extract statistical features of multiple data access times based on the weighted access frequency vector to obtain a multi-dimensional feature matrix, wherein each row of the multi-dimensional feature matrix corresponds to a feature vector of a time window slice, and each column corresponds to a feature type;

[0138] A cluster label vector acquisition module 340 is configured to cluster the feature vectors of each time window slice in the multidimensional feature matrix using a Gaussian mixture model to obtain a cluster label vector;

[0139] The access frequency prediction module 350 is used to input the cluster label vector into the access frequency prediction model to obtain the access frequency prediction value of the data within a preset time step in the future, and each access frequency prediction value corresponds to a confidence interval;

[0140] The data migration module 360 ​​is used to determine a migration decision matrix based on the access frequency prediction value and the data storage cost parameter, and perform data migration based on the migration decision matrix, wherein the migration decision matrix balances access speed and storage cost by dynamically adjusting the data storage location.

[0141] The embodiment of the present invention further provides an electronic device 400, such as Figure 4 As shown, including:

[0142] at least one processor 401;

[0143] a memory 402 for storing the at least one processor-executable instruction;

[0144] The at least one processor is configured to execute the instructions to implement the method described in the first aspect.

[0145] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the method described in the first aspect.

[0146] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0147] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and purpose of the present invention.

Claims

1. A method for migrating frequently accessed data storage for big data, characterized in that: The method comprises: Obtaining data access logs, preprocessing the data access logs to obtain a time window statistical matrix, where the time window statistical matrix includes multiple time window slices, each time window slice includes a window start time, a window end time, and the number of data accesses within the time window; Performing a time-attenuation weighted calculation on multiple data access times in the time window statistical matrix to obtain a weighted access frequency vector, wherein the weighting coefficient corresponding to the data access times of each time window slice is inversely proportional to the time difference between the time window slice and the current time; Based on the weighted access frequency vector, the statistical features of multiple data access times are extracted to obtain a multi-dimensional feature matrix, where each row of the multi-dimensional feature matrix corresponds to the feature vector of a time window slice, and each column corresponds to a feature type; Clustering the feature vectors of each time window slice in the multidimensional feature matrix using a Gaussian mixture model to obtain a cluster label vector; Input the cluster label vector into the access frequency prediction model to obtain the access frequency prediction value of the data within a preset time step in the future, and each access frequency prediction value corresponds to a confidence interval; Based on the access frequency prediction value and the data storage cost parameter, a migration decision matrix is ​​determined, and data migration is performed based on the migration decision matrix, wherein the migration decision matrix balances access speed and storage cost by dynamically adjusting the data storage location.

2. The method according to claim 1, characterized in that The data access log is preprocessed to obtain a time window statistical matrix, including: Collecting data request details from the data access log, and extracting data query items from the data request details; Cleaning, merging, and deduplication operations are performed on the data query items to obtain a continuously readable data request standard detail; A plurality of time window slices are extracted from the continuously readable data request standard details, and the plurality of time window slices are combined into a time window statistical matrix.

3. The method according to claim 1, characterized in that The statistical characteristics of the number of data accesses are extracted based on the weighted access frequency vector to obtain a multi-dimensional feature matrix, including: Perform first-order difference calculation on the weighted access frequency vector to eliminate the trend component in the weighted access frequency vector and transform the weighted access frequency vector from a non-stationary sequence to a stationary sequence; Calculating statistics of the weighted access frequency vector; the statistics include mean, variance and autocorrelation coefficient; For each time window, a multidimensional feature vector of the time window is composed based on the original value corresponding to the time window in the weighted access frequency vector, the first-order difference result, mean, variance and autocorrelation coefficient of the time window; The multidimensional feature vectors corresponding to multiple time windows are combined into a multidimensional feature matrix.

4. The method according to claim 1, wherein The Gaussian mixture model is used to cluster the feature vectors of each time window slice in the multidimensional feature matrix to obtain a cluster label vector, including: The probability of the feature vector of each time window being assigned to multiple clusters is calculated through the soft clustering method of Gaussian mixture model; Through the mixed Gaussian distribution of the Gaussian mixture model, each cluster is described by a Gaussian distribution, parameterized by mean, covariance and mixing weight to form a probability density function, identify similar groups of access patterns, and cluster the feature vectors of each time window slice based on the identified similar groups. A cluster label is assigned to each time window to obtain a cluster label vector.

5. The method according to claim 1, wherein Inputting the cluster label vector into the access frequency prediction model to obtain the access frequency prediction value and confidence interval of the data within a future preset time step includes: Eliminate trends and periodicity through non-seasonal and seasonal differences, so that the cluster label vector meets the stationary requirements; Use autocorrelation function and partial autocorrelation function to determine seasonal and non-seasonal parameters, and fit model parameters by least squares or maximum likelihood estimation; Based on the stabilized cluster label vector and model parameters, the access frequency prediction value and confidence interval of the data within the preset time step in the future are predicted.

6. The method according to any one of claims 1 to 5, characterized in that The step of determining a migration decision matrix based on the access frequency prediction value and the data storage cost parameter, and performing data migration based on the migration decision matrix, includes: Calculate the storage cost of different storage solutions based on the access frequency prediction value; Solving the lowest-cost storage solution that satisfies the constraints is done by using an objective function of a linear programming model with constraints. The objective function considers the unit storage cost of storing data on high-speed media or low-cost media, the impact of migration frequency on total storage cost, and the cost of migrating frequently accessed data from low-cost media to high-speed media. The constraints include ensuring that the usage of the high-speed media does not exceed its total capacity and storing frequently accessed data on the high-speed media when the predicted access frequency of the frequently accessed data exceeds a preset threshold. Based on the solved minimum cost storage solution, a migration decision matrix is ​​determined, and the storage gateway is triggered to perform data migration operations according to the migration decision matrix.

7. A frequently accessed data storage and migration device for big data, characterized in that: The device comprises: A data access log processing module is used to obtain data access logs, pre-process the data access logs, and obtain a time window statistical matrix. The time window statistical matrix includes multiple time window slices, each time window slice includes a window start time, a window end time, and the number of data accesses within the time window; A timeliness decay weighted calculation module is used to perform timeliness decay weighted calculation on the multiple data access times in the time window statistical matrix to obtain a weighted access frequency vector, wherein the weighting coefficient corresponding to the data access times of each time window slice is inversely proportional to the time difference between the time window slice and the current time; A multi-dimensional feature matrix extraction module is used to extract the statistical features of multiple data access times based on the weighted access frequency vector to obtain a multi-dimensional feature matrix, where each row of the multi-dimensional feature matrix corresponds to the feature vector of a time window slice, and each column corresponds to a feature type; A clustering label vector acquisition module is used to cluster the feature vectors of each time window slice in the multidimensional feature matrix using a Gaussian mixture model to obtain a clustering label vector; An access frequency prediction module is used to input the cluster label vector into an access frequency prediction model to obtain access frequency prediction values ​​of the data within a preset time step in the future, where each access frequency prediction value corresponds to a confidence interval; The data migration module is used to determine a migration decision matrix based on the access frequency prediction value and the data storage cost parameter, and perform data migration based on the migration decision matrix, wherein the migration decision matrix balances access speed and storage cost by dynamically adjusting the data storage location.

8. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when the computer program is executed by a processor.

Images