A method and apparatus for big data frequent visit data storage migration

Abstract

The application belongs to the field of data storage, and discloses a method and device for migrating frequently-visited data storage of big data, which comprises the following steps: obtaining a data access log, preprocessing the data access log to obtain a time window statistical matrix; performing time-efficiency decay weighting calculation on a plurality of data access times in the time window statistical matrix to obtain a weighted access frequency vector; obtaining a multi-dimensional feature matrix based on the weighted access frequency vector; clustering the feature vectors of each time window slice in the multi-dimensional feature matrix respectively through a Gaussian mixture model to obtain a clustering label vector; inputting the clustering label vector into an access frequency prediction model to obtain an access frequency prediction value of data in a future preset time step; determining a migration decision matrix based on the access frequency prediction value and a data storage cost parameter, and performing data migration based on the migration decision matrix; and the application can reduce the complexity of data migration and balance the access speed and storage cost.

Description

Technical Field

[0001] This invention belongs to the field of data storage technology, and specifically relates to a method and apparatus for frequent data storage and migration of big data. Background Technology

[0002] With the widespread adoption of large-scale big data storage systems such as data lakes and data middle platforms, and the continuous expansion of enterprise data, the physical storage and access requirements for data are also constantly increasing, leading to a significant increase in hardware costs. Currently, data storage solutions in related technologies typically store data in a fixed location on big data storage media such as data lakes and data middle platforms, meaning that the data storage location cannot be dynamically adjusted.

[0003] In practical applications, some data is frequently accessed and can be referred to as frequently accessed data. To achieve fast access to frequently accessed data, related data storage solutions often require manual migration to high-speed storage areas, leading to increased labor costs and high complexity in data migration. Summary of the Invention

[0004] The purpose of this invention is to reduce the labor costs and complexity of data migration, while ensuring low data storage costs while meeting data access speed requirements.

[0005] In a first aspect, embodiments of the present invention provide a method for frequent data storage and migration for big data, the method comprising:

[0006] Obtain data access logs, preprocess the data access logs to obtain a time window statistics matrix, the time window statistics matrix includes multiple time window slices, each time window slice includes the window start time, the window end time and the number of data accesses within the time window;

[0007] The time-sensitivity attenuation weighted calculation is performed on the number of data accesses in the time window statistical matrix to obtain a weighted access frequency vector. The weighting coefficient corresponding to the number of data accesses in each time window slice is inversely proportional to the time difference between the time window slice and the current time.

[0008] Based on the weighted access frequency vector, statistical features of multiple data access times are extracted to obtain a multidimensional feature matrix. Each row of the multidimensional feature matrix corresponds to the feature vector of a time window slice, and each column corresponds to a feature type.

[0009] The feature vectors of each time window slice in the multidimensional feature matrix are clustered using a Gaussian mixture model to obtain cluster label vectors.

[0010] The clustering label vector is input into the access frequency prediction model to obtain the predicted access frequency of the data within a preset time step in the future. Each predicted access frequency value corresponds to a confidence interval.

[0011] Based on the predicted access frequency and data storage cost parameters, a migration decision matrix is ​​determined, and data migration is performed based on the migration decision matrix. The migration decision matrix balances access speed and storage cost by dynamically adjusting the data storage location.

[0012] Optionally, the preprocessing of the data access logs to obtain a time window statistical matrix includes:

[0013] Collect data request details from the data access log, and extract data query items from the data request details;

[0014] The data query items are cleaned, merged, and deduplicated to obtain a continuous and readable data request standard detail.

[0015] Multiple time window slices are extracted from the continuously readable data request standard details, and the multiple time window slices are combined into a time window statistical matrix.

[0016] Optionally, the step of extracting statistical features of data access frequency based on a weighted access frequency vector to obtain a multi-dimensional feature matrix includes:

[0017] The first-order difference calculation is performed on the weighted access frequency vector to eliminate the trend component in the weighted access frequency vector, thus transforming the weighted access frequency vector from a non-stationary sequence into a stationary sequence.

[0018] Calculate the statistics of the weighted access frequency vector; the statistics include the mean, variance, and autocorrelation coefficient.

[0019] For each time window, the first-order difference result, mean, variance and autocorrelation coefficient of the time window are composed of the original value corresponding to the time window in the weighted access frequency vector;

[0020] The multidimensional feature vectors corresponding to multiple time windows are combined to form a multidimensional feature matrix.

[0021] Optionally, the step of clustering the feature vectors of each time window slice in the multidimensional feature matrix using a Gaussian mixture model to obtain cluster label vectors includes:

[0022] The probability of the feature vector of each time window being assigned to multiple clusters is calculated using the soft clustering method of Gaussian mixture model.

[0023] By using the Gaussian mixture model with a mixture Gaussian distribution, each cluster is described by a Gaussian distribution. The probability density function is parameterized by the mean, covariance, and mixture weights. Similar groups of access patterns are identified, and the feature vectors of each time window slice are clustered based on the identified similar groups. Cluster labels are assigned to each time window to obtain cluster label vectors.

[0024] Optionally, the step of inputting the clustering label vector into the access frequency prediction model to obtain the predicted access frequency value and confidence interval of the data within a preset time step includes:

[0025] By eliminating trends and periodicity through non-seasonal and seasonal differencing, the cluster label vectors can meet the stationarity requirement.

[0026] Seasonal and non-seasonal parameters are determined using autocorrelation and partial autocorrelation functions, and the model parameters are fitted using the least squares method or the maximum likelihood estimation method.

[0027] Based on the stabilized cluster label vectors and model parameters, the predicted access frequency and confidence interval of the data within a preset time step are calculated.

[0028] Optionally, determining the migration decision matrix based on the access frequency prediction value and data storage cost parameters, and performing data migration based on the migration decision matrix, includes:

[0029] Calculate the storage cost of different storage solutions based on the predicted access frequency.

[0030] The objective function of a constrained linear programming model is used to solve for the lowest cost storage scheme that satisfies the constraints. The objective function considers the unit storage cost of data stored in high-speed media or low-cost media, the impact of migration frequency on total storage cost, and the cost of migrating frequently accessed data from low-cost media to high-speed media. The constraints include ensuring that the usage of high-speed media does not exceed its total capacity, and storing frequently accessed data in high-speed media when the predicted access frequency of frequently accessed data is greater than a preset threshold.

[0031] Based on the solved lowest-cost storage scheme, a migration decision matrix is ​​determined, and the storage gateway is triggered to perform data migration operations according to the migration decision matrix.

[0032] Secondly, embodiments of the present invention provide a frequent access data storage and migration apparatus for big data, the apparatus comprising:

[0033] The data access log processing module is used to acquire data access logs, preprocess the data access logs, and obtain a time window statistics matrix. The time window statistics matrix includes multiple time window slices, and each time window slice includes the window start time, the window end time, and the number of data accesses within the time window.

[0034] The time-sensitivity decay weighted calculation module is used to perform time-sensitivity decay weighted calculation on the number of data accesses in the time window statistical matrix to obtain a weighted access frequency vector, wherein the weighting coefficient corresponding to the number of data accesses in each time window slice is inversely proportional to the time difference between the time window slice and the current time.

[0035] The multidimensional feature matrix extraction module is used to extract statistical features of multiple data access times based on the weighted access frequency vector to obtain a multidimensional feature matrix. Each row of the multidimensional feature matrix corresponds to the feature vector of a time window slice, and each column corresponds to a feature type.

[0036] The clustering label vector acquisition module is used to cluster the feature vectors of each time window slice in the multidimensional feature matrix using a Gaussian mixture model to obtain clustering label vectors.

[0037] The access frequency prediction module is used to input the clustering label vector into the access frequency prediction model to obtain the access frequency prediction value of the data within a preset time step in the future. Each access frequency prediction value corresponds to a confidence interval.

[0038] The data migration module is used to determine a migration decision matrix based on the predicted access frequency and data storage cost parameters, and to perform data migration based on the migration decision matrix. The migration decision matrix balances access speed and storage cost by dynamically adjusting the data storage location.

[0039] Thirdly, embodiments of the present invention provide an electronic device, including:

[0040] At least one processor;

[0041] Memory for storing the at least one processor-executable instruction;

[0042] The at least one processor is configured to execute the instructions to implement the method described in the first aspect.

[0043] Fourthly, embodiments of the present invention provide a computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method described in the first aspect.

[0044] Fifthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0045] The technical solution provided by this invention analyzes data access logs to obtain a time window statistical matrix, i.e., time-series data. Then, by combining time-sensitivity attenuation weighted calculation, extraction of multi-dimensional feature matrices, and time-series clustering, the access frequency prediction model predicts the access frequency of the data within a preset time step in the future. A storage medium migration strategy is defined, and the storage location of the data is dynamically adjusted according to the access frequency prediction and the storage medium migration strategy, which reduces the complexity of data migration and migrates high-frequency data to high-speed media to achieve high-speed data access.

[0046] Furthermore, the predicted access frequency within a preset time step is typically dynamic. If the predicted access frequency of frequently accessed data becomes low, the data stored on high-speed media can be migrated back to low-cost media. This invention not only satisfies data access speed requirements but also ensures low data storage costs. Attached Figure Description

[0047] Figure 1 A schematic diagram of an overall technical architecture provided for an embodiment of the present invention;

[0048] Figure 2 A flowchart of a method for frequent access data storage and migration for big data, provided as an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of a frequent access data storage and migration device for big data provided in an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0051] The present invention will be described in detail below through embodiments.

[0052] With the widespread adoption of large-scale big data storage systems such as data lakes and data middleware, and the continuous expansion of enterprise data, the physical storage and access requirements for data are also constantly increasing, leading to a significant increase in hardware costs. Currently, data storage solutions in related technologies typically store data in a fixed location on big data storage media such as data lakes and data middleware, for example, on solid-state drives (SSDs) or hard disk drives (HDDs). In other words, these data storage methods cannot dynamically adjust the data storage location.

[0053] In practical applications, some data is frequently accessed and can be referred to as frequently accessed data. To achieve fast access to frequently accessed data, related data storage solutions often require manual migration to high-speed storage areas, leading to increased labor costs and high complexity in data migration.

[0054] To address the aforementioned technical problems in related technologies, this invention analyzes data access logs and combines dynamic time window slicing, time-dependent attenuation weighted calculation, and time-series clustering to predict and calculate the data access frequency. It also defines a storage media migration strategy. Based on the predicted access frequency and the storage media migration strategy, it dynamically and intelligently migrates frequently accessed data to high-performance storage media (high-speed media, such as SSDs) to achieve high-speed data access.

[0055] This invention is a big data storage scheme based on data access frequency. The purpose is to effectively identify the data access frequency stored in the platform and store it in the corresponding high-speed storage area (high-speed medium) according to its access frequency, so as to improve space utilization efficiency and computing efficiency.

[0056] This invention involves two concepts: a primary database and a cache database. The cache database stores frequently accessed data; that is, it stores frequently accessed data on high-speed media. The primary database stores all data. In this embodiment, the execution entity can be a high-frequency access data system. For frequently accessed data, the high-frequency access data system requests data migration from the primary database to the cache database. When the data access frequency decreases, the high-frequency access data system cleans up the less frequently accessed data in the cache database.

[0057] To ensure clarity, the overall technical solution provided by the embodiments of the present invention will be described in detail below in conjunction with the overall system architecture.

[0058] like Figure 1 As shown, the overall technical solution of this invention embodiment may include the following steps:

[0059] 1. Business data used daily by enterprises is uniformly collected into the enterprise business service data platform, i.e., the data lake. From Figure 1 It is known that the data stored in the data center can include data from the financial field, procurement field, trade field, marketing field, etc.

[0060] 2. Each business unit requests the required business data from the data lake through the enterprise data service bus according to its data needs. The enterprise data service bus records the data access log for each request, which facilitates the calculation of the data request frequency later. The data access log may include the data request period, data request conditions, etc.

[0061] 3. Data preprocessing involves collecting detailed data request items from the data request log and extracting query items from the logs. Simultaneously, the log data is cleaned, and finally, through merging or deduplication, continuous and readable standard data request details are obtained. Then, multiple time window slices are extracted from these standard data request details. Each time window slice includes: window start time, window end time, and data access count. The final output is a window statistics matrix M∈R. N×3 The window statistics matrix consists of N rows, each representing a time window slice. Each row contains the window start time, window end time, and number of data accesses.

[0062] 4. Perform time-related decay weighted calculation on the window statistics matrix. Specifically, the data is processed based on the time decay factor. Recent access records have higher reference value for predicting future access trends. Therefore, the influence is strengthened by weighted calculation. Finally, a weighted access frequency vector is output, which is convenient for subsequent extraction of time series features.

[0063] In other words, if the start and end times of a time window slice are close to the current time, then the time decay factor for the data access count of that time window slice is low, and the weighting coefficient corresponding to the data access count of that time window slice is large when performing time-sensitivity decay weighted calculation. Similarly, if the start and end times of a time window slice are far from the current time, then the time decay factor for the data access count of that time window slice is high, and the weighting coefficient corresponding to the data access count of that time window slice is small when performing time-sensitivity decay weighted calculation.

[0064] 5. By combining weighted access frequency vectors with first-order difference, mean, and variance methods, statistical features of time-series data are extracted to construct a multi-dimensional feature space, ultimately outputting a feature matrix X. Each row of the feature matrix X corresponds to a feature vector of a time window slice, and each column corresponds to a feature type. For the sake of completeness and clarity, the specific implementation of how to obtain the feature matrix X will be described in detail in the following embodiments.

[0065] 6. Input the feature matrix X into a Gaussian Mixture Model (GMM) to identify similar groups of data access patterns for pre-classification of storage categories. This process mainly includes: feature extraction, parameter estimation, cluster assignment, and finally, obtaining cluster label vectors. This clustering label vector This can be referred to as time-series data after clustering. For a complete and clear description of the scheme, the clustering label vectors will be discussed in the following examples. The specific implementation method will be explained in detail.

[0066] 7. Cluster label vectors Input the access frequency prediction model, analyze the time series characteristics of historical data through the access frequency prediction model, predict the future trend of data access frequency changes, and identify hot data (if the predicted access frequency is high, then the data is hot data), and finally output the predicted value sequence. and the confidence interval. Where t is the current time, The predicted data access frequency at time t+1. This represents the predicted data access frequency at time t+h.

[0067] 8. After predicting the data access frequency over a predetermined time interval h, data migration is performed in conjunction with a storage media migration strategy. The core objective is to balance performance (e.g., access speed) and storage cost (e.g., storage fees) by dynamically adjusting the data storage location. For example, frequently accessed data is migrated to high-speed media (e.g., SSDs), while less frequently accessed data is migrated to low-cost media (e.g., HDDs). In other words, the storage tier of data is dynamically adjusted based on the predicted data access frequency. Furthermore, when the access frequency of frequently accessed data decreases, less frequently accessed data in the high-speed media is removed. To ensure a complete and clear description of the solution, the specific implementation of data migration based on predicted data access frequency and the storage media migration strategy will be detailed in the following embodiments.

[0068] The technical solution provided by this invention analyzes data access logs and combines calculations such as dynamic time window slicing, timeliness attenuation weighted calculation, and time-series clustering to predict the data access frequency. It then defines a "storage media migration strategy" and intelligently migrates the data storage location based on the predicted access frequency and the storage media migration strategy. This reduces the complexity of data migration and migrates frequently accessed data to high-speed media to achieve high-speed data access.

[0069] After a detailed description of the overall technical solution of the embodiments of the present invention, the following will describe in detail a method for frequent access data storage and migration for big data provided by the embodiments of the present invention.

[0070] like Figure 2 As shown in the figure, an embodiment of the present invention provides a method for frequent data storage and migration for big data, which may include the following steps:

[0071] S210: Obtain data access logs, preprocess the data access logs, and obtain a time window statistical matrix.

[0072] The time window statistics matrix includes multiple time window slices, each of which includes the window start time, window end time, and the number of data accesses within the time window.

[0073] As one implementation of this invention, preprocessing the data access logs to obtain a time window statistical matrix may include the following steps, namely steps a1 to a3:

[0074] Step a1: Collect data request details from the data access log and extract data query items from the data request details.

[0075] Step a2 involves cleaning, merging, and deduplicating the data query items to obtain a continuous and readable standard detail of data requests.

[0076] Step a3: Extract multiple time window slices from the continuously readable data request standard details, and combine the multiple time window slices into a time window statistical matrix.

[0077] Data acquisition is the process of obtaining raw data from a system or device, specifically data access logs. Time window slicing involves dividing the data access logs into analyzable time segments to establish basic statistics. Specifically, by accumulating the number of accesses within the window, the access frequency over a specific time period is quantified, providing basic statistics for subsequent analyses (such as timeliness decay and clustering).

[0078] Specifically, the data access log records a data access timestamp sequence {t1, t2, ..., t...} n The sliding time window is used to calculate the number of window visits. Where Δt is the window length, which can be 1 hour, 1 day, or 1 week, etc., W is the window range (e.g., 2025.5.6-2025.5.8), II (accesS k ∈W) is an indicator function, if the access time is access... k Within window W, the value is 1; otherwise, it is 0. After counting the number of data accesses within each time window slice, the window statistics matrix M∈R can be output. N×3 Each line contains [window start time, window end time, number of data accesses].

[0079] S220 performs time-dependent attenuation weighted calculation on the number of accesses to multiple data in the time window statistical matrix to obtain a weighted access frequency vector.

[0080] The weighting coefficient for the number of data accesses in each time window slice is inversely proportional to the time difference between the time window slice and the current time.

[0081] After obtaining the time window statistical matrix, it is input into the time-sensitivity decay model. The data is processed based on the time decay factor, meaning that the value of data decreases over time. Recent access records are more valuable for predicting future access trends, therefore their influence needs to be strengthened through weighted calculations.

[0082] Specifically, the formula for the exponential decay model is: Where λ is the attenuation coefficient, the larger the value of λ, the faster the weight decays (for applications with high timeliness requirements); the smaller the value of λ, the more weight is retained from historical data. λ can typically be set between 0.1 and 0.3, t now `t` is the current timestamp, i.e., the current moment; `t` is the timestamp of historical access data, located between the window start time and the window end time; `e`... -λ It is an exponential function, controlling the rate of weight decay. (t) now The larger the value of λ(t), the older the data, and the smaller the weight w(t). For example, if the time difference between the current time and the data access time is 5 days, when λ = 0.2, the weight is e. -0.2×5 =e -1 ≈0.368 If the time difference is 10 days, the weight decreases to e. -2 ≈0.135

[0083] The final output is a weighted access frequency vector. Where w(t1) is the weighting coefficient corresponding to C1, w(t2) is the weighting coefficient corresponding to C2, and w(t) is the weighting coefficient corresponding to C2. n ) is C n The corresponding weighting coefficients. The weighted access frequency vector reflects the actual access intensity of the current window and is the basis for subsequent feature calculations.

[0084] S230, based on the weighted access frequency vector, extracts statistical features of multiple data access times to obtain a multidimensional feature matrix.

[0085] In this multidimensional feature matrix, each row corresponds to a feature vector of a time window slice, and each column corresponds to a feature type.

[0086] The purpose of this step is to extract the statistical features of time series data and construct a multidimensional feature space. The algorithm used is difference transformation and statistical calculation.

[0087] As one implementation of this invention, S230, based on the weighted access frequency vector, extracts statistical features of the number of data accesses to obtain a multi-dimensional feature matrix, which may include the following steps, namely steps b1 to b4:

[0088] Step b1 involves performing a first-order difference calculation on the weighted access frequency vector to eliminate the trend component in the weighted access frequency vector, transforming the weighted access frequency vector from a non-stationary sequence into a stationary sequence.

[0089] Step b2, calculate the statistics of the weighted access frequency vector; the statistics include the mean, variance and autocorrelation coefficient;

[0090] Step b3: For each time window, based on the original value corresponding to the time window in the weighted access frequency vector, the first difference result of the time window, the mean, the variance and the autocorrelation coefficient, a multidimensional feature vector of the time window is formed.

[0091] Step b4: Combine the multidimensional feature vectors corresponding to multiple time windows into a multidimensional feature matrix.

[0092] Specifically, first-order differencing is a fundamental preprocessing method in time series analysis, used to eliminate trend components in data and transform non-stationary series into stationary series. The formula is: in, It is the result after first-order difference, representing the change between two adjacent time points; F t It is the observation value of the time series at time t; F t-1 These are the observations of the time series at time t-1.

[0093] Statistical calculations include calculating the mean μ and variance σ. 2 And the autocorrelation coefficient ACF(k). The final output feature matrix X∈R N ×5 Where N is the number of time window slices, and 5 represents the 5 feature dimensions extracted from each time window. These 5 feature dimensions include [original value, first difference, μ, σ]. 2 [ACF(k)]. Each row of X corresponds to a feature vector of a time window slice, and each column corresponds to a feature type.

[0094] S240 uses a Gaussian mixture model to cluster the feature vectors of each time window slice in the multidimensional feature matrix to obtain cluster label vectors.

[0095] Specifically, after obtaining the multidimensional feature matrix X, a Gaussian mixture model is used to cluster the feature vectors of each time window slice in the multidimensional feature matrix to obtain cluster label vectors.

[0096] As one implementation of this invention, S240, clustering the feature vectors of each time window slice in the multidimensional feature matrix using a Gaussian mixture model to obtain cluster label vectors, may include the following steps, namely steps c1 and c2:

[0097] Step c1: Calculate the probability that the feature vector of each time window is assigned to multiple clusters using the soft clustering method of Gaussian mixture model.

[0098] Step c2 involves using the Gaussian mixture model to describe each cluster by a Gaussian distribution. The cluster is then parameterized by the mean, covariance, and mixture weights to form a probability density function. Similar groups of access patterns are identified, and the feature vectors of each time window slice are clustered based on the identified similar groups. Cluster labels are then assigned to each time window to obtain a cluster label vector.

[0099] This step uses a Gaussian Mixture Model (GMM), assuming the data consists of a mixture of multiple Gaussian distributions (normal distributions). In time-series clustering, GMM works through soft clustering and Gaussian mixture distributions, as follows:

[0100] 1. Soft clustering. Calculates the probability that the feature vectors (such as mean, variance, autocorrelation coefficient, etc.) of each time window slice will be assigned to multiple clusters, rather than rigidly dividing them into a single cluster.

[0101] 2. Mixture Gaussian Distribution: Each cluster is described by a Gaussian distribution, which is parameterized by the mean (μ), covariance (Σ), and mixture weights (π) to form a probability density function.

[0102] Its function is to identify groups with similar access patterns and pre-classify storage categories.

[0103] The above probability density function can be expressed as: Where K is the number of clusters, determined by the BIC criterion; π k Mixed weights for the k-th Gaussian distribution This indicates the probability percentage of the cluster in the overall data; Let be the mean vector of the k-th Gaussian distribution, describing the center location of the cluster; ∑ k Let be the covariance matrix of the k-th Gaussian distribution, reflecting the correlation between features, such as the correlation between access frequency and difference.

[0104] The specific implementation process is as follows: After obtaining the multidimensional feature matrix X, parameter estimation is performed. Specifically, the Expectation-Maximization (EM) algorithm is used to optimize the GMM parameters. The E-step is used to calculate each eigenvector. The probability of belonging to each cluster (posterior probability). M-step: Update π based on the posterior probability. k , and ∑ k First, maximize the log-likelihood of the data. Next, perform cluster assignment; specifically, assign cluster labels to each time window based on the posterior probability. Implement temporal pattern recognition. The final output is a clustering label vector.

[0105] S250: Input the clustering label vector into the access frequency prediction model to obtain the predicted access frequency value of the data within a preset time step in the future.

[0106] Each predicted access frequency value corresponds to a confidence interval.

[0107] Specifically, the clustering label vectors are input into the access frequency prediction model. Based on the access frequency prediction model, the time series characteristics of historical data are analyzed to predict future access volume trends and identify hot data.

[0108] As one implementation of this invention, inputting the clustering label vector into the access frequency prediction model to obtain the predicted access frequency value and confidence interval of the data within a preset future time step may include the following steps, namely steps d1 to d3:

[0109] Step d1 eliminates trends and periodicity through non-seasonal and seasonal differencing, ensuring that the cluster label vectors meet the stationarity requirement.

[0110] Step d2: Use the autocorrelation function and partial autocorrelation function to determine the seasonal and non-seasonal parameters, and fit the model parameters using the least squares method or the maximum likelihood estimation method.

[0111] Step d3: Based on the stabilized cluster label vectors and model parameters, predict the access frequency and confidence interval of the data within a preset time step in the future.

[0112] Specifically, a seasonal ARIMA model is used, and the model equation is as follows:

[0113]

[0114] Where B is the lag operator; for the non-seasonal part, d is the difference order, eliminating trends; p is the order of the autoregressive (AR) term, representing the linear relationship between the current value and historical values; q is the order of the moving average (MA) term, representing the linear relationship between the current value and historical errors; φ i θ is the autoregressive parameter, representing the coefficient of the non-seasonal autoregressive term; j These are the moving average parameters, representing the coefficients of the non-seasonal moving average term. For the seasonal component, P and Q are the orders of the seasonal AR and MA terms, respectively; D is the seasonal difference order; and s is the seasonal period. For example, if it's a daily period, s = 24; if it's a weekly period, s = 7. t The error term follows a white noise distribution. φ I θ is the autoregressive parameter, representing the coefficient of the seasonal autoregressive term; J It is a moving average parameter, representing the coefficient of the seasonal moving average term.

[0115] The specific implementation steps are as follows:

[0116] 1. Data Stabilization: Through non-seasonal differencing (1-B) d and seasonal differences (1-B) S ) D Eliminate trends and periodicity to make the sequence meet the stationarity requirement.

[0117] 2. Parameter estimation: Use the autocorrelation function (ACF) and partial autocorrelation function (PACF) to determine p, d, q and seasonal parameters P, D, Q, S; fit the model parameters φ using the least squares method or the maximum likelihood estimation method. i θ j φ I θ J .

[0118] 3. Prediction Generation: Based on historical data and model parameters, calculate the predicted value for the next h steps. The confidence interval is used for storage medium adjustment decisions. h can be determined based on actual conditions and is not limited here. The final output is a sequence of predicted values. and confidence interval.

[0119] S260 determines the migration decision matrix based on the predicted access frequency and data storage cost parameters, and performs data migration based on the migration decision matrix.

[0120] Among them, the migration decision matrix balances access speed and storage cost by dynamically adjusting the data storage location.

[0121] Specifically, after obtaining the predicted access frequency of data within a preset time step through an access frequency prediction model, the storage media migration strategy determines the data migration strategy based on the predicted access frequency and data storage cost parameters. This process is an optimized decision-making process for migrating data between different storage media (such as SSDs, HDDs, and cloud storage). Its core objective is to balance performance (such as access speed) and cost (such as storage fees) by dynamically adjusting the data storage location. For example, frequently accessed data (data with high access frequency) is migrated to high-speed media (such as SSDs), and less frequently accessed data is migrated to low-cost media (such as HDDs). Moreover, the predicted access frequency within the preset time step is usually dynamically changing. If the predicted access frequency of frequently accessed data becomes lower, then the frequently accessed data stored on high-speed media can be migrated back to low-cost media. In summary, this invention dynamically adjusts the data storage location based on the predicted access frequency within a preset time step, which not only meets the data access speed requirements but also ensures low data storage costs.

[0122] As one implementation of this invention, determining a migration decision matrix based on access frequency prediction and data storage cost parameters, and performing data migration based on the migration decision matrix, may include the following steps, namely steps e1 to e3:

[0123] Step e1: Calculate the cost of different storage solutions based on the predicted access frequency.

[0124] Step e2: Solve for the lowest cost storage scheme that satisfies the constraints using the objective function of the constrained linear programming model.

[0125] The objective function considers the unit storage cost of data stored in high-speed or low-cost media, the impact of migration frequency on total storage cost, and the cost of migrating frequently accessed data from low-cost media to high-speed media. The constraints include ensuring that the usage of high-speed media does not exceed its total capacity, and storing frequently accessed data in high-speed media when the predicted access frequency of the frequently accessed data is greater than a preset threshold.

[0126] The objective function of the linear programming model is x represents the unit storage cost of the i-th type of data on SSD and HDD, respectively; i y i γ is the decision variable, indicating whether to store data on SSD or HDD (value is 0 or 1); γ is the weighting coefficient, controlling the impact of migration frequency on total cost. For predicting future access frequency, high-frequency data ( The penalty cost is reduced when migrating from large (large) disks to SSDs; This is the migration flag (value can be 0 or 1).

[0127] The constraints are:

[0128] Storage capacity limit: ∑x i ≤C ssd Ensure that the amount of SSD used does not exceed its total capacity (C). ssd .

[0129] Access latency requirements: When the predicted access frequency value When the threshold τ is exceeded, the data is forced to remain on the SSD to meet the low latency requirement.

[0130] Step e3: Based on the solved lowest-cost storage scheme, determine the migration decision matrix and trigger the storage gateway to perform data migration operations.

[0131] Specifically, after finding the lowest-cost storage scheme that satisfies the constraints, the migration decision matrix D∈{0,1} is determined based on the found lowest-cost storage scheme.N×2 The two columns represent SSD or HDD respectively, indicating that the data should be migrated to SSD or HDD.

[0132] The technical solution provided by this invention analyzes data access logs to obtain a time window statistical matrix, i.e., time-series data. Then, by combining time-sensitivity attenuation weighted calculation, extraction of multi-dimensional feature matrices, and time-series clustering, the access frequency prediction model predicts the access frequency of the data within a preset time step in the future. A storage medium migration strategy is defined, and the storage location of the data is dynamically adjusted according to the access frequency prediction and the storage medium migration strategy, which reduces the complexity of data migration and migrates high-frequency data to high-speed media to achieve high-speed data access.

[0133] Furthermore, the predicted access frequency within a preset time step is typically dynamic. If the predicted access frequency of frequently accessed data becomes low, the data stored on high-speed media can be migrated back to low-cost media. This invention not only satisfies data access speed requirements but also ensures low data storage costs.

[0134] Secondly, embodiments of the present invention provide a frequent access data storage and migration device 30 for big data, such as... Figure 3 As shown, the device includes:

[0135] The data access log processing module 310 is used to acquire data access logs, preprocess the data access logs, and obtain a time window statistics matrix. The time window statistics matrix includes multiple time window slices, and each time window slice includes the window start time, the window end time, and the number of data accesses within the time window.

[0136] The timeliness decay weighted calculation module 320 is used to perform timeliness decay weighted calculation on the number of data accesses in the time window statistical matrix to obtain a weighted access frequency vector, wherein the weighting coefficient corresponding to the number of data accesses in each time window slice is inversely proportional to the time difference between the time window slice and the current time.

[0137] The multidimensional feature matrix extraction module 330 is used to extract statistical features of multiple data access times based on the weighted access frequency vector to obtain a multidimensional feature matrix. Each row of the multidimensional feature matrix corresponds to the feature vector of a time window slice, and each column corresponds to a feature type.

[0138] The clustering label vector acquisition module 340 is used to cluster the feature vectors of each time window slice in the multidimensional feature matrix using a Gaussian mixture model to obtain clustering label vectors.

[0139] The access frequency prediction module 350 is used to input the clustering label vector into the access frequency prediction model to obtain the access frequency prediction value of the data within a preset time step in the future. Each access frequency prediction value corresponds to a confidence interval.

[0140] The data migration module 360 ​​is used to determine a migration decision matrix based on the access frequency prediction value and data storage cost parameters, and to perform data migration based on the migration decision matrix. The migration decision matrix balances access speed and storage cost by dynamically adjusting the data storage location.

[0141] This invention also provides an electronic device 400, such as... Figure 4 As shown, it includes:

[0142] At least one processor 401;

[0143] Memory 402 for storing the at least one processor-executable instruction;

[0144] The at least one processor is configured to execute the instructions to implement the method described in the first aspect.

[0145] Fourthly, embodiments of the present invention provide a computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method described in the first aspect.

[0146] Fifthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0147] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A method for frequent data storage and migration for big data, characterized in that, The method includes: Obtain data access logs, preprocess the data access logs to obtain a time window statistics matrix, the time window statistics matrix includes multiple time window slices, each time window slice includes the window start time, the window end time and the number of data accesses within the time window; The time-sensitivity attenuation weighted calculation is performed on the number of data accesses in the time window statistical matrix to obtain a weighted access frequency vector. The weighting coefficient corresponding to the number of data accesses in each time window slice is inversely proportional to the time difference between the time window slice and the current time. Based on the weighted access frequency vector, statistical features of multiple data access times are extracted to obtain a multidimensional feature matrix. Each row of the multidimensional feature matrix corresponds to the feature vector of a time window slice, and each column corresponds to a feature type. The feature vectors of each time window slice in the multidimensional feature matrix are clustered using a Gaussian mixture model to obtain cluster label vectors. The clustering label vector is input into the access frequency prediction model to obtain the predicted access frequency of the data within a preset time step in the future. Each predicted access frequency value corresponds to a confidence interval. Based on the predicted access frequency and data storage cost parameters, a migration decision matrix is ​​determined, and data migration is performed based on the migration decision matrix. The migration decision matrix balances access speed and storage cost by dynamically adjusting the data storage location.

2. The method according to claim 1, characterized in that, The preprocessing of the data access logs to obtain a time window statistical matrix includes: Collect data request details from the data access log, and extract data query items from the data request details; The data query items are cleaned, merged, and deduplicated to obtain a continuous and readable data request standard detail. Multiple time window slices are extracted from the continuously readable data request standard details, and the multiple time window slices are combined into a time window statistical matrix.

3. The method according to claim 1, characterized in that, The method of extracting statistical features of data access frequency based on the weighted access frequency vector to obtain a multi-dimensional feature matrix includes: The first-order difference calculation is performed on the weighted access frequency vector to eliminate the trend component in the weighted access frequency vector, thus transforming the weighted access frequency vector from a non-stationary sequence into a stationary sequence. Calculate the statistics of the weighted access frequency vector; the statistics include the mean, variance, and autocorrelation coefficient. For each time window, the first-order difference result, mean, variance and autocorrelation coefficient of the time window are composed of the original value corresponding to the time window in the weighted access frequency vector; The multidimensional feature vectors corresponding to multiple time windows are combined to form a multidimensional feature matrix.

4. The method according to claim 1, characterized in that, The clustering of feature vectors for each time window slice in the multidimensional feature matrix using a Gaussian mixture model yields cluster label vectors, including: The probability of the feature vector of each time window being assigned to multiple clusters is calculated using the soft clustering method of Gaussian mixture model. By using the Gaussian mixture model with a mixture Gaussian distribution, each cluster is described by a Gaussian distribution. The probability density function is parameterized by the mean, covariance, and mixture weights. Similar groups of access patterns are identified, and the feature vectors of each time window slice are clustered based on the identified similar groups. Cluster labels are assigned to each time window to obtain cluster label vectors.

5. The method according to claim 1, characterized in that, The step of inputting the clustering label vector into the access frequency prediction model to obtain the predicted access frequency value and confidence interval of the data within a preset time step includes: By eliminating trends and periodicity through non-seasonal and seasonal differencing, the cluster label vectors can meet the stationarity requirement. Seasonal and non-seasonal parameters are determined using autocorrelation and partial autocorrelation functions, and the model parameters are fitted using the least squares method or the maximum likelihood estimation method. Based on the stabilized cluster label vectors and model parameters, the predicted access frequency and confidence interval of the data within a preset time step are calculated.

6. The method according to any one of claims 1 to 5, characterized in that, The process of determining a migration decision matrix based on the predicted access frequency and data storage cost parameters, and then performing data migration based on the migration decision matrix, includes: Calculate the storage cost of different storage solutions based on the predicted access frequency. The objective function of a constrained linear programming model is used to solve for the lowest cost storage scheme that satisfies the constraints. The objective function considers the unit storage cost of data stored in high-speed media or low-cost media, the impact of migration frequency on total storage cost, and the cost of migrating frequently accessed data from low-cost media to high-speed media. The constraints include ensuring that the usage of high-speed media does not exceed its total capacity, and storing frequently accessed data in high-speed media when the predicted access frequency of frequently accessed data is greater than a preset threshold. Based on the solved lowest-cost storage scheme, a migration decision matrix is ​​determined, and the storage gateway is triggered to perform data migration operations according to the migration decision matrix.

7. A device for frequent data storage and migration of big data, characterized in that, The device includes: The data access log processing module is used to acquire data access logs, preprocess the data access logs, and obtain a time window statistics matrix. The time window statistics matrix includes multiple time window slices, and each time window slice includes the window start time, the window end time, and the number of data accesses within the time window. The time-sensitivity decay weighted calculation module is used to perform time-sensitivity decay weighted calculation on the number of data accesses in the time window statistical matrix to obtain a weighted access frequency vector, wherein the weighting coefficient corresponding to the number of data accesses in each time window slice is inversely proportional to the time difference between the time window slice and the current time. The multidimensional feature matrix extraction module is used to extract statistical features of multiple data access times based on the weighted access frequency vector to obtain a multidimensional feature matrix. Each row of the multidimensional feature matrix corresponds to the feature vector of a time window slice, and each column corresponds to a feature type. The clustering label vector acquisition module is used to cluster the feature vectors of each time window slice in the multidimensional feature matrix using a Gaussian mixture model to obtain clustering label vectors. The access frequency prediction module is used to input the clustering label vector into the access frequency prediction model to obtain the access frequency prediction value of the data within a preset time step in the future. Each access frequency prediction value corresponds to a confidence interval. The data migration module is used to determine a migration decision matrix based on the predicted access frequency and data storage cost parameters, and to perform data migration based on the migration decision matrix. The migration decision matrix balances access speed and storage cost by dynamically adjusting the data storage location.

8. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images