A data hot and cold layering method, device, equipment and storage medium

Abstract

The application discloses a data cold and hot layering method and device, equipment and a storage medium, and belongs to the technical field of data processing. The method comprises the following steps: acquiring a plurality of initial feature data corresponding to each dimension data at a preset moment, and performing normalization processing on each initial feature data to obtain a plurality of feature data; calculating the minimum weight of each feature data; calculating the comprehensive cold and hot score of all dimension data based on the minimum weight; and obtaining a data cold and hot layering result based on the comprehensive cold and hot score and a cold and hot classification threshold range. Through the application, the technical problems of low storage resource utilization efficiency caused by the single evaluation dimension, the inability to dynamically adapt to business changes, the neglect of the internal correlation between features, and the lag of the evaluation result behind the actual access mode evolution in the prior art are solved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and specifically relates to a method, apparatus, device and storage medium for data hot and cold stratification. Background Technology

[0002] As enterprises deepen their digital transformation, the amount of data generated by various business systems is growing exponentially. To control the rapidly rising storage costs while ensuring data access performance, a tiered storage architecture consisting of both high-speed and low-cost storage media has become the mainstream solution in the industry. The core of this architecture lies in placing frequently accessed "hot" data in the high-speed tier and automatically migrating infrequently accessed "cold" data to the low-cost tier, thereby achieving the optimal balance between performance and cost. Therefore, accurate and efficient identification of data hot / cold status is a key technological prerequisite for realizing intelligent data tiered management.

[0003] In the current field of data hot / cold stratification, methods based on static thresholds, fixed-weight models, or complex machine learning are mainly used. However, these techniques generally suffer from problems such as a single evaluation dimension, inability to dynamically adapt to business changes, neglect of the inherent correlation between features, and evaluation results lagging behind the evolution of actual access patterns. This leads to inaccurate allocation of storage resources and makes it difficult to achieve the optimal balance between cost and performance.

[0004] Therefore, there is an urgent need for a technical solution that can accurately and intelligently identify and stratify hot and cold data to improve the efficiency of storage resource utilization. Summary of the Invention

[0005] This application provides a data hot and cold tiering method, apparatus, device, and storage medium, which can solve the technical problems in the prior art, such as single evaluation dimensions, inability to dynamically adapt to business changes, neglect of the inherent correlation between features, and evaluation results lagging behind the evolution of actual access patterns, resulting in low storage resource utilization efficiency.

[0006] To achieve the above objectives, this application provides the following technical solution: A data hot / cold stratification method, the method comprising: Acquire several initial feature data corresponding to each dimension of data at a preset time, and normalize each initial feature data to obtain multiple feature data. Calculate the final weight for each feature data; Based on the final weights, calculate the overall hot / cold score for all dimensions of data; Based on the comprehensive hot and cold score and the hot and cold classification threshold range, the data hot and cold stratification results are obtained.

[0007] Based on the same inventive concept, this application also provides a data hot / cold stratification device, the device comprising: The data acquisition module is configured to acquire several initial feature data corresponding to each dimension of data at a preset time, and to normalize each initial feature data to obtain multiple feature data. The first calculation module is configured to calculate the final weight of each feature data; The second calculation module is configured to calculate the overall hot / cold score of all dimensions of data based on the final weight. The data hot / cold stratification module is configured to obtain data hot / cold stratification results based on the comprehensive hot / cold score and the hot / cold classification threshold range.

[0008] Based on the same inventive concept, this application also provides an electronic device, including: a memory and a processor; the processor is used to read and execute a computer program stored in the memory to implement the steps of the aforementioned data hot and cold stratification method.

[0009] Based on the same inventive concept, embodiments of this application also provide a computer storage medium storing computer-executable instructions, which, when executed, implement the steps of the aforementioned data hot and cold stratification method.

[0010] Compared with the prior art, this application has the following advantages: A multi-dimensional integrated feature system is constructed, covering six dimensions of feature indicators: access, business, trends, collaboration, time, and lifecycle. This system enables a leap from a purely technical indicator to an evaluation model that deeply integrates with the business context. By capturing data signals of cooling or heating in real time through trends, it dynamically adapts to business changes and achieves forward-looking data analysis. The final decision is based on the integration of information from all dimensions, avoiding a single indicator and making the judgment of data cooling or heating more objective.

[0011] This application solves the technical problems in the prior art, such as the single evaluation dimension, inability to dynamically adapt to business changes, neglect of the inherent correlation between features, and evaluation results lagging behind the evolution of actual access patterns, resulting in low storage resource utilization efficiency.

[0012] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 A flowchart illustrating the method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the data acquisition and normalization process according to an embodiment of this application; Figure 3 This is a schematic diagram of the dynamic weight calculation process according to an embodiment of this application; Figure 4 This is a schematic diagram of the comprehensive cold and hot scoring and classification process according to an embodiment of this application; Figure 5 This is a schematic diagram of an automated storage process according to an embodiment of this application; Figure 6 This is a functional module diagram of an embodiment of the data hot and cold stratification device of this application; Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] To address the shortcomings of existing technologies, refer to Figure 1 This application discloses a data hot / cold stratification method, including: Step S10: Obtain several initial feature data corresponding to each dimension of data at a preset time, and normalize each initial feature data to obtain multiple feature data. In this embodiment, refer to Figure 2 For each initial feature data object, a feature set with six dimensions is defined: access dimension, business dimension, trend dimension, collaboration dimension, time dimension, and lifecycle dimension.

[0017] Access Dimension (V): Random read frequency (times / hour), number of full table scans and / or data update frequency (times / hour).

[0018] Business Dimension (B): Business importance level, number of related tables and / or identification of the core business process to which it belongs.

[0019] Trend dimension (T): visit growth rate (%) and / or daily data growth rate (%).

[0020] Collaboration Dimension (C): Number of concurrent users, table join frequency (times / day), and / or number of cross-application calls.

[0021] Time dimension (Tm): Last access time and / or data creation age (days).

[0022] Lifecycle dimension (L): Data validity period (days) and / or statutory retention period (years).

[0023] Business application logs and configuration management databases collect multi-dimensional initial feature data in real time or near real time. Then, the initial feature data for each dimension is normalized to obtain multiple normalized initial feature data, which are denoted as multiple feature data.

[0024] In this embodiment, the example is to normalize the initial feature data of the access dimension (random read frequency) and the initial feature data of the business dimension (business importance level) to obtain the feature data.

[0025] Specifically, the initial feature data is normalized and linearly mapped to the interval [0, 1]. The normalization formula is as follows:

[0026] In the formula, This represents the j-th initial feature data after normalization. This represents the j-th initial feature data. This represents the minimum value in the initial feature data. This represents the maximum value in the initial feature data.

[0027] Example 1: Calculate the normalization of the initial feature data. The specific data is shown in Table 1 below.

[0028] Table 1

[0029] In Table 1 above, the normalized calculation of the random read frequency for time 1 is as follows: ((180 130) / 60=50 / 60≈0.8333; Normalized calculation of business importance level for time 1: ((6) 3) / 5 = 3 / 5 = 0.6000.

[0030] Similarly, the normalized random read frequencies in Table 1 above are [0.8333, 1.0000, 0.6667, 0.5000, 0.3333, 0.1667, 0.0000]; the normalized business importance levels are [0.6000, 1.0000, 0.0000, 0.6000, 0.4000, 0.0000, 0.0000].

[0031] The original data of different dimensions are converted to a unified scale [0, 1] to facilitate subsequent correlation coefficient calculation, weight calculation and comprehensive score calculation, so as to ensure that all features are comparable in the calculation and avoid the result being dominated by some feature values ​​that are too large.

[0032] Step S20: Calculate the final weight of each feature data; In this embodiment, refer to Figure 3 This method measures the strength of the association between recent feature data and the "time decay" pattern. The weights are automatically updated over time, with historical feature data having lower weights, earlier feature data gradually decaying, and recent feature data having higher weights, ensuring that the more recent feature data has a greater impact. It dynamically captures the strength of the association between features and the data lifecycle decay pattern, enabling the system to perceive changes in the historical state of the data.

[0033] The relationship between the historical value sequence of feature data x over the past M time windows and the time decay vector t is as follows: λ represents the attenuation factor, 0 < λ < 1, and the default value is 0.9, usually 0.8~0.95.

[0034] For the j-th feature data, first calculate the comprehensive correlation of the j-th feature data; Specifically, the correlation coefficient within each sliding time window is calculated using a first preset formula; The first preset formula is as follows:

[0035] In the formula, This represents the correlation coefficient between the j-th feature data within the i-th time window and time t. This represents the j-th feature data. This represents the i-th time window. This represents the sample mean of the j-th feature data within the i-th time window. This represents the i'th moment within the i-th time window. The average time within the i-th time window, N represents the number of j-th feature data within the i-th time window; N' represents the number of moments within the i-th time window; where each moment corresponds to one j-th feature data; based on the correlation coefficients between the j-th feature data and time t within all time windows, an exponentially weighted average is performed using the second preset formula to obtain the exponentially weighted average correlation coefficient of the j-th feature data from time t to time T; The second preset formula is as follows:

[0036] In the formula, This represents the exponentially weighted average correlation coefficient of the j-th feature data from time t to time T. Indicates the attenuation factor; This represents the i-th time window. Let M represent the correlation coefficient between the j-th feature data within the i-th time window and time t, and M represent the number of time windows.

[0037] Based on the comprehensive relevance of the j-th feature data, the basic weight of the j-th feature data is calculated using the basic weight formula. The basic weight formula is as follows:

[0038] In the formula, This represents the basic weight of the j-th feature data. This represents the j-th feature data. This represents the exponentially weighted average correlation coefficient of the j-th feature data from time t to time T; P represents the total number of feature data, and j represents the index of the feature data, j = 1,2,...,P; For example, the normalized random read frequencies are: [0.8333, 1.0000, 0.6667, 0.5000, 0.3333, 0.1667, 0.0000].

[0039] Calculate the i-th time window The correlation coefficient between the j-th feature data (e.g., random read frequency) and time t r = -0.9697; The correlation coefficients of the j-th feature data (e.g., random read frequency) with time t within all time windows are calculated by exponential weighting, resulting in the exponential weighted average correlation coefficient R of the j-th feature data (e.g., random read frequency) from time t to time T, where R = -0.9697.

[0040] Calculate the basic weight of the j-th feature data (e.g., random read frequency) from time t to time T based on the exponentially weighted average correlation coefficient R. .

[0041] .

[0042] Similarly, the basic weights for business importance levels can be calculated. =-0.4105.

[0043] Both feature data are negatively correlated with the time decay pattern, indicating that time is advancing, but the feature value is decreasing. The negative correlation with random read frequency is even stronger, suggesting a more pronounced decreasing trend over time.

[0044] Furthermore, we identify and strengthen pivotal feature data within the feature network, considering the mutual influence and dependencies between feature data. We look beyond the feature data itself to its position within the network, adapting to the evolving relationships between feature data. By quantifying the intrinsic correlations between features, we enhance the basic weights, making the evaluation results more aligned with complex business logic.

[0045] Based on the base weight of the j-th feature data (e.g., random read frequency), the enhanced weight of the j-th feature data (e.g., random read frequency) is calculated using the enhanced weight formula. The formula for enhancing weights is as follows:

[0046] In the formula, This represents the weight of the j-th feature data after enhancement. Represents the coupling gain coefficient. This represents the average of the absolute values ​​of the correlation coefficients between the j-th feature data and other feature data within the system; , Where P represents the total number of feature data. The covariance between the j-th feature data and other feature data y in the system is used to measure the linear correlation between the two. This represents the standard deviation of the j-th feature data. The standard deviation of other characteristic data within the system; denominator p 1: Indicates the number of feature data other than itself, used for calculating the average. The coupling gain coefficient is a hyperparameter used to control the degree to which the feature coupling strength affects the weight enhancement. It is a scaling factor for coupling strength, with a value of 0.1 to 0.5. Generally, the characteristic value is 0.2, which neither overly dominates the basic weights nor fails to reflect the coupling effect, thus avoiding drastic changes in weights due to fluctuations in coupling relationships.

[0047] For example, the normalized data is calculated. (e.g., random read frequency) and normalized data The covariance between (e.g., business importance level).

[0048] .

[0049] This represents the random read frequency at time i' within the i-th time window. Let represent the mean of the random read frequency within the i-th time window. This represents the importance level of the business at time i' within the i-th time window. Let N represent the mean of the business importance levels within the i-th time window, and let N represent the number of j-th feature data within the i-th time window. It's easy to see that the random read frequency and the number of business importance levels are the same within the i-th time window.

[0050] Calculate the normalized data (e.g., random read frequency) and normalized data Correlation coefficients between (e.g., business importance levels).

[0051] .

[0052] Based on the correlation coefficient (| Using | = 0.12) and coupling gain coefficient α = 0.2, calculate the average of the absolute values ​​of the correlation coefficients between the j-th feature data (e.g., random read frequency) and other feature data (e.g., business importance level) within the system. , .

[0053] Calculate the weights after random read frequency enhancement: .

[0054] Calculate the weights after enhancing the importance level of the business: .

[0055] Furthermore, assigning higher weight adjustment sensitivity to feature data currently showing a significant upward or downward trend enables the assessment to be predictive. Trend correction provides foresight of future changes, and weights can respond to changing trends in the relevance of feature data. Final adjustments are made based on the recent rate of change of weights, giving the assessment system the ability to predict future data popularity, thus achieving a technological paradigm shift from perception-response to prediction-planning.

[0056] The weight of the enhanced j-th feature data is corrected using the third preset formula to obtain the final weight of the j-th feature data. The third preset formula is as follows:

[0057] In the formula, This represents the final weight of the j-th feature data. β represents the positive trend gain coefficient, standard: 0.2~0.3; γ represents the negative trend gain coefficient, standard: =0.4~0.6; Indicates the strength of the trend. , This represents the exponentially weighted average correlation coefficient of the j-th feature data from time t to time T. Let represent the exponentially weighted average correlation coefficient of the j-th feature data from time t to time T-1; and Indicating a trend strength threshold, preferably, =0.05, =-0.05; Represents positive numbers, with a range of 1000. ~ Avoid having a denominator of 0; For example, the exponentially weighted average correlation coefficient of random read frequency from time t to time T is calculated as R(V1, t, T) = -0.9697; the exponentially weighted average correlation coefficient of business importance level from time t to time T is calculated as R(B1, t, T) = -0.6757; the exponentially weighted average correlation coefficient of random read frequency from time t to time T is calculated as R(V1, t, T-1) = -0.85; and the exponentially weighted average correlation coefficient of random read frequency of business importance level from time t to time T is calculated as R(B1, t, T-1) = -0.60.

[0058] Calculate the trend strength of random read frequency: .

[0059] Calculate the trend strength of business importance levels: .

[0060] The final weight of the random read frequency is calculated as follows: .

[0061] Similarly, the final weights for the business importance levels are: .

[0062] By doing so, the final weight of each feature data can be calculated.

[0063] Step S30: Based on the final weights, calculate the comprehensive hot / cold score for all dimensions of data; In this embodiment, refer to Figure 4 The feature score is obtained by multiplying each normalized feature data within each dimension by the final weight of that feature and then summing the results. Individual feature data have limited importance; they need to be combined into business dimensions, resulting in feature data aggregation.

[0064]

[0065] In the formula, This represents the score of the feature data in the d-th dimension. This represents the j-th feature data. = , This represents the j-th initial feature data; This represents the final weight of the j-th feature data.

[0066] The final weights of each dimension are not preset, but dynamically determined by the ratio of the sum of the final weights of all its subordinate features to the sum of all feature weights. This ensures that the core of the evaluation system always revolves around the most indicative feature dimension at any given time. The weight of the d-th dimension at time T is obtained by summing the final weights of all feature data within each dimension and then dividing by the sum of all feature data's final weights. The importance of different dimensions changes over time. The system automatically identifies the most important dimension at any given time, avoiding the subjectivity of manually preset weights.

[0067]

[0068] In the formula, This represents the weight of the data in the d-th dimension at time T. This represents the set of multiple feature data contained in the d-th dimension. This represents the final weight of the j-th feature data; The score of each dimension of data is multiplied by the dynamic weight of that dimension, and then summed to generate a single comprehensive evaluation index—the comprehensive hot and cold score—to support decision-making and classification.

[0069] Specifically, the formula for calculating the overall hot / cold score is as follows:

[0070] In the formula, D represents the total number of dimensions used to evaluate the hot / cold rating. Score represents the weight of the d-th dimension at time T. d This represents the score of the d-th dimension. For example, taking the feature data included in the access dimension data (V) as the random read frequency and the feature data included in the business dimension data (B) as the business importance level, the final weight of the random read frequency is... The final weight of business importance level .

[0071] Normalized random read frequency at time T=6 Normalized business importance level at time T=6 .

[0072] Calculate the feature scores for the access dimension data (V): ; Calculate the feature scores for the business dimension data (B): .

[0073] Weights of the access dimension data (V) at time T=6: ; Weights of business dimension data (B) at time T=6: .

[0074] Calculate the combined hot / cold score of access dimension data (V) and business dimension data (B) at time T=6: = =-0.0559.

[0075] Step S40: Based on the comprehensive hot and cold score and the hot and cold classification threshold range, obtain the data hot and cold stratification results.

[0076] In some specific embodiments, step S40 includes: The overall hot / cold score of all dimensions of data is compared with the hot / cold classification threshold range to obtain the comparison results; If the comparison result is that the comprehensive hot / cold score is greater than or equal to the upper limit of the hot / cold classification threshold range, then each feature data in all dimensions of data is determined to be hot data. If the comparison result shows that the comprehensive cold and hot score is less than the lower limit of the cold and hot classification threshold range, then each feature data in all dimension data is determined as cold data; If the comparison result shows that the comprehensive cold and hot score is greater than or equal to the lower limit of the cold and hot classification threshold range and less than the upper limit of the cold and hot classification threshold range, then each feature data in all dimension data is determined as warm data.

[0077] In this embodiment, continue to refer to Figure 4 , taking the cold and hot classification threshold range as [-0.1, 0.3] as an example, the upper limit value T_hot of the cold and hot classification threshold range is 0.3, and the lower limit value T_cold of the cold and hot classification threshold range is -0.1.

[0078] When the comprehensive cold and hot score of all dimension data ≥ T_hot, each feature data in all dimension data is determined as hot data.

[0079] When the comprehensive cold and hot score of all dimension data T_cold ≤ < T_hot, each feature data in all dimension data is determined as warm data.

[0080] When the comprehensive cold and hot score of all dimension data < T_cold, each feature data in all dimension data is determined as cold data.

[0081] Exemplarily, taking the feature data included in the access dimension data (V) as the random read frequency and the feature data included in the service dimension data (B) as the service importance level as an example, since the comprehensive cold and hot score of the access dimension data (V) and the service dimension data (B) at T = 6 = -0.0559, -0.1 < -0.0559 < 0.3, therefore, each feature data in the access dimension data (V) and the service dimension data (B) at T = 6 is determined as warm data, that is, the random read frequency and the service importance level at T = 6 are determined as warm data.

[0082] In this embodiment, several initial feature data corresponding to each dimension of data at a preset time are obtained, and each initial feature data is normalized to obtain multiple feature data; the final weight of each feature data is calculated; based on the final weight, a comprehensive hot / cold score for all dimensions of data is calculated; based on the comprehensive hot / cold score and the hot / cold classification threshold range, a data hot / cold stratification result is obtained. A multi-dimensional fusion feature system is constructed, covering a feature indicator system of six dimensions: access, business, trend, collaboration, time, and lifecycle. This achieves a leap from a purely technical indicator to a deeply integrated evaluation model based on business context; trends are used to capture data cooling or heating signals in real time, dynamically adapting to business changes and achieving forward-looking data analysis; the final decision is based on the fusion of information from all dimensions, avoiding a single indicator and making the judgment of data hot / cold more objective. This embodiment solves the technical problems in the prior art, such as a single evaluation dimension, inability to dynamically adapt to business changes, neglect of the inherent correlation between features, and evaluation results lagging behind the actual evolution of access patterns, leading to low storage resource utilization efficiency.

[0083] Furthermore, in one embodiment, the method further includes: Based on the data hot and cold stratification results, the feature data cross-layer migration task is automatically triggered by interacting with the cloud platform API or distributed storage system to store the feature data to the corresponding storage layer.

[0084] In this embodiment, refer to Figure 5 The system periodically executes the above evaluation process. After the scoring and classification results are updated, the automated storage strategy execution engine automatically triggers cross-layer data migration tasks through interaction with the cloud platform API or distributed storage system. For example, hot data is stored in the highest performance storage layer, warm data is stored in a storage layer with a performance-cost balance, and cold data is archived to the lowest cost storage layer. This ensures that data is always located in the most appropriate storage layer.

[0085] Based on the same inventive concept, this application also provides a data hot and cold stratification device.

[0086] In one embodiment, reference is made to Figure 6 , Figure 6 This is a functional module diagram of an embodiment of the data hot / cold stratification device of this application. Figure 6 As shown, the data hot / cold stratification device includes: The data acquisition module 10 is configured to acquire several initial feature data corresponding to each dimension of data at a preset time, and to normalize each initial feature data to obtain multiple feature data. The first calculation module 20 is configured to calculate the final weight of each feature data; The second calculation module 30 is configured to calculate the comprehensive hot / cold score of all dimensions of data based on the final weight. The data hot and cold stratification module 40 is configured to obtain data hot and cold stratification results based on the comprehensive hot and cold score and the hot and cold classification threshold range.

[0087] Optionally, in one embodiment, the various dimension data include: access dimension data, business dimension data, trend dimension data, collaboration dimension data, time dimension data, and lifecycle dimension data; Among them, the initial characteristic data corresponding to the access dimension data include: random read frequency, number of full table scans and / or data update frequency; The initial characteristic data corresponding to the business dimension data includes: business importance level, number of related tables, and / or identification of the core business process to which it belongs; The initial characteristic data corresponding to the trend dimension data include: the daily growth rate of visit volume and / or the daily growth rate of data volume; Several initial characteristic data corresponding to the collaboration dimension data include: number of concurrent users, table join frequency, and / or number of cross-application calls; Several initial feature data corresponding to the time dimension data include: last access time and / or data creation age; Several initial characteristic data corresponding to the lifecycle dimension data include: data validity period and / or legal retention period.

[0088] Optionally, in one embodiment, the first computing module 20 is configured to: For the j-th feature data, calculate the comprehensive correlation of the j-th feature data; Based on the comprehensive relevance of the j-th feature data, the basic weight of the j-th feature data is calculated using the basic weight formula. The basic weight formula is as follows:

[0089] In the formula, This represents the basic weight of the j-th feature data. This represents the j-th feature data. This represents the exponentially weighted average correlation coefficient of the j-th feature data from time t to time T; P represents the total number of feature data, and j represents the index of the feature data, j = 1,2,...,P; Based on the base weight of the j-th feature data, the weight of the j-th feature data after enhancement is calculated using the enhanced weight formula; The formula for enhancing weights is as follows:

[0090] In the formula, This represents the weight of the j-th feature data after enhancement. Represents the coupling gain coefficient. This represents the average of the absolute values ​​of the correlation coefficients between the j-th feature data and all other feature data in the system; The weight of the enhanced j-th feature data is corrected using the third preset formula to obtain the final weight of the j-th feature data. The third preset formula is as follows:

[0091] In the formula, This represents the final weight of the j-th feature data. γ represents the positive trend gain coefficient, and γ represents the negative trend gain coefficient. Indicates the strength of the trend. , This represents the exponentially weighted average correlation coefficient of the j-th feature data from time t to time T. Let represent the exponentially weighted average correlation coefficient of the j-th feature data from time t to time T-1; and Indicates the threshold for trend strength; Represents positive numbers; By analogy, the final weight of each feature data is calculated.

[0092] Optionally, in one embodiment, the first computing module 20 is further configured to: The correlation coefficient between the j-th feature data within the i-th time window and time t is calculated using the first preset formula. The first preset formula is as follows:

[0093] In the formula, This represents the correlation coefficient between the j-th feature data within the i-th time window and time t. This represents the j-th feature data. This represents the i-th time window. This represents the sample mean of the j-th feature data within the i-th time window. This represents the i'th moment within the i-th time window. The average time within the i-th time window, N represents the number of j-th feature data within the i-th time window; N' represents the number of moments within the i-th time window; where each moment corresponds to one j-th feature data. Based on the correlation coefficient between the j-th feature data and time t within all time windows, an exponentially weighted average is obtained by performing an exponentially weighted average of the j-th feature data from time t to time T using the second preset formula. The second preset formula is as follows:

[0094] In the formula, This represents the exponentially weighted average correlation coefficient of the j-th feature data from time t to time T. Indicates the attenuation factor; This represents the i-th time window. Let M represent the correlation coefficient between the j-th feature data within the i-th time window and time t, and M represent the number of time windows.

[0095] Optionally, in one embodiment, the second computing module 30 is configured to: Substitute the final weights of each feature data into the comprehensive hot / cold score calculation formula to calculate the comprehensive hot / cold score of all dimensions of data. The formula for calculating the overall hot / cold score is as follows:

[0096] In the formula, D represents the total number of dimensions used to evaluate the hot / cold rating. Score represents the weight of the d-th dimension at time T. d This represents the score of the d-th dimension.

[0097] In the formula, This represents the set of multiple feature data contained in the d-th dimension. This represents the final weight of the j-th feature data;

[0098] In the formula, This represents the j-th feature data. = , This represents the j-th initial feature data.

[0099] Optionally, in one embodiment, the data hot / cold stratification module 40 is configured to: The overall hot / cold score of all dimensions of data is compared with the hot / cold classification threshold range to obtain the comparison results; If the comparison result is that the comprehensive hot / cold score is greater than or equal to the upper limit of the hot / cold classification threshold range, then each feature data in all dimensions of data is determined to be hot data. If the comparison result is that the comprehensive hot / cold score is less than the lower limit of the hot / cold classification threshold range, then each feature data in all dimensions of data is determined to be cold data. If the comparison result shows that the comprehensive hot / cold score is greater than or equal to the lower limit of the hot / cold classification threshold range and less than the upper limit of the hot / cold classification threshold range, then each feature data in all dimensions of data is determined to be temperature data.

[0100] Optionally, in one embodiment, the method further includes: Based on the data hot and cold stratification results, the feature data cross-layer migration task is automatically triggered by interacting with the cloud platform API or distributed storage system to store the feature data to the corresponding storage layer.

[0101] The functions of each module in the above-mentioned data hot and cold stratification device correspond to the steps in the above-mentioned data hot and cold stratification method embodiment, and their functions and implementation processes will not be described in detail here.

[0102] Based on the same inventive concept, embodiments of this application also provide an electronic device, the structure of which is as follows: Figure 7 As shown, it includes: a memory and a processor, wherein the processor is used to read and execute the computer program stored in the memory to implement the aforementioned data hot and cold stratification method.

[0103] Based on the same inventive concept, this application also provides a computer storage medium storing computer-executable instructions, which, when executed, implement the aforementioned data hot and cold stratification method.

[0104] Finally, it should be noted that while some processes described in the embodiments of this application include multiple operations or steps that appear in a specific order, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0105] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for data hot and cold stratification, characterized in that, The method includes: Acquire several initial feature data corresponding to each dimension of data at a preset time, and normalize each initial feature data to obtain multiple feature data. Calculate the final weight for each feature data; Based on the final weights, calculate the overall hot / cold score for all dimensions of data; Based on the comprehensive hot and cold score and the hot and cold classification threshold range, the data hot and cold stratification results are obtained.

2. The data hot / cold stratification method according to claim 1, characterized in that, The data in each dimension includes: access dimension data, business dimension data, trend dimension data, collaboration dimension data, time dimension data, and lifecycle dimension data; Among them, the initial characteristic data corresponding to the access dimension data include: random read frequency, number of full table scans and / or data update frequency; The initial characteristic data corresponding to the business dimension data includes: business importance level, number of related tables, and / or identification of the core business process to which it belongs; The initial characteristic data corresponding to the trend dimension data include: the daily growth rate of visit volume and / or the daily growth rate of data volume; Several initial characteristic data corresponding to the collaboration dimension data include: number of concurrent users, table join frequency, and / or number of cross-application calls; Several initial feature data corresponding to the time dimension data include: last access time and / or data creation age; Several initial characteristic data corresponding to the lifecycle dimension data include: data validity period and / or legal retention period.

3. The data hot / cold stratification method according to claim 1, characterized in that, The calculation of the final weight for each feature data includes: For the j-th feature data, calculate the comprehensive correlation of the j-th feature data; Based on the comprehensive relevance of the j-th feature data, the basic weight of the j-th feature data is calculated using the basic weight formula. The basic weight formula is as follows: In the formula, This represents the basic weight of the j-th feature data. This represents the j-th feature data. Let represent the exponentially weighted average correlation coefficient of the j-th feature data from time t to time T; P represents the total number of feature data, and j represents the index of the feature data, j = 1,2,...,P; Based on the base weight of the j-th feature data, the weight of the j-th feature data after enhancement is calculated using the enhanced weight formula; The formula for enhancing weights is as follows: In the formula, This represents the weight of the j-th feature data after enhancement. Represents the coupling gain coefficient. This represents the average of the absolute values ​​of the correlation coefficients between the j-th feature data and all other feature data in the system; The weight of the enhanced j-th feature data is corrected using the third preset formula to obtain the final weight of the j-th feature data. The third preset formula is as follows: In the formula, This represents the final weight of the j-th feature data. γ represents the positive trend gain coefficient, and γ represents the negative trend gain coefficient. Indicates the strength of the trend. , This represents the exponentially weighted average correlation coefficient of the j-th feature data from time t to time T. Let represent the exponentially weighted average correlation coefficient of the j-th feature data from time t to time T-1; and Indicates the threshold for trend strength; Represents positive numbers; By analogy, the final weight of each feature data is calculated.

4. The data hot / cold stratification method according to claim 3, characterized in that, The calculation of the comprehensive correlation of the j-th feature data includes: The correlation coefficient between the j-th feature data within the i-th time window and time t is calculated using the first preset formula. The first preset formula is as follows: In the formula, This represents the correlation coefficient between the j-th feature data within the i-th time window and time t. This represents the j-th feature data. This represents the i-th time window. This represents the sample mean of the j-th feature data within the i-th time window. This represents the i'th moment within the i-th time window. The average time within the i-th time window, N represents the number of j-th feature data within the i-th time window; N' represents the number of moments within the i-th time window; where each moment corresponds to one j-th feature data. Based on the correlation coefficient between the j-th feature data and time t within all time windows, an exponentially weighted average is obtained by performing an exponentially weighted average of the j-th feature data from time t to time T using the second preset formula. The second preset formula is as follows: In the formula, This represents the exponentially weighted average correlation coefficient of the j-th feature data from time t to time T. Indicates the attenuation factor; This represents the i-th time window. Let M represent the correlation coefficient between the j-th feature data within the i-th time window and time t, and M represent the number of time windows.

5. The data hot / cold stratification method according to claim 1, characterized in that, The calculation of the comprehensive hot / cold score for all dimensions of data based on the final weights includes: Substitute the final weights of each feature data into the comprehensive hot / cold score calculation formula to calculate the comprehensive hot / cold score of all dimensions of data. The formula for calculating the overall hot / cold score is as follows: In the formula, D represents the total number of dimensions used to evaluate the hot / cold rating. Score represents the weight of the d-th dimension at time T. d This represents the score of the d-th dimension. In the formula, This represents the set of multiple feature data contained in the d-th dimension. This represents the final weight of the j-th feature data; In the formula, This represents the j-th feature data. = , This represents the j-th initial feature data.

6. The data hot / cold stratification method according to claim 1, characterized in that, The data stratification results obtained based on the comprehensive hot / cold score and the hot / cold classification threshold range include: The overall hot / cold score of all dimensions of data is compared with the hot / cold classification threshold range to obtain the comparison results; If the comparison result is that the comprehensive hot / cold score is greater than or equal to the upper limit of the hot / cold classification threshold range, then each feature data in all dimensions of data is determined to be hot data. If the comparison result is that the comprehensive hot / cold score is less than the lower limit of the hot / cold classification threshold range, then each feature data in all dimensions of data is determined to be cold data. If the comparison result shows that the comprehensive hot / cold score is greater than or equal to the lower limit of the hot / cold classification threshold range and less than the upper limit of the hot / cold classification threshold range, then each feature data in all dimensions of data is determined to be temperature data.

7. The data hot / cold stratification method according to any one of claims 1 to 6, characterized in that, The method further includes: Based on the data hot and cold stratification results, the feature data cross-layer migration task is automatically triggered by interacting with the cloud platform API or distributed storage system to store the feature data to the corresponding storage layer.

8. A data hot / cold stratification device, characterized in that, The device includes: The data acquisition module is configured to acquire several initial feature data corresponding to each dimension of data at a preset time, and to normalize each initial feature data to obtain multiple feature data. The first calculation module is configured to calculate the final weight of each feature data; The second calculation module is configured to calculate the overall hot / cold score of all dimensions of data based on the final weight. The data hot / cold stratification module is configured to obtain data hot / cold stratification results based on the comprehensive hot / cold score and the hot / cold classification threshold range.

9. An electronic device, characterized in that, include: Memory, processor; The processor is configured to read and execute the computer program stored in the memory to implement the steps of the data hot and cold stratification method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed, implement the steps of the data hot and cold stratification method according to any one of claims 1-7.

Images