Cold and hot data storage method and system oriented to cross-domain heterogeneous storage system

Abstract

The invention belongs to the technical field of cold and hot data identification management, and provides a cold and hot data storage method and system oriented to a cross-domain heterogeneous storage system. Cold and hot data portraying is carried out by comprehensively considering multiple dimensions of data access frequency, data updating frequency, business importance, file correlation and data freshness; the cold and hot attributes of the data are comprehensively judged, the actual value and the use requirement of the data are reflected, the data are accurately classified into hot data, temperature data and cold data, the importance and the access requirement of the data can be dynamically evaluated, the hot data, the temperature data and the cold data are stored in different storage spaces in a layered mode, and the data storage efficiency is improved. Therefore, data storage management is more effectively carried out, the experience feeling of the cross-domain heterogeneous storage system is improved, and popularization and application are facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical field of cold and hot data identification and management, and in particular, relates to a cold and hot data storage method and system for a cross-domain heterogeneous storage system. Background Art

[0002] With the rapid development of high-performance computing and Internet technologies, supercomputing Internet plays an increasingly important role in scientific research, engineering computing and other fields. It is usually composed of multiple data centers, which realize resource sharing and collaborative computing through high-speed networks. The cross-domain heterogeneous storage of multiple data centers has greatly improved the computing power and data processing capabilities, but it also brings the complexity of data management. How to effectively store and manage this data, especially under limited storage resources, is an important issue facing current technology. Generally speaking, in the field of data storage and management, data is usually divided into hot data and cold data according to its access frequency and importance.

[0003] Existing methods for identifying hot and cold data have limitations in dealing with data diversity, dynamics, and complexity. In particular, in the face of a rapidly changing data environment, existing methods are unable to flexibly and accurately identify and classify hot and cold data. Specifically, traditional methods for identifying hot and cold data mainly rely on a single access frequency indicator, resulting in inaccurate classification results that cannot fully reflect the actual value and usage needs of the data. A single classification method based on access frequency may lead to misjudgment of hot data, resulting in a waste of high-cost storage resources or failure to archive cold data in a timely manner. Inaccurate data classification results make it impossible to effectively store and manage data in a rapidly changing data environment and with limited storage resources, thus affecting the user experience and promotion and application of cross-domain heterogeneous storage systems. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a hot and cold data storage method and system for cross-domain heterogeneous storage systems. The present invention performs hot and cold data profiling by comprehensively considering multiple dimensions such as data access frequency, data update frequency, business importance, file relevance and data freshness, comprehensively judges the hot and cold attributes of the data, can accurately classify the data, and can dynamically evaluate the importance and access requirements of the data, thereby more effectively performing data storage management.

[0005] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0006] In a first aspect, the present invention provides a cold and hot data storage method for a cross-domain heterogeneous storage system, comprising:

[0007] Obtain multiple indicators of data related to cross-domain heterogeneous storage systems; the multiple indicators include access frequency, data update frequency, business importance, file relevance, and data freshness;

[0008] Standardize multiple indicators;

[0009] Perform weighted summation on multiple indicators after standardization to obtain a comprehensive evaluation value for each data point;

[0010] Sort all data points according to the comprehensive evaluation value and classify the data into hot data, warm data and cold data according to the stratified threshold method;

[0011] Store hot data, warm data, and cold data in different storage spaces.

[0012] Furthermore, the access frequency is equal to the ratio of the accumulation of the recorded data in all access times within a certain time period to the time period;

[0013] The data update frequency is equal to the ratio of the sum of all update times within a certain period of time to the period of time;

[0014] Importance of the business b for:

[0015]

[0016] Among them, R j The criticality score of the jth business; W j Indicates the importance weight of the jth business to the overall business;

[0017] The file relevance is equal to the ratio of the strength of all association relationships to the number of association relationships, where the strength of the association relationship is the number of common visits or citations;

[0018] The data freshness F n for:

[0019]

[0020] Among them, t c Indicates the current time; T u Indicates the last update time of the data; α indicates the adjustment parameter.

[0021] Furthermore, the values ​​of the access frequency, the data update frequency, the business importance, the file relevance and the data freshness are scaled to within a preset range.

[0022] Furthermore, the weight of each indicator is:

[0023]

[0024] Among them, w j represents the weight of the jth standardized indicator; M represents the total number of indicators; T j represents the Theil index of the jth standardized indicator.

[0025] Furthermore, the data is sorted in descending order according to the comprehensive evaluation value; data with a comprehensive score greater than a first preset value is classified as hot data; data with a comprehensive score between the first preset value and the second preset value is classified as warm data; data with a comprehensive score lower than the second preset value is classified as cold data; wherein the first preset value is greater than the second preset value.

[0026] Furthermore, the hot data is stored in the memory, the warm data is stored in the local storage system, and the cold data is stored in the off-site storage system.

[0027] In a second aspect, the present invention further provides a cold and hot data storage system for a cross-domain heterogeneous storage system, comprising:

[0028] The data collection module is configured to: obtain multiple indicators of data related to the cross-domain heterogeneous storage system; wherein the multiple indicators include access frequency, data update frequency, business importance, file relevance, and data freshness;

[0029] The standardization processing module is configured to: perform standardization processing on multiple indicators;

[0030] The comprehensive evaluation value calculation module is configured to: perform weighted summation on the multiple indicators after standardization to obtain a comprehensive evaluation value for each data point;

[0031] The data classification module is configured to: sort all data points according to the comprehensive evaluation value, and classify the data into hot data, warm data, and cold data according to the stratified threshold method;

[0032] The storage management module is configured to store hot data, warm data and cold data in different storage spaces in a hierarchical manner.

[0033] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the hot and cold data storage method for a cross-domain heterogeneous storage system described in the first aspect.

[0034] In a fourth aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the program, the steps of the hot and cold data storage method for a cross-domain heterogeneous storage system described in the first aspect are implemented.

[0035] In a fifth aspect, the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the hot and cold data storage method for a cross-domain heterogeneous storage system described in the first aspect are implemented.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] By comprehensively considering multiple dimensions such as data access frequency, data update frequency, business importance, file relevance, and data freshness to profile hot and cold data, we can comprehensively judge the hot and cold attributes of the data, reflect the actual value and usage needs of the data, and accurately classify the data into hot data, warm data, and cold data. We can dynamically evaluate the importance and access needs of the data, and store the hot data, warm data, and cold data in different storage spaces in a layered manner, thereby more effectively managing data storage and improving the experience of cross-domain heterogeneous storage systems, which is conducive to promotion and application.

[0038] The present invention quantifies multiple indicators such as data access frequency, data update frequency, business importance, file relevance and data freshness. Based on this, a comprehensive evaluation value of each data point is obtained by weighted summation, and the data is classified and stored and managed, which ensures the accuracy of data recognition, improves the management quality of data classification and storage, and ensures the computing power and data processing capabilities of cross-domain heterogeneous storage systems in complex data environments.

[0039] Through the present invention, as the amount of data continues to grow, it is possible to flexibly identify and divide hot and cold data, effectively cope with storage challenges in a big data environment, and ensure that the system can still operate efficiently when the amount of data increases rapidly; storing hot data in a high-performance storage medium can significantly improve the access speed and response time of data, meet the demand for fast reading of high-value data, and optimize user experience and system performance. Through the division of hot and cold data, computing and storage resources can be more reasonably allocated, more resources can be allocated to data with high data value, and the operating efficiency and stability of the overall system can be improved; transferring cold data to a low-cost storage medium can significantly reduce the operating cost of data storage; at the same time, by reducing the frequent backup and maintenance of cold data, maintenance costs can also be reduced. The hot and cold data division strategy based on the 80 / 20 rule effectively improves the utilization efficiency of storage resources and data access performance by identifying and prioritizing high-value data; by storing and managing data in layers, storage costs can be reduced and the operating efficiency and stability of the system can be improved. This strategy is simple and intuitive, easy to implement and adjust, and is suitable for large-scale data management and optimization scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings in the specification that constitute a part of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments of this embodiment and their descriptions are used to explain this embodiment and do not constitute improper limitations on this embodiment.

[0041] Figure 1 This is an architecture diagram of the cold and hot data storage method of Example 1 of the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0043] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0044] Hot data refers to data with high access frequency and high real-time requirements. It is usually the core data required for current business operations, such as data with an access frequency exceeding the preset frequency.

[0045] Cold data refers to data with low access frequency and strong historical nature. Although this data is important, it usually does not need to be accessed frequently, such as data whose access frequency does not exceed the preset frequency.

[0046] Embodiment 1:

[0047] As described in the background technology, effective hot and cold data management is of great significance for optimizing storage resources, improving data access efficiency, and reducing data storage costs. By reasonably identifying and managing hot and cold data, hot data can be stored on high-performance storage media to meet the needs of fast access, while cold data can be transferred to low-cost storage devices to reduce storage costs. Existing hot and cold data identification methods have limitations in dealing with data diversity, dynamics, and complexity. In particular, when faced with a rapidly changing data environment, existing methods are difficult to flexibly and accurately identify and classify hot and cold data. Therefore, there is an urgent need for a more intelligent, dynamic, and multi-dimensional hot and cold data identification method to improve the efficiency and effectiveness of data management.

[0048] In order to solve at least one of the above problems, this embodiment provides a hot and cold data storage method for cross-domain heterogeneous storage systems. By comprehensively considering multiple dimensions such as data access frequency, data update frequency, business importance, file relevance and data freshness to perform hot and cold data profiling, and comprehensively judging the hot and cold attributes of the data, the importance and access requirements of the data can be dynamically evaluated, thereby more effectively managing data.

[0049] Specifically, the traditional hot and cold data identification method mainly relies on a single access frequency indicator, resulting in inaccurate classification results and failing to fully reflect the actual value and usage needs of the data. This implementation is a cross-domain heterogeneous storage system for multiple data centers. By introducing multi-dimensional indicators, it comprehensively evaluates data access patterns, business importance, and correlations, thereby improving the accuracy of data classification. In addition, a single access frequency-based classification method may lead to the misjudgment of hot data, resulting in a waste of high-cost storage resources or the failure to archive cold data in a timely manner. This embodiment evaluates data value in multiple dimensions, optimizes data storage strategies, identifies and centrally manages key hot data, optimizes the use of high-performance storage resources, and reduces storage costs. The optional steps of the method in this embodiment are:

[0050] S1. In order to comprehensively evaluate the value of data, key indicators such as access frequency, data update frequency, business importance, file relevance, and data freshness are analyzed and calculated in detail. These indicators measure the usage, importance, and change dynamics of data from different perspectives, helping us accurately evaluate the value of data. The following will introduce the calculation methods of the above indicators one by one.

[0051] S1.1. Determine the access frequency:

[0052] The access frequency Fv measures the number of times data is accessed within a certain period of time, reflecting the activity and usage demand of the data. Data with high access frequency is usually data that is frequently used in business operations, indicating that it has high immediate value and business relevance. Optionally, the calculation method of access frequency Fv is as follows:

[0053]

[0054] Where T represents the time interval; N is the number of access records in time period T; A i Indicates the number of records visited for the i-th time.

[0055] S1.2. Determine the data update frequency:

[0056] Data update frequency measures the number of times data is modified or updated within a certain period of time, reflecting the activity and dynamics of the data. Frequently updated data usually means that it is active in the business and has high usage and reference value. u The calculation method is as follows:

[0057]

[0058] Among them, F u Indicates the frequency of data update; T indicates the time interval, which can be one week, one month, etc.; P indicates the number of update records within the time period T; Um Indicates the number of records updated for the mth time.

[0059] S1.3. Determine business importance:

[0060] The business importance index is used to measure the criticality and influence of data in business processes and corporate operations. It reflects the importance of data in achieving corporate goals, supporting business decisions, and meeting legal compliance. Data with high business importance is often the core data that is indispensable to the enterprise and has a higher management priority and protection requirements. b The calculation method is as follows:

[0061]

[0062] Among them, I b Indicates business importance; R j The criticality score of the jth business, optional, the criticality score range is 1-10 points; W j Indicates the importance weight of the j-th business to the overall business. The importance weight is optional and ranges from 0 to 1.

[0063] S1.4. Determine file relevance:

[0064] File relevance measures the strength of the relationship and frequency of interaction between a file and other files. Files with high relevance are usually referenced and used by multiple files in the business process, reflecting their important position and influence in the data network. Data with high file relevance often plays an important role in multiple business scenarios and has high integration value and dependency. f The calculation formula is as follows:

[0065]

[0066] Among them, C f to represent the file relevance; L represents the number of associations between files; R k Indicates the strength of the kth association relationship, such as the number of co-visits or citations, and the value is specified as 0-1.

[0067] S1.5. Determine data freshness:

[0068] Data freshness reflects the timeliness and current applicability of data. Fresh data usually means that the data has been updated recently, has high timeliness and real-time application value. It is a key indicator for evaluating whether data can meet business needs in a timely manner and support quick decision-making. Unlike data update frequency, data freshness focuses on the latest update time of data, reflecting the timeliness and current application value of data. Optionally, data freshness F nThe calculation method is as follows:

[0069]

[0070] Among them, F n Indicates data freshness; t c Indicates the current time; T u Indicates the most recent update time of the data; α represents the adjustment parameter, and the specific value can be adjusted appropriately according to the specific situation of the data.

[0071] S2. Data preprocessing:

[0072] S3.1. Data standardization:

[0073] After collecting and calculating access frequency, update frequency, business importance, file relevance, and data freshness, these indicators are standardized. The purpose of standardization is to scale features of different dimensions to the same scale to ensure that each feature contributes to the comprehensive score in a balanced manner during subsequent analysis and model training. Here, the Min-Max standardization method is proposed to scale the value of each indicator to the range of [0, 1] for subsequent comprehensive score calculation and hot and cold data determination. The Min-Max standardization formula is given below:

[0074]

[0075] Among them, X' represents the standardized value; X is the original data, max(X) and min(X) represent the minimum and maximum values ​​in the data set respectively.

[0076] After data standardization, indicators such as data access frequency, data update frequency, business importance, file relevance, and data freshness have been normalized to the [0, 1] range.

[0077] S3.2. Perform weighted calculation on the standardized indicators in step S3.1 to comprehensively evaluate the value of each data point:

[0078] In order to reflect the relative importance of different indicators to data value assessment, a weight needs to be assigned to each standardized indicator. The Theil Index method is used here to determine the weight of each indicator. This method can effectively measure the imbalance of each indicator and assign weights based on the degree of variation of each indicator in the data set, thereby more objectively reflecting its importance in data value assessment. Before calculating the Theil Index, you need to first calculate the mean of each indicator. The calculation formula is as follows:

[0079]

[0080] Among them, x jrepresents the mean of the jth standardized index; N represents the total number of data points; x' ij represents the jth standardized index value of the ith data point; after calculating the standardized means of the five indicators such as the access frequency, the Theil coefficient of each indicator is calculated according to the following formula:

[0081]

[0082] Among them, X' ij Represents the jth standardized index value of the ith data point.

[0083] The weight calculation formula for each indicator is:

[0084]

[0085] Among them, w j represents the weight of the jth standardized indicator; M represents the total number of indicators; T j The Theil index represents the jth standardized indicator. Multiply the standardized indicator values ​​of each data point by the corresponding weight, and sum up the weighted values ​​to obtain the comprehensive evaluation value of each data point, that is, the data value V i The final calculation method is:

[0086]

[0087] Among them, w j It represents the weight of the jth standardized indicator, and the meanings of other variables are the same as those mentioned above. So far, the process of evaluating data value based on multiple objectives has been calculated. The following steps will make specific judgments on hot and cold data based on the calculated data value.

[0088] S3, cold and hot data determination and storage:

[0089] Optionally, according to the 80 / 20 rule, in data storage and management, usually 80% of access requests are concentrated on 20% of the data. This embodiment can be based on this principle, first sorting the data according to the data value, dividing the data into hot data, warm data and cold data, and storing them in different storage spaces in layers, and performing hierarchical division according to the storage location and network transmission speed. Among them, the server memory or cache can store the 20% hot data with the highest data value; the local storage system or the off-site storage system with a network transmission speed of more than one gigabit can store warm data with a data value between 20% and 50%; the off-site storage system stores the 50% cold data with the lowest data value. The specific operation steps are as follows:

[0090] S3.1. Data sorting:

[0091] Optionally, sort the data in descending order based on the comprehensive evaluation value.

[0092] S3.2, Layered threshold setting:

[0093] Optionally, data with a comprehensive score greater than a first preset value is classified as hot data; data with a comprehensive score between the first preset value and a second preset value is classified as warm data; data with a comprehensive score lower than the second preset value is classified as cold data; wherein the first preset value is greater than the second preset value. Specifically:

[0094] S3.2.1. Memory threshold: The 80th percentile of the comprehensive score, that is, the 20% of data with the highest comprehensive score will be classified as hot data and stored in memory.

[0095] S3.2.2, SSD threshold: The 50th percentile of the comprehensive score, that is, data with a comprehensive score between 50% and 80% will be classified as warm data and stored in the SSD.

[0096] S3.2.3, HDD data: Data with a comprehensive score lower than 50% will be classified as cold data and stored in the HDD.

[0097] S3.3. Data classification: Classify data into hot data, warm data, and cold data based on stratification thresholds.

[0098] S3.4, Storage Strategy:

[0099] Optionally, store hot data in memory to provide the fastest access speed. Store warm data in local storage systems to provide faster access speed. Store cold data in off-site storage systems to provide the lowest cost storage solution.

[0100] Embodiment 2:

[0101] This embodiment provides a cold and hot data storage system for a cross-domain heterogeneous storage system, including:

[0102] The data collection module is configured to: obtain multiple indicators of data related to the cross-domain heterogeneous storage system; wherein the multiple indicators include access frequency, data update frequency, business importance, file relevance, and data freshness;

[0103] The standardization processing module is configured to: perform standardization processing on multiple indicators;

[0104] The comprehensive evaluation value calculation module is configured to: perform weighted summation on the multiple indicators after standardization to obtain a comprehensive evaluation value for each data point;

[0105] The data classification module is configured to: sort all data points according to the comprehensive evaluation value, and classify the data into hot data, warm data, and cold data according to the stratified threshold method;

[0106] The storage management module is configured to store hot data, warm data and cold data in different storage spaces in a hierarchical manner.

[0107] The working method of the system is the same as the cold and hot data storage method for a cross-domain heterogeneous storage system in Example 1, and will not be described in detail here.

[0108] Embodiment 3:

[0109] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the cold and hot data storage method for a cross-domain heterogeneous storage system described in Embodiment 1 are implemented.

[0110] Embodiment 4:

[0111] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, the steps of the hot and cold data storage method for a cross-domain heterogeneous storage system described in Example 1 are implemented.

[0112] Embodiment 5:

[0113] This embodiment provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the cold and hot data storage method for a cross-domain heterogeneous storage system described in Embodiment 1 are implemented.

[0114] The above description is only a preferred embodiment of the present embodiment and is not intended to limit the present embodiment. For those skilled in the art, the present embodiment may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiment shall be included in the protection scope of the present embodiment.

Claims

1. A method for storing hot and cold data in a cross-domain heterogeneous storage system, characterized in that: include: Obtain multiple indicators of data related to cross-domain heterogeneous storage systems; the multiple indicators include access frequency, data update frequency, business importance, file relevance, and data freshness; Standardize multiple indicators; Perform weighted summation on multiple indicators after standardization to obtain a comprehensive evaluation value for each data point; Sort all data points according to the comprehensive evaluation value and classify the data into hot data, warm data and cold data according to the stratified threshold method; Store hot data, warm data, and cold data in different storage spaces.

2. A method for storing hot and cold data in a cross-domain heterogeneous storage system according to claim 1, characterized in that: The access frequency is equal to the ratio of the sum of all access times recorded in a certain time period to the time period; The data update frequency is equal to the ratio of the sum of all update times within a certain period of time to the period of time; Importance of the business b for: Among them, R j The criticality score of the jth business; W j Indicates the importance weight of the jth business to the overall business; The file relevance is equal to the ratio of the strength of all association relationships to the number of association relationships, where the strength of the association relationship is the number of common visits or citations; The data freshness F n for: Among them, t c Indicates the current time; T u Indicates the last update time of the data; α indicates the adjustment parameter.

3. A method for storing hot and cold data in a cross-domain heterogeneous storage system according to claim 1, characterized in that: The values ​​of the access frequency, the data update frequency, the business importance, the file relevance, and the data freshness are scaled to within a preset range.

4. A method for storing hot and cold data in a cross-domain heterogeneous storage system according to claim 1, characterized in that: The weight of each indicator is: Among them, w j represents the weight of the jth standardized indicator; M represents the total number of indicators; T j represents the Theil index of the jth standardized indicator.

5. The method for storing hot and cold data in a cross-domain heterogeneous storage system according to claim 1, characterized in that: Sort the data in descending order according to the comprehensive evaluation value; classify the data with a comprehensive score greater than a first preset value as hot data; classify the data with a comprehensive score between the first preset value and the second preset value as warm data; classify the data with a comprehensive score lower than the second preset value as cold data; wherein the first preset value is greater than the second preset value.

6. A method for storing hot and cold data in a cross-domain heterogeneous storage system according to claim 1, characterized in that: The hot data is stored in the memory, the warm data is stored in the local storage system, and the cold data is stored in the off-site storage system.

7. A hot and cold data storage system for cross-domain heterogeneous storage systems, characterized in that: include: The data collection module is configured to: obtain multiple indicators of data related to the cross-domain heterogeneous storage system; wherein the multiple indicators include access frequency, data update frequency, business importance, file relevance, and data freshness; The standardization processing module is configured to: perform standardization processing on multiple indicators; The comprehensive evaluation value calculation module is configured to: perform weighted summation on the multiple indicators after standardization to obtain a comprehensive evaluation value for each data point; The data classification module is configured to: sort all data points according to the comprehensive evaluation value, and classify the data into hot data, warm data, and cold data according to the stratified threshold method; The storage management module is configured to store hot data, warm data and cold data in different storage spaces in a hierarchical manner.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the cold and hot data storage method for a cross-domain heterogeneous storage system as described in any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: When the processor executes the program, the steps of the hot and cold data storage method for a cross-domain heterogeneous storage system are implemented as described in any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the cold and hot data storage method for a cross-domain heterogeneous storage system are implemented as described in any one of claims 1 to 6.

Images