Data centralized control platform intelligent hierarchical storage system based on distributed architecture

Abstract

The invention discloses a data centralized control platform intelligent hierarchical storage system based on a distributed architecture, and relates to the field of data storage. According to the data centralized control platform intelligent hierarchical storage system based on the distributed architecture, key index characteristics of data are continuously collected through an intelligent hierarchical engine, a corresponding quantification mode is independently designed for each type of characteristics of the data for quantification processing, quantification results are normalized, then a comprehensive value score calculation formula is introduced, and a comprehensive value score is obtained. By designing a heterogeneous storage pool and integrating various storage rear ends, a logical storage hierarchy is formed, grading is carried out according to access characteristics of the data, and threshold division association is carried out on the data and the comprehensive value score of the data, so that the data can be automatically subjected to data grading storage according to a scoring rule; the data storage scheme is simplified, and meanwhile it is ensured that the high-value data has the advantages of being high in speed, low in delay and rapid in response during access.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, in particular to an intelligent hierarchical storage system of a data set control platform based on a distributed architecture. BACKGROUND

[0002] The data set control platform is a platform for centralized management and control of data, which provides data access, metadata management, data quality management, data directory, data provenance, permission control, lifecycle policy definition and other functions, and realizes comprehensive control of data assets.

[0003] In such a data set control platform, the data flow is large, and the total amount of data is large, which makes the maintenance cost of the entire data set control platform high. SUMMARY

[0004] In view of the shortcomings of the prior art, the present application provides an intelligent hierarchical storage system of a data set control platform based on a distributed architecture, which solves the problem that the data set control platform has large data flow and volume, and in order to ensure timely response when data is called, it is necessary to ensure that the commonly used data stream has the characteristics of high speed, low delay and fast response, which makes the entire data set control platform data storage framework bloated, the data classification is not clear, and the cost of data storage service is high.

[0005] To achieve the above purpose, the present application is realized by the following technical scheme: an intelligent hierarchical storage system of a data set control platform based on a distributed architecture, comprising a data access layer, a data set control platform layer, an intelligent hierarchical engine, a distributed storage layer and a health and alarm layer.

[0006] The intelligent hierarchical engine is used for continuously collecting key indicator characteristics of data, and then intelligently classifying according to data characteristics.

[0007] CVS = ∑(W i · F i-norm )

[0008] Wherein CVS represents the comprehensive value score CVS ∈ (0, 1); Wi W i i represents the weight of the i-th feature, (∑W i i=1 i-norm ) = 1, and W i-norm i ∈ (0, 1); i F now i represents the value of the i-th feature after quantization and normalization, and F i i ∈ (0, 1);

[0009] By setting a heterogeneous storage pool in the distributed storage layer, integrating multiple storage backends, forming a logical storage hierarchy, and then storing data in different storage locations according to the CVS value of the data.

[0010] Preferably, the key indicator features of the data include access heat, access frequency, business value label, data size, retention period, and data correlation degree. The intelligent grading engine allocates weight coefficients to the access heat, access frequency, business value label, data size, retention period, and data correlation degree according to the data business type according to the features of the data,

[0011] m = a + b + g + l + x + f,

[0012] where m = 1, and a, b, g, l, x, f ∈ (0, 1).

[0013] Preferably, the quantification of the access heat uses a time decay model. The closer the data's most recent access time, the higher the access frequency, the closer the most recent access time, the higher the access frequency, and the higher the score. The higher the score, the higher the score, and the exponential decay function is used. The formula for a single access time point is:

[0014] heat_score(t i ) = e^(-l·(t now -t i ))

[0015] where t i represents the timestamp of the i-th access; t now represents the current time; and l represents the decay factor, and l > 0, the larger l, the faster the decay;

[0016] Then sum all the heat_score of the access events in the recent time window T periodically;

[0017]

[0018] At the same time, in order to avoid the absolute value difference being too large, the AH needs to be normalized to the interval [0, 1].

[0019] Preferably, the quantification method of the access frequency is to count the average number of accesses per unit time, and the formula is:

[0020]

[0021] where S 总 represents the total number of visits, T represents the unit time.

[0022] Preferably, the quantitative method of the service value label is directly labeling data by a data administrator, using gradient threshold grading;

[0023] The data size quantitative method is to set threshold labels by directly reading data storage values and dividing intervals according to gradients;

[0024] The quantitative process of the retention period is to normalize the data distance to the expiration time to the interval [0, 1] according to the length of the data distance to the expiration time, the closer to the expiration, the lower the value, and the expired data is forced to be 0;

[0025] In the quantitative process of the data correlation degree, if the data is dependent on multiple important downstream tasks or reports, it is marked as high value; the number of tasks or importance weighted sum dependent on the data is calculated, and the formula is:

[0026] LD = min(1, log10(1+Number_of_Critical_Downstream_Tasks))

[0027] After normalization, LD_norm ∈ [0, 1].

[0028] Preferably, the intelligent grading engine can match the rules defined by the policy management engine according to the data portrait, automatically determine the current storage level of the data and when it needs to be migrated, and use a learning model to predict future access patterns to make more proactive grading decisions and predict future data trends in advance.

[0029] Preferably, the data access layer integrates with the data set control platform by providing a unified data write interface to collect metadata and perform preliminary classification.

[0030] Preferably, the data set control platform layer includes:

[0031] Metadata management: store and manage the structure, attributes, sources, and blood relationship of data, etc.

[0032] Data directory: provide search, discovery, and understanding of data assets;

[0033] Data quality management: monitor and improve data accuracy, completeness, and consistency;

[0034] Data bloodline: track the origin, conversion, and flow of data;

[0035] Policy management engine: define, store and execute data tiering policies, policies based on metadata, access patterns, business rules, etc.

[0036] Privileges and security management: control access to data and storage policies.

[0037] Preferably, the distributed storage layer is used to receive the migration instruction of the intelligent tiering engine, responsible for safely and efficiently migrating data blocks or objects between storage levels in the background, while providing a unified access path and interface for users and applications, shielding the complexity of the underlying storage location and migration, and providing a storage environment for the data classified by the intelligent tiering engine by setting a heterogeneous storage pool, the heterogeneous storage pool includes;

[0038] T0, for ultra-high performance storage environment, with very low latency and very high IOPS;

[0039] T1 is a high-performance storage environment, with low latency and high IOPS;

[0040] T2 is a standard performance storage environment, with medium IOPS and high capacity;

[0041] T3 is a low-cost and large-capacity storage environment, using high-density HDD;

[0042] T4 is object archiving, with very low cost, for long-term archiving.

[0043] Preferably, the intelligent tiering engine analyzes multiple characteristics of the data, calculates the final total score of the data according to the score of each characteristic and the weight coefficient of each characteristic, and then performs intelligent tiering storage according to the threshold division of the heterogeneous storage pool.

[0044] The application discloses a kind of data set control platform intelligent tiering storage systems based on distributed architecture, it has beneficial effects as follows:

[0045] 1. The intelligent hierarchical storage system of the data set control platform based on the distributed architecture continuously collects key indicator characteristics of data through the intelligent hierarchical engine, separately designs corresponding quantification methods for each type of data characteristics for quantification processing, normalizes the quantification results, then calculates the comprehensive value score of the data by introducing a comprehensive value score calculation formula, simultaneously designs a heterogeneous storage pool in the distributed storage layer, integrates multiple storage backends through the heterogeneous storage pool, forms a logical storage hierarchy, and classifies the data according to the access characteristics and the threshold value division association of the comprehensive value score of the data, so as to realize automatic data hierarchical storage according to the scoring rules after the intelligent hierarchical engine performs intelligent comprehensive value scoring on the data, simplify the data storage scheme, and ensure that the high-value data has the characteristics of high speed, low delay and fast response during access. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0047] Fig. 1 The functional block diagram of the intelligent hierarchical storage system of the present application;

[0048] Fig. 2 The data hierarchical processing flowchart of the intelligent hierarchical engine of the present application;

[0049] Fig. 3 The data hierarchical characteristic item schematic diagram of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application are described clearly and completely. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0051] The embodiments of the present application provide an intelligent hierarchical storage system of a data set control platform based on a distributed architecture, which solves the problem that the data set control platform has large data flow and volume, in order to ensure timely response during data call, the commonly used data stream needs to have the characteristics of high speed, low delay and fast response, which makes the entire data set control platform data storage framework bloated, the data classification is not clear, and the cost of data storage service is high.

[0052] For better understanding of the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings of the specification and specific embodiments.

[0053] The embodiment of the application discloses a data set control platform intelligent hierarchical storage system based on a distributed architecture.

[0054] According to the drawings Figs. 1-3 It is shown that it includes a data access layer, a data set control platform layer, an intelligent hierarchical engine, a distributed storage layer and a health warning layer;

[0055] The intelligent hierarchical engine is used for continuously collecting key indicator characteristics of data, and then intelligently classifying according to the data characteristics. The steps of intelligent classification include feature extraction, feature quantization, weight distribution, comprehensive scoring, classification decision and prediction enhancement in turn, and a comprehensive value score calculation formula is introduced.

[0056] CVS = ∑ (W i ·F i-norm )

[0057] Wherein CVS represents the comprehensive value score CVS ∈ (0, 1); W i represents the weight of the i-th feature, (∑W i = 1, and W i ∈ (0, 1)); F i-norm represents the value of the i-th feature after quantization and normalization, and F i-norm ∈ (0, 1);

[0058] By setting a heterogeneous storage pool in the distributed storage layer, integrating multiple storage backends, forming a logical storage hierarchy, and then storing data in different storage locations according to the CVS value of the data.

[0059] The key indicator characteristics of the data include access heat, access frequency, business value label, data size, retention period and data correlation degree. The intelligent hierarchical engine respectively allocates weight coefficients to the access heat, access frequency, business value label, data size, retention period and data correlation degree according to the data business type according to the characteristics of the data,

[0060] m = α + β + γ + λ + χ + φ

[0061] Wherein m = 1, and α, β, γ, λ, χ, φ ∈ (0, 1).

[0062] The quantization of access heat adopts a time decay model. The closer the data's recent access time, the higher the access frequency, the closer the recent access time, the higher the access frequency, and the higher the score. A commonly used exponential decay function score is higher. An exponential decay function is used. The formula for a single access time point is:

[0063] heat_score(t i )=e^(-λ·(t now -t i ))

[0064] where t i denotes the timestamp of the ith access; t now denotes the current time; λ denotes the decay factor, and λ > 0, the larger λ, the faster the decay;

[0065] Then sum up the heat_score of all access events in the recent time window T periodically;

[0066]

[0067] At the same time, in order to avoid the absolute value difference being too large, the AH needs to be normalized to the interval [0, 1].

[0068] The quantification method of access frequency is to count the average number of accesses per unit time, and the formula is:

[0069]

[0070] where S 总 denotes the total number of accesses, and T denotes the unit time.

[0071] The quantification method of business value label is to directly label the data by the data administrator, and the gradient threshold classification is adopted;

[0072] The data size quantification method is to directly read the data storage value, set the threshold label according to the gradient division interval;

[0073] The quantification process of retention period is to quantify the data distance to the expiration time, the closer to the expiration, the lower the value, and the normalized value is in the interval [0, 1], and the expired data is forced to be 0;

[0074] In the quantification process of data correlation degree, if the data is dependent on multiple important downstream tasks or reports, it is marked as high value; the number of tasks or importance weighted sum dependent on the data is counted, and the formula is:

[0075] LD=min(1,log10(1+Number_of_Critical_Downstream_Tasks))

[0076] After normalization, LD_norm ∈ [0, 1].

[0077] The intelligent tiering engine can automatically determine the current storage tier of the data and when migration is needed according to the data portrait, match the rules defined by the policy management engine, and make more proactive tiering decisions by predicting future access patterns using a learning model.

[0078] The data access layer integrates with the data control platform by providing a unified data write interface for metadata collection and preliminary classification.

[0079] The data control platform layer includes;

[0080] Metadata management: stores and manages the structure, attributes, sources, and blood relationship of data;

[0081] Data directory: provides search, discovery, and understanding of data assets;

[0082] Data quality management: monitors and improves data accuracy, completeness, and consistency;

[0083] Data lineage: traces the origin, transformation, and flow of data;

[0084] Policy management engine: defines, stores, and executes data tiering policies based on metadata, access patterns, business rules, etc.;

[0085] Permission and security management: controls access to data and storage policies.

[0086] The distributed storage layer receives migration instructions from the intelligent tiering engine, safely and efficiently migrates data blocks or objects between storage tiers in the background, provides a unified access path and interface for users and applications, and hides the complexity of underlying storage locations and migration. It also provides a storage environment for data classified by the intelligent tiering engine by setting up a heterogeneous storage pool, which includes;

[0087] T0 is a super-high-performance storage environment with extremely low latency and extremely high IOPS;

[0088] T1 is a high-performance storage environment with low latency and high IOPS;

[0089] T2 is a standard-performance storage environment with medium IOPS and high capacity;

[0090] T3 is a low-cost and large-capacity storage environment using high-density HDDs;

[0091] T4 is an object archive with extremely low cost for long-term archiving.

[0092] The intelligent grading engine analyzes multiple features of data, calculates a final total score of the data according to a score of each feature and a weight coefficient of each feature by using a weighting formula, and then performs intelligent grading storage according to a threshold value division of a heterogeneous storage pool.

[0093] Working principle; the intelligent grading storage system continuously collects key indicator features of data by the intelligent grading engine, separately designs corresponding quantification methods for each type of data feature for quantification processing, normalizes the quantification results, calculates the comprehensive value score of the data by introducing a comprehensive value score calculation formula, designs a heterogeneous storage pool in the distributed storage layer, integrates multiple storage backends by the heterogeneous storage pool, forms a logical storage hierarchy, and performs grading according to the access characteristics of the data and threshold value division correlation with the comprehensive value score of the data, so as to realize automatic data grading storage according to the scoring rules after the intelligent grading engine performs intelligent comprehensive value scoring on the data, simplify the data storage scheme, and ensure that the high-value data has the characteristics of high speed, low delay and fast response during access.

[0094] The above shows and describes the basic principles and main features of the present application and the advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above-mentioned embodiments, and the above-mentioned embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A data control platform intelligent hierarchical storage system based on a distributed architecture, characterized in that, It includes a data access layer, a data control platform layer, an intelligent grading engine, a distributed storage layer, and a health and alarm layer; The intelligent grading engine is used to continuously collect key indicator features of data, and then perform intelligent grading based on the data features. The intelligent grading steps include: feature extraction, feature quantification, weight allocation, comprehensive scoring, grading decision and prediction enhancement, and introduce a comprehensive value score calculation formula. CVS=∑(W i ·F i-norm ) Where CVS represents the comprehensive value score CVS∈(0,1); W i Represents the weight of the i-th feature, (∑W i =1, and W i ∈(0,1)); F i-norm Let F represent the value of the i-th feature after quantization and normalization, and F i-norm ∈(0,1); By setting up heterogeneous storage pools in the distributed storage layer and integrating multiple storage backends to form a logical storage hierarchy, the data is then stored in different storage locations based on the data's CVS value.

2. The intelligent hierarchical storage system for a data control platform based on a distributed architecture as described in claim 1, characterized in that: Key data metrics include access popularity, access frequency, business value tags, data size, retention period, and data relevance. The intelligent grading engine assigns weights to these metrics based on the data's business type. m=α+β+γ+λ+χ+φ Where m=1, and α, β, γ, λ, χ, φ∈(0,1).

3. The intelligent hierarchical storage system for a data control platform based on a distributed architecture as described in claim 2, characterized in that: The quantification of access popularity adopts a time decay model. The more recent the access time and the higher the access frequency, the higher the score. The commonly used exponential decay function scores higher. The formula for a single access time point is as follows: heat_score(t i )=e^(-λ·(t now -t i )) Where t i This represents the timestamp of the i-th visit; t now λ represents the current time; λ represents the decay factor, and λ > 0. The larger λ is, the faster the decay. Then, the heat_score of all access events within the most recent time window T is summed periodically; Meanwhile, to avoid excessively large differences in absolute values, AH needs to be normalized to the [0,1] interval.

4. The intelligent hierarchical storage system for a data control platform based on a distributed architecture as described in claim 3, characterized in that: The method for quantifying the access frequency is by statistically analyzing the average number of accesses per unit time, as shown in the formula; Where S 总 This represents the total number of visits, and T represents the unit of time.

5. The intelligent hierarchical storage system for a data control platform based on a distributed architecture as described in claim 4, characterized in that: The method for quantifying the business value tags is that the data administrator directly tags the data, using a gradient threshold for gradation; The data size quantization method is to directly read the data storage value and set threshold labels according to the gradient division interval; The quantification process of the retention period is based on the length of time the data is away from the expiration date. The closer to the expiration date, the lower the value. After quantification, the data is normalized to the range of [0,1]. Expired data is forcibly processed as 0. In the process of quantifying the data correlation, if the data is depended on by multiple important downstream tasks or reports, it is marked as high value. The formula is as follows: The weighted sum of the number or importance of tasks that depend on this data. After normalization, LD = min(1, log10(1+Number_of_Critical_Downstream_Tasks)) is LD_norm∈[0,1].

6. The intelligent hierarchical storage system for a data control platform based on a distributed architecture as described in claim 5, characterized in that: The intelligent tiering engine can automatically determine the current storage tier of data and when it needs to be migrated based on data profiles and the rules defined by the strategy management engine. It also uses a learning model to predict future access patterns, making more proactive tiering decisions and predicting future data trends in advance.

7. The intelligent hierarchical storage system for a data control platform based on a distributed architecture as described in claim 1, characterized in that: The data access layer integrates with the data control platform by providing a unified data writing interface to perform metadata collection and preliminary classification.

8. The intelligent hierarchical storage system for a data control platform based on a distributed architecture as described in claim 1, characterized in that: The data set control platform layer includes: Metadata management: storing and managing the structure, attributes, source, lineage, etc. of data; Data Catalog: Provides search, discovery, and understanding of data assets; Data quality management: monitoring and improving the accuracy, completeness, and consistency of data; Data lineage: tracing the origin, transformation, and flow of data; Policy management engine: Defines, stores, and executes data hierarchy policies, which are based on metadata, access patterns, business rules, etc. Access and security management: Controlling access permissions to data and storage policies.

9. The intelligent hierarchical storage system for a data control platform based on a distributed architecture as described in claim 1, characterized in that: The distributed storage layer is used to receive migration instructions from the intelligent grading engine, and is responsible for safely and efficiently migrating data blocks or objects between storage layers in the background. At the same time, it provides a unified access path and interface for users and applications, shields the complexity of the underlying storage location and migration, and provides a storage environment for the data graded by the intelligent grading engine by setting up a heterogeneous storage pool. The heterogeneous storage pool includes: T0 is an ultra-high performance storage environment with extremely low latency and extremely high IOPS; T1 is a high-performance storage environment with low latency and high IOPS; T2 is a standard performance storage environment with medium IOPS and high capacity; T3 provides a low-cost, high-capacity storage environment using high-density HDDs; T4 is an object archive with extremely low cost, used for long-term archiving.

10. The intelligent hierarchical storage system for a data control platform based on a distributed architecture as described in claim 9, characterized in that: The intelligent grading engine analyzes multiple features of the data, calculates the final total score of the data based on the score of each feature and the weighting formula of each feature, and then performs intelligent grading storage according to the threshold of the heterogeneous storage pool.

Images