Intelligent tiered storage method for unstructured data

Abstract

The application discloses an intelligent hierarchical storage method for unstructured data and relates to the technical field of unstructured data processing, and the specific process is as follows: determining an initial value through data feature analysis; outputting a dynamic value by establishing a decay model; generating a preliminary storage level according to the value decay degree; training a prediction model by using access records to evaluate future access probability; adjusting the data value weight according to the access probability; and determining the optimal storage level based on the adjusted comprehensive value index. Through the double value evaluation mechanism, the method realizes the dynamic matching of storage resources and access heat; by fusing various reference factors, the method realizes the real-time dynamic quantification of data value and breaks through the limitation of static evaluation; through the collaborative optimization of the value decay model and machine learning prediction, the core pain points of the traditional method, such as storage strategy lag and low resource utilization, are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unstructured data processing, in particular to an intelligent hierarchical storage method for unstructured data. BACKGROUND

[0002] With the rapid development of big data and artificial intelligence technology, the scale of unstructured data (such as text, image, audio, video, etc.) presents an explosive growth. According to IDC statistics, unstructured data has accounted for more than 80% of the total amount of global data, and its storage management has become the core challenge of enterprise IT infrastructure. Traditional hierarchical storage technology mainly divides data into hot, warm, cold and other levels based on static rules such as data access frequency and storage cost, but has the following limitations: single value evaluation dimension: existing methods mostly rely on manual preset strategies (such as LRU-based access frequency statistics), without considering multi-dimensional characteristics such as data time decay, content sensitivity, business value, etc., leading to a disconnection between storage decisions and actual data value. For example, medical image data needs to be stored for a long time due to compliance requirements, but its access frequency may be very low, and traditional cold storage strategies cannot meet the compliance and rapid retrieval requirements. SUMMARY

[0003] In view of the deficiencies in the prior art, the present application provides an intelligent hierarchical storage method for unstructured data.

[0004] In order to achieve the above purpose, the technical scheme of the present application is as follows:

[0005] The intelligent hierarchical storage method for unstructured data comprises the following steps:

[0006] Obtain unstructured data, and determine the initial value of the unstructured data based on the unstructured data;

[0007] Construct a decay function based on the initial value of the unstructured data, and obtain the decayed decay value;

[0008] Determine a storage level factor one according to the decayed decay value, and generate a storage level decision one according to the storage level factor one;

[0009] Obtain the historical access record of the unstructured data, and determine the access probability prediction value of the unstructured data according to the historical access record;

[0010] According to the access probability prediction value, the decay value of the unstructured data is corrected, and the corrected decay value is output;

[0011] According to the corrected decay value, determine a storage level factor two, and generate a storage level decision two.

[0012] The application further discloses an intelligent hierarchical storage system for unstructured data, characterized in that the intelligent hierarchical storage method for unstructured data is executed, and the intelligent hierarchical storage system comprises

[0013] A data acquisition and preprocessing module is configured to acquire unstructured data, and extract a data generation timestamp and a historical access record of the unstructured data based on the unstructured data.

[0014] An initial value evaluation module is configured to determine a timeliness factor, an access value factor and a data sensitivity, and determine an initial value based on the determined timeliness factor, access value factor and data sensitivity; wherein the timeliness factor and the access value factor are determined based on the data generation timestamp and the historical access record, respectively.

[0015] A value decay and correction module is configured to construct a decay function based on the initial value of the unstructured data, and obtain a corrected decay value according to an access probability prediction value.

[0016] A storage decision and execution module is configured to determine a storage level factor one according to the decayed decay value, and generate a storage level decision one according to the storage level factor one; determine a storage level factor two according to the corrected decay value, and generate a storage level decision two according to the storage level factor two; and generate a storage level decision three based on a hysteresis interval and a fluctuation compensation value to judge a migration condition.

[0017] A dynamic parameter optimization module is configured to update a sensitivity weight, and update a feature lifetime parameter and a decay curve shape parameter based on a storage cost and an access delay; and dynamically adjust an upper threshold and a lower threshold of the hysteresis interval according to a fluctuation standard deviation of the decayed decay value.

[0018] Compared with the prior art, the application has the following beneficial effects:

[0019] 1. The application breaks through the traditional single frequency statistics, and integrates timeliness decay, access variance normalization and multi-modal sensitivity into a unified quantitative framework, thereby effectively reducing the error rate; the application automatically identifies data that needs to be stored for a long time or stored with high security through sensitivity analysis, thereby reducing the cost of manual annotation.

[0020] 2. Different storage decisions are generated based on the decay value and the corrected value, respectively, strategy redundancy and cross-validation are supported, decision one reflects the natural decay state of data, and decision two reflects external access intervention, thereby facilitating administrators to perform root cause analysis. DETAILED DESCRIPTION

[0021] The disclosure of the application will be described with reference to the accompanying drawings. It should be understood that the drawings are only for illustrative purposes, and are not intended to limit the scope of protection of the application. In the drawings, the same reference numerals are used to refer to the same components. Among them:

[0022] Figure 1 is a step flowchart of the present application;

[0023] Figure 2 is an initial value data flowchart of the present application;

[0024] Figure 3 is a revised decay value data flowchart of the present application;

[0025] Figure 4 is a data sensitivity update flowchart of the present application;

[0026] Figure 5 is a migration judgment flowchart of the present application. DETAILED DESCRIPTION

[0027] It is easy to understand that according to the technical solution of the present application, a person skilled in the art can propose a plurality of structure modes and implementation modes which can be replaced with each other without changing the essential spirit of the present application. Therefore, the following specific embodiments and the accompanying drawings are only exemplary descriptions of the technical solution of the present application, and should not be regarded as the whole or as the limitation or restriction of the technical solution of the present application.

[0028] SUMMARY

[0029] As described above, unstructured data (such as images, videos, logs, documents, etc.) has become the core asset of enterprises with the popularization of cloud computing, Internet of Things and other technologies, and due to the huge volume, it is difficult to store data.

[0030] The following are two relatively common processing solutions:

[0031] Depend on manual preset strategy (such as LRU algorithm statistics recent access times) or simple rules (such as file creation time), without comprehensive consideration of data time decay, content sensitivity, business relevance and other multiple features.

[0032] Adopt fixed archiving period (such as 30 days later automatically degrade), but the data value decay law is significantly different due to type and scene difference.

[0033] However, the above solutions have their own shortcomings.

[0034] For the first kind, medical image data needs to be saved for a long time due to compliance requirements, but its access frequency may be extremely low. Although the traditional cold storage strategy reduces the cost, it leads to a sharp rise in delay when urgently reviewing, affecting the efficiency of diagnosis and treatment.

[0035] For the second kind, the heat of news video usually presents a "burst period-slow decay" curve, and the fixed life cycle model will degrade it too early, missing the long-tail access benefit.

[0036] In view of the above defects in the prior art, the basic idea of the present application is to combine rules and machine learning models, dynamically optimize data storage strategies through a value decay model and a self-learning compensation mechanism. Specifically, first, the value decay model is used to calculate the value of data decay over time, and then the self-learning compensation mechanism is used to dynamically adjust the decay curve by combining access probability prediction and value correction, thereby maximizing the utilization value of data while optimizing storage costs. Finally, the system dynamically selects the most suitable storage strategy based on the corrected value, achieving efficient and intelligent data management.

[0037] After introducing the basic idea of the present application, the embodiments of the present application will be specifically introduced with reference to the accompanying drawings.

[0038] Embodiment one:

[0039] As shown in the figure, the intelligent hierarchical storage method for unstructured data includes the following steps: Figure 1

[0040] Obtain unstructured data and determine the initial value of the unstructured data based on the unstructured data.

[0041] As shown in the figure, determining the initial value of the unstructured data based on the unstructured data includes: Figure 2

[0042] Based on the unstructured data, obtain the data generation timestamp of the unstructured data , the historical access record of the th access event , and data sensitivity .

[0043] According to the data generation timestamp , calculate the timeliness factor :

[0044] , where is the current time, is the industry decay coefficient.

[0045] According to the historical access record , calculate the access value factor :

[0046] , where, is the time decay weight of the th access event, is the anti-zero constant, is the variance of the historical access record .

[0047] ​​Generally, the closer the access event is to the current time, the greater the weight; the farther the access event is from the current time, the smaller the weight. The commonly used time decay weight calculation method is the exponential decay model:

[0048] where, is the current time, is the time of the th access event, is the decay coefficient, which controls the speed of weight decay over time.

[0049] Suppose the historical access records are as follows:

[0050]

[0051] Calculate the access value factor :

[0052] .

[0053] According to the timeliness factor , the access value factor , and the data sensitivity , calculate the initial value :

[0054] where , , are the weights of the timeliness factor , the access value factor , and the data sensitivity , respectively, in calculating the initial value .

[0055] The calculation process of the data sensitivity is as follows:

[0056] Use a natural language processing model to detect sensitive keywords (such as "ID number" and "bank card number") in unstructured data and calculate the text sensitivity ;

[0057] Example: .

[0058] Use a target detection model to identify sensitive areas (such as faces and license plates) in unstructured data and calculate the image sensitivity ;

[0059] Example: .

[0060] Compute audio sensitivity using a voiceprint recognition model to detect sensitive voice content (e.g. personal identity information) ;

[0061] Example: .

[0062] Tag data with compliance labels (e.g. GDPR=0.9, HIPAA=0.95) to assign compliance sensitivity values according to regulatory requirements ;

[0063] Example: .

[0064] Assign scores according to the degree of impact of data on business (e.g. transaction logs=0.9, test data=0.2) to compute business value sensitivity ;

[0065] Example: .

[0066] Compute data sensitivity :

[0067] ,

[0068] where , , , , are the weights of text sensitivity , image sensitivity , audio sensitivity , compliance sensitivity values , business value sensitivity respectively, and , is the dynamic weight of the th factor.

[0069] Example:

[0070] Financial transaction log data:

[0071] Data content: "bank card number" keyword detected, =0.8.

[0072] Regulatory compliance: protected by GDPR, =0.9.

[0073] Business value: transaction logs are crucial to business, =0.9.

[0074] Overall sensitivity: .

[0075] Test data:

[0076] Data content: No sensitive keywords, = 0.1.

[0077] Regulatory compliance: No special regulatory requirements, = 0.5.

[0078] Business value: Test data has low impact on business, = 0.2.

[0079] Overall sensitivity: .

[0080] Construct a decay function based on the initial value of unstructured data, and obtain the decayed decay value.

[0081] The decayed decay value is:

[0082] ;

[0083] Where, is the initial value;

[0084] is the feature lifetime parameter, indicating the time required for the data value to decay to 36.8% of the initial value, controlling the overall shape of the decay curve, The larger the value, the slower the decay rate.

[0085] Example:

[0086] Hours: Indicates that the data value decays to 36.8% of the initial value after 72 hours.

[0087] is the decay curve shape parameter, which controls the shape of the decay curve.

[0088] : The decay rate gradually slows down (fast decay in the early stage, slow decay in the later stage);

[0089] : The decay rate is constant (exponential decay);

[0090] : The decay rate gradually accelerates (slow decay in the early stage, fast decay in the later stage).

[0091] Example:

[0092] : Indicates that the data value decay rate gradually accelerates.

[0093] ​The value of data at time t reflects the current importance or sensitivity of the data, and is used to determine the storage level and migration strategy of the data.

[0094] According to the decayed decay value, a storage level factor one is determined, and a storage level decision one is generated according to the storage level factor one;

[0095] The storage level is divided according to the value threshold, and the specific table is as follows:

[0096] Storage tier Media type Replica policy Activation condition Hot NVMe SSD Three replicas V(t) > 0.8 Warm QLC SSD Compression + double replica 0.4 ≤ V(t) ≤ 0.8 Cold Blue light bank Erasure code V(t) < 0.4

[0097] As shown in the above table, the decay function As the input of the storage level decision, the data is divided into hot, warm and cold levels for storage according to the decay value.

[0098] As shown in Figure 3 , the historical access record of unstructured data is obtained, and the access probability prediction value of unstructured data is determined according to the historical access record;

[0099] The implementation steps of determining the access probability prediction value of unstructured data according to the historical access record are as follows:

[0100] Input the historical access record , and normalize the historical access record to the interval , cut the time series data into fixed length sliding windows as input features of the model;

[0101] Using the trained LSTM-TCN hybrid model, the sliding window is processed to obtain the access probability prediction value , ;

[0102] In the LSTM-TCN hybrid model, the loss function is:

[0103] , wherein is the true value, is the predicted value;

[0104] Using the Adam optimizer, the data set is divided into training set and validation set, and the model is iteratively trained until the loss function converges or the maximum training number of rounds is reached.

[0105] According to the access probability prediction value, the decay value of the unstructured data is corrected, and the corrected decay value is output;

[0106] The implementation process is as follows:

[0107] ​Obtain the decay value of the current time point ; ; ;

[0108] Perform value correction calculation to obtain the corrected decay value , the correction formula is:

[0109] , wherein, is an industry correction coefficient, used to control the amplitude of correction, 0.5 is a benchmark value, used to judge the level of access probability.

[0110] If , it means that the access probability is high;

[0111] At this time is positive, and the correction factor is also positive.

[0112] In the formula, is greater than 1, so , that is, the value is corrected upwards.

[0113] If , it means that the access probability is low;

[0114] At this time is negative, and the correction factor is also negative.

[0115] In the formula, is less than 1, so , that is, the value is corrected downwards.

[0116] If , it means that the access probability is at the benchmark level;

[0117] At this time , the correction factor is 0.

[0118] In the formula, , so , that is, the value remains unchanged.

[0119] Example:

[0120] Use the LSTM-TCN model to predict the access probability for the next 7 days: ;

[0121] Calculate the access probability deviation: ;

[0122] Calculate the correction factor: ;

[0123] Dynamic correction decay: .

[0124] According to the revised decay value, the storage level factor two is determined, and the storage level decision two is generated.

[0125] Embodiment two, this embodiment is based on embodiment one, increase the process of data sensitivity adjustment, as shown in Figure 4 ;

[0126] After calculating the data sensitivity , the feedback score of each factor is calculated using the reinforcement learning model , and the weight is dynamically adjusted:

[0127] , wherein is the exponential function value of the feedback score of the i-th factor, is a normalization factor, ensuring that the sum of all weights is 1.

[0128] According to the dynamically adjusted weight, the data sensitivity is updated again .

[0129] Example:

[0130] Assume that the values of text sensitivity , image sensitivity , audio sensitivity , compliance sensitivity value , business value sensitivity are 0.8, 0.6, 0.9, 0.5, and 1.0, respectively, and the initial feedback scores of the five parameters are 3, 2, 4, 1, and 5, respectively.

[0131] Calculate ;

[0132] Calculate the normalization factor ;

[0133] Calculate the weight ;

[0134] Update the weight using the reinforcement learning model (suppose it is updated to 4, 3, 5, 2, and 6);

[0135] Recalculate the weight ;

[0136] Recalculate the sensitivity .

[0137] The calculation of data sensitivity is dynamically optimized through the reinforcement learning model, thereby improving the accuracy and adaptability of data processing. ​

[0138] Specifically: use reinforcement learning model to calculate feedback score of each factor , and amplify score difference through exponential function . Adjust weight of each factor dynamically according to feedback score , ensure that weight can reflect actual importance of each factor. Recalculate data sensitivity according to adjusted weight, make it more accurate and reliable. Through dynamic adjustment of weight, data sensitivity can adapt to changes in data characteristics and environment, improve data processing and analysis effect. Make the calculation process of data sensitivity have self-learning and adaptive ability, reduce manual intervention, and improve the intelligent level of the system.

[0139] This step dynamically adjusts the weight through the reinforcement learning model, optimizes the calculation of data sensitivity, and ensures that data processing is more accurate, flexible and intelligent, providing a more reliable basis for subsequent storage strategy optimization and data analysis.

[0140] In example three, based on example one, the following process is added:

[0141] Based on storage cost , access delay , update parameter characteristic life parameter and decay curve shape parameter using Bayesian optimization framework.

[0142] Bayesian optimization is a global optimization method based on probability model, suitable for scenarios where target function calculation cost is high or cannot be directly differentiated.

[0143] Example:

[0144] Initial parameters: , ;

[0145] Input: storage cost =1000 dollars, access delay =50 milliseconds;

[0146] Decay function: ;

[0147] Process: update parameters using Bayesian optimization:

[0148] , ;

[0149] Input: optimized decay function: ;

[0150] Through Bayesian optimization framework, dynamically update characteristic life parameter ​and decay curve shape parameters , which can effectively balance storage cost and access delay, optimize data storage strategy. This method has the following advantages:

[0151] Global optimization: Bayesian optimization can find global optimal solution, avoid falling into local optimum.

[0152] Efficiency: through the surrogate model and acquisition function, reduce the number of times of calculating objective function.

[0153] Adaptability: can dynamically adjust parameters and decay curve according to data characteristics and environmental changes.

[0154] Embodiment four, this embodiment is based on embodiment one, increase the process of migration judgment;

[0155] Specifically as Figure 5 shown, the specific implementation is as follows:

[0156] The storage hierarchy factor determined according to the decayed decay value further comprises:

[0157] The upper threshold value of the hysteresis interval , the lower threshold value ;

[0158] The fluctuation standard deviation of the decayed decay value :

[0159] The fluctuation compensation value , , wherein is a compensation constant;

[0160] Judge the migration condition:

[0161] ;

[0162] Temperature migration: data from cold layer to warm layer or warm layer to hot layer;

[0163] Temperature migration: data from hot layer to warm layer or warm layer to cold layer;

[0164] Generate storage hierarchy decision three.

[0165] Example: the decayed decay value is 0.81, the upper threshold value is 0.8, the lower threshold value is 0.4, the compensation constant is 0.1, and the fluctuation standard deviation of the decayed decay value is 0.11.

[0166] Calculate the fluctuation compensation value .

[0167] Temperature-induced migration threshold: 0.8 + 0.011 = 0.811;

[0168] Cooling migration threshold: 0.4 − 0.011 = 0.389;

[0169] Current value Migration is not triggered if the temperature rise migration threshold is not exceeded.

[0170] By introducing a hysteresis interval, frequent migrations caused by short-term fluctuations in data value are avoided, reducing system load. Ineffective migration operations are reduced, and storage resource allocation is optimized. The hysteresis interval is dynamically adjusted to adapt to the fluctuating characteristics of different data values.

[0171] Standard deviation of the fluctuation of the decayed value It is based on the collection time window The attenuation value after the internal attenuation sequence The obtained sequence ,in These are the sampling time points within the time window. The number of samples; the standard deviation of the fluctuation of the attenuated value after attenuation. The calculation formula is:

[0172] In the formula, For data value sequence The mean.

[0173] Standard deviation of the fluctuation of the decayed value Used to measure the attenuation value after decay. The degree of fluctuation reflects the stability of data value. It is a statistical measure of the change in data value over a period of time, used to optimize storage migration strategies and avoid frequent migrations.

[0174] Example: In the time window The attenuation value is collected once per hour within the hour. :

[0175] ;

[0176] Calculate the mean: ;

[0177] Calculate the variance: ;

[0178] Calculate the standard deviation: .

[0179] By introducing , avoid frequent migration caused by short-term fluctuation of data value, reduce system load; dynamically adjust hysteresis interval, adapt to fluctuation characteristics of different data values; reduce invalid migration operation, optimize storage resource allocation.

[0180] The technical scope of the present application is not limited to the above description, and those skilled in the art can make various modifications and changes to the above embodiments without departing from the technical idea of the present application, and these modifications and changes should all belong to the protection scope of the present application.

Claims

1. An intelligent tiered storage method for unstructured data, characterized in that: The method comprises the following steps: acquiring unstructured data and determining an initial value of the unstructured data based on the unstructured data; constructing a decay function based on the initial value of the unstructured data and obtaining a decayed decay value; determining a storage level factor one according to the decayed decay value and generating a storage level decision one according to the storage level factor one; acquiring a historical access record of the unstructured data and determining an access probability prediction value of the unstructured data according to the historical access record; correcting the decay value of the unstructured data according to the access probability prediction value and outputting a corrected decay value; determining a storage level factor two according to the corrected decay value and generating a storage level decision two; the method for determining the initial value of the unstructured data based on the unstructured data comprises: Data generation timestamp based on unstructured data acquisition of unstructured data , the first access event history access record and data sensitivity ; Generating a timestamp according to data Computing a time-to-live factor : wherein is the current time, is the industry decay coefficient; According to historical access records Computing an access value factor : wherein is the time decay weight for the is a prevention zero constant, is the variance of the historical access record ; According to the timeliness factor , the access value factor , the data sensitivity , the initial value is calculated : wherein , , are, in sequence, a timeliness factor , an access value factor , a data sensitivity weighting the initial value .

2. The intelligent tiered storage method for unstructured data according to claim 1, wherein: The data sensitivity The calculation process is as follows: Detecting sensitive keywords in unstructured data using natural language processing models and calculating text sensitivity ; Identifying sensitive areas in unstructured data using a target detection model, computing image sensitivity ; Detecting sensitive speech content using a voiceprint recognition model, computing audio sensitivity ; Tag data with compliance labels, assign compliance sensitivity values according to regulatory requirements ; According to the data, the degree of influence on the business is given a score, and the business value sensitivity is calculated ; Computing data sensitivity : wherein , , , , the weights of the text sensitivity , the image sensitivity , the audio sensitivity , the compliance sensitivity value , the business value sensitivity , and , is the dynamic weight of the th factor. 3.The intelligent tiered storage method for unstructured data of claim 2, wherein: calculating the feedback score of each factor using a reinforcement learning model and dynamically adjusting the weight: , where is an exponential function value of the feedback score of the th factor; According to the dynamically adjusted weight, the data sensitivity is updated again .

4. The intelligent tiered storage method for unstructured data according to claim 1, wherein: the decayed decay value is: wherein, is an initial value, is a characteristic lifetime parameter, is a decay curve shape parameter; Based on storage costs , access latency , updating the parameter characteristic lifetime parameter and decay curve shape parameter with a Bayesian optimization framework 5. The intelligent tiered storage method for unstructured data according to claim 4, wherein: the method for determining the storage level factor one according to the decayed decay value further comprises: defining an upper threshold of the hysteresis interval , a lower threshold ​ computing a wavelet standard deviation of the attenuated values of the attenuation : Computing a wave fluctuation compensation value , wherein is a compensation constant; judging the migration condition: ； generating a storage level decision three.

6. The intelligent tiered storage method for unstructured data according to claim 5, wherein: The fluctuation standard deviation of the decayed decay value is obtained according to a data value sequence in a collection time window The data value sequence in the collection time window is obtained according to a data value sequence in a collection time window Wherein are sampling time points in the time window respectively, is the sampling number; and the calculation formula of the fluctuation standard deviation of the decayed decay value is: wherein is the mean of the data value sequence .

7. The intelligent tiered storage method for unstructured data according to claim 6, wherein: The implementation steps for determining the access probability prediction value of the unstructured data according to the historical access record are as follows: input historical access records and normalizing the historical access records to intervals, the time series data is cut into fixed-length sliding windows as input features for the model; Using the trained LSTM-TCN hybrid model, the sliding window is processed to obtain the access probability prediction value , ; in the LSTM-TCN hybrid model, the loss function is: wherein is the actual observed historical access record at the th time point, is the predicted number of accesses at the th time point predicted by the model from the input features, represents the total number of data samples used to compute the loss; using the Adam optimizer to divide the data set into a training set and a validation set, iteratively training the model until the loss function converges or the maximum number of training rounds is reached.

8. The intelligent tiered storage method for unstructured data according to claim 7, wherein: The implementation process of correcting the decay value of the unstructured data according to the access probability prediction value is as follows: acquiring a current time point of the decayed decay value and the access probability prediction value ; A value correction calculation is performed to obtain a corrected decay value The correction formula is: wherein, is an industry correction factor.

Images