A machine learning-based unstructured storage tiering policy optimization method

Abstract

This invention discloses a machine learning-based method for optimizing unstructured storage tiering strategies, relating to the field of data storage management technology. The method includes: real-time monitoring of file access events, generating access logs, and extracting multi-dimensional feature datasets; using a trained multi-dimensional decision tree model to allocate corresponding storage tiers to storage objects, obtaining storage tier decisions; allocating storage objects to different storage tiers according to the storage tier decisions, and performing resource configuration and storage operations; real-time monitoring of access to storage objects in the new storage tiers, collecting file access performance, storage costs, and response times, and obtaining feedback data; by real-time monitoring of file access events and extracting multi-dimensional feature datasets containing basic attributes, access behavior, and contextual information, efficient response to hot data and rational resource utilization are ensured, significantly improving the overall performance utilization and cost-effectiveness of storage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage management, and particularly relates to a non-structured storage tiering strategy optimization method based on machine learning. BACKGROUND

[0002] With the rapid development of information technology, large-scale data storage is increasingly widely applied in the fields of cloud computing, big data analysis and artificial intelligence, and non-structured data (such as text, image, audio and video) shows an explosive growth trend. In order to cope with the storage efficiency and access performance challenges brought by massive non-structured data, storage tiering technology has gradually become a key means to improve the overall performance of the system.

[0003] When facing complex and changeable access modes, the traditional tiering strategy is difficult to fully capture the multidimensional correlation and spatiotemporal dynamic characteristics of data access behaviors, especially in the scene of high feature dimension and nonlinear change of access mode. The access strategy relies on a single or limited dimension of access features (such as access frequency and recent access time), lacks the ability to fuse and model multi-source heterogeneous features such as file metadata, user behavior context, application type and access path structure, and thus the discrimination accuracy of "cold" and "hot" data is insufficient, and mismigration or delayed migration phenomenon is prone to occur. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a non-structured storage tiering strategy optimization method based on machine learning to solve the problem of inaccurate cold and hot data discrimination and storage level migration decision lag caused by the lack of fusion modeling ability of multidimensional access features.

[0006] To solve the above technical problems, the present application provides the following technical scheme:

[0007] The present application provides a non-structured storage tiering strategy optimization method based on machine learning, which comprises: monitoring file access events in real time, generating access logs, and extracting a multidimensional feature data set; initializing a multidimensional decision tree model using the multidimensional feature data set, training the multidimensional decision tree model, and obtaining a trained multidimensional decision tree model; using the trained multidimensional decision tree model to allocate a corresponding storage level to a storage object, and obtaining a storage level decision; according to the storage level decision, allocating the storage object to different storage layers and performing resource configuration and storage operation; monitoring the access situation of the storage object in the new storage level in real time, collecting file access performance, storage cost and response time, and obtaining feedback data.

[0008] As a preferred scheme of the unstructured storage hierarchical strategy optimization method based on machine learning, the access log comprises a file identifier, an access timestamp, an operation type, a file size, a user identifier, and business information.

[0009] As a preferred scheme of the unstructured storage hierarchical strategy optimization method based on machine learning, the real-time monitoring of file access events, the generation of an access log, and the extraction of a multi-dimensional feature dataset are performed according to the following steps.

[0010] The access events of the file are continuously monitored, the access events are saved as an access log, repeated records of the access log are performed and missing values are completed, abnormal data are detected and corrected, and a preprocessed access log is generated.

[0011] The file type and file capacity are extracted from the preprocessed access log and are used as basic attribute features. The recent, medium-term, and long-term access frequencies are calculated by using a time window statistical method and are combined with timestamp analysis to obtain access behavior features.

[0012] The business association degree and data importance level are determined by analyzing the user access mode, file usage scenario, business field, and permission level in the access behavior features, and are used as context features. The basic attribute features, access behavior features, and context features are integrated and encoded by using one-hot encoding, standardization processing, and numerical scoring to generate a multi-dimensional feature dataset.

[0013] As a preferred scheme of the unstructured storage hierarchical strategy optimization method based on machine learning, the multi-dimensional decision tree model is initialized by using the multi-dimensional feature dataset according to the following steps.

[0014] The multi-dimensional feature dataset is statistically analyzed to calculate the mean, variance, maximum value, minimum value, and distribution of all features, generate a statistical analysis result, and calculate the Pearson correlation coefficient and mutual information value between the features to generate a feature correlation analysis result.

[0015] According to the statistical analysis result and the feature correlation analysis result, the maximum depth of the decision tree and the minimum number of leaf node samples are set, and the optimal feature is selected for splitting based on information gain to obtain the parameters of the initialized multi-dimensional decision tree model.

[0016] As a preferred scheme of the unstructured storage hierarchical strategy optimization method based on machine learning, the multi-dimensional decision tree model is trained to obtain a trained multi-dimensional decision tree model according to the following steps.

[0017] The multi-dimensional feature dataset is divided into a training set and a test set by using stratified sampling, and the training set and the test set are subjected to feature normalization processing.

[0018] The multi-dimensional decision tree model is trained by using the training set, a recursive partitioning algorithm is used to construct a decision tree structure, an optimal splitting feature is selected according to an information gain value at each node, and a preliminarily trained multi-dimensional decision tree model is generated;

[0019] The performance of the preliminarily trained multi-dimensional decision tree model after pruning optimization is evaluated by using the test set, the accuracy, recall rate, F1 score and confusion matrix indicators are calculated, a performance evaluation report is generated, and the decision tree parameters are optimized and adjusted to obtain the trained multi-dimensional decision tree model.

[0020] As a preferred scheme of the unstructured storage hierarchical strategy optimization method based on machine learning, wherein: the trained multi-dimensional decision tree model is used to allocate a corresponding storage level to the storage object to obtain a storage level decision, and the specific steps are as follows,

[0021] The storage object to be allocated is received, and real-time feature data of the storage object is extracted to generate a feature vector of the current storage object;

[0022] The feature vector of the current storage object is input into the trained multi-dimensional decision tree model for prediction analysis, and a preliminary storage level recommendation result is output;

[0023] The current capacity utilization, performance load and network bandwidth state of each storage layer are collected, and the preliminary storage level recommendation result is adaptively adjusted to generate a storage level decision.

[0024] As a preferred scheme of the unstructured storage hierarchical strategy optimization method based on machine learning, wherein: the current capacity utilization, performance load and network bandwidth state of each storage layer are collected, and the preliminary storage level recommendation result is adaptively adjusted to generate a storage level decision, and the specific steps are as follows,

[0025] The current capacity utilization, performance load and network bandwidth state of each storage layer are collected, and the capacity usage percentage, computing resource load status and network transmission bandwidth occupation rate of each storage layer are read in real time through a storage management interface;

[0026] The current capacity utilization, performance load and network bandwidth state are compared and analyzed with the preliminary storage level recommendation result, the preliminary storage level recommendation result is dynamically adjusted according to the adaptability rules, the priority and applicability of each storage layer are re-evaluated, and a storage level decision is generated.

[0027] As a preferred scheme of the unstructured storage hierarchical strategy optimization method based on machine learning, the method comprises the following steps of:

[0028] The target storage layer identifier is obtained by parsing the target storage layer identifier field in the storage hierarchy decision, and the corresponding storage resource configuration parameters are prepared according to the characteristics of the target storage layer.

[0029] The connection session of the target storage layer is initialized according to the storage resource configuration parameters, a secure communication link is established, access permission and resource availability are verified, and the storage object is transmitted from the source storage layer to the target storage layer through a block transmission mechanism.

[0030] As a preferred scheme of the unstructured storage hierarchical strategy optimization method based on machine learning, the method comprises the following steps of:

[0031] The access data of the storage object in the new target storage hierarchy is continuously collected, the performance indicators including read-write delay, throughput and error rate are recorded, and the original performance indicator data is generated. The storage cost related data is collected, the capacity utilization of each storage layer is counted, and the resource consumption details are generated.

[0032] The original performance indicator data and the resource consumption details are aggregated and analyzed, the performance average value, peak value and trend change are counted according to the time dimension, the abnormal fluctuation and performance bottleneck are identified, and the feedback data is obtained.

[0033] As a preferred scheme of the unstructured storage hierarchical strategy optimization method based on machine learning, the method comprises the following steps of:

[0034] The feedback data is deeply analyzed, the correlation mode of the performance indicators and the cost data is extracted, the performance characteristics of the storage hierarchy decision under different access modes are identified, and the strategy performance evaluation report is generated.

[0035] According to the strategy performance evaluation report and the new access mode characteristics in the feedback data, the multi-dimensional decision tree model is incrementally trained, and the multi-dimensional decision tree model parameters are updated

[0036] The application has the beneficial effects that: by monitoring file access events in real time and extracting a multi-dimensional feature data set containing basic attributes, access behaviors and context information, efficient response of hot data and reasonable use of resources are ensured, the overall performance utilization rate and cost-effectiveness of storage are significantly improved, and an intelligent storage hierarchical management system with adaptive evolution capability is formed. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Fig. 1 Flowchart of the method for optimizing the unstructured storage hierarchical strategy based on machine learning.

[0039] Fig. 2 Flowchart of generating a multi-dimensional feature data set.

[0040] Fig. 3 Flowchart of training a multi-dimensional decision tree model.

[0041] Fig. 4 Flowchart of storage level allocation and adjustment. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings.

[0043] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0044] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0045] REFERENCE Figs. 1-4 For one embodiment of the present application, the embodiment provides a method for optimizing an unstructured storage hierarchical strategy based on machine learning, comprising the following steps:

[0046] S1. Monitor file access events in real time, generate access logs, and extract multi-dimensional feature datasets.

[0047] S1.1. Access logs include file identifier, access timestamp, operation type, file size, user identifier, and business information.

[0048] S1.2. Continuously monitor file access events, save access events as access logs, repeatedly record access logs and fill in missing values, detect and correct abnormal data, and generate preprocessed access logs.

[0049] Specifically, continuously monitor file access events, capture detailed information such as operation type, timestamp, user ID and file path for each file access through a real-time monitoring mechanism, and persist access events as structured access logs;

[0050] Duplicate records in the access log are removed using a hash comparison algorithm or a timestamp-based duplicate detection method. Missing values ​​are filled in using mean interpolation. Abnormal data is detected and corrected using a statistical anomaly detection algorithm (e.g., Z-score method with a threshold of 3 in this example) to generate a preprocessed access log.

[0051] S1.3. Extract file type and file size from the preprocessed access logs and use them as basic attribute features. Calculate the recent, medium-term, and long-term access frequencies using a time window statistical method and combine this with timestamp analysis to obtain access behavior characteristics.

[0052] Specifically, the fields of each access record are traversed from the preprocessed access log. String processing is used to extract the file extension from the file path field (e.g., extract "pdf" from the path "document.pdf") as the file type feature, and the value of the file size field in the log is directly read as the file capacity feature to generate basic attribute features.

[0053] Based on basic attribute features, a time window statistical method is adopted to set up recent time windows (e.g., 7 days for the example), medium-term time windows (e.g., 30 days for the example), and long-term time windows (e.g., 90 days for the example) to count the access frequency of each time period. Combined with timestamp analysis, the hourly distribution, weekday and holiday distribution patterns of access events are analyzed to obtain access time distribution features as access behavior features.

[0054] S1.4. By analyzing user access patterns, file usage scenarios, business domains, and permission levels in access behavior characteristics, the business relevance and data importance level are determined and used as context features. Through one-hot encoding, standardization processing, and numerical scoring, the basic attribute features, access behavior features, and context features are integrated and encoded to generate a multi-dimensional feature dataset.

[0055] Specifically, based on user access patterns (such as high-frequency access periods and access frequency fluctuations), file usage scenarios (such as collaborative editing, read-only access, and batch downloading), business domains (such as R&D departments and finance departments), and permission levels (such as administrators and ordinary users) contained in access behavior characteristics, a multi-dimensional evaluation matrix is ​​used to analyze the business relevance level and data importance level to form structured context features;

[0056] One-hot encoding is applied to the structured context features to generate binary vectors, and Z-score standardization is used to unify the dimensions. The feature values ​​are converted into numerical values ​​by weighting the contribution of feature variance (e.g., business relevance weight 0.6, data importance weight 0.4). The processed feature vectors are then concatenated and aligned with dimensions to generate a multi-dimensional feature dataset.

[0057] It should be noted that the multi-dimensional evaluation matrix is ​​a tool for transforming qualitative business rules into quantitative evaluation indicators. By defining multiple evaluation dimensions and quantification rules, business knowledge is embedded into the feature engineering process.

[0058] S2. Initialize the multidimensional decision tree model using the multidimensional feature dataset, and train the multidimensional decision tree model to obtain the trained multidimensional decision tree model.

[0059] S2.1. Perform statistical analysis on the multi-dimensional feature dataset, calculate the mean, variance, maximum, minimum and distribution of all features, generate statistical analysis results, and calculate the Pearson correlation coefficient and mutual information value between features to generate feature correlation analysis results.

[0060] Specifically, descriptive statistical analysis is performed on multi-dimensional feature datasets (revealing the basic distribution characteristics and inherent laws of the data through quantitative indicators and visualization methods), calculating the arithmetic mean, variance, maximum and minimum values ​​of each feature, and analyzing the distribution pattern and central tendency of feature values ​​through frequency distribution histograms to generate statistical analysis results;

[0061] Based on the statistical analysis results, the degree of linear correlation is calculated using the Pearson correlation coefficient, and the mutual information value is calculated to measure the nonlinear dependency, generating characteristic correlation analysis results.

[0062] It should be noted that the expressions for calculating the Pearson correlation coefficient and mutual information value between features are as follows:

[0063]

[0064] in, Representing variables and The strength and direction of the linear correlation between them (the direction is directly calculated) (The sign of the value is determined) Represents the total number of samples. Represents the sample index variable. Indicates the first Each sample in features (variables) The value on, Indicates the first Each sample has features (variables) The value on, Representing characteristic variables The sample mean, Representing characteristic variables The sample mean.

[0065] ;

[0066] in, Representing variables and The degree of information sharing between them Represents random variables A specific value (e.g., file size). Represents random variables A specific value (e.g., access frequency). Represent the joint probability (through) = and = (The frequency of occurrence is obtained) express = The probability of occurrence express = The probability of occurrence.

[0067] It should also be noted that, from the multi-dimensional feature dataset, a feature vector to be analyzed is selected, such as the file access frequency feature. This involves finding the minimum and maximum values ​​of the file access frequency feature among all samples, and dividing the entire value range into several continuous and equally wide intervals. For example, if the file access frequency range is 0 to 100 times / day, the interval width can be set to 10, resulting in 10 intervals: 0-10, 11-20, ..., 91-100. All values ​​of the file access frequency feature in the dataset are traversed, and the number of samples for each value falling into each interval is counted to obtain the frequency. For example, if 150 files are found to have an access frequency in the range of 0-10 times / day, the feature value is plotted on the horizontal axis, and the frequency is plotted on the vertical axis. Adjacent rectangular bars represent each interval and its corresponding frequency. The height of the rectangular bars represents the frequency of data occurrence within that interval, generating a frequency distribution histogram.

[0068] S2.2. Based on the statistical analysis results and feature correlation analysis results, set the maximum depth of the decision tree and the minimum number of leaf node samples, and select the optimal feature for splitting based on information gain to obtain the initialized multi-dimensional decision tree model parameters.

[0069] Specifically, based on the mean, variance, and distribution of each feature in the statistical analysis and feature correlation analysis results, as well as the Pearson correlation coefficient and mutual information value in the feature correlation analysis results, the feature dimension and correlation strength are evaluated. Feature dimension evaluation is achieved by counting the number of features and analyzing the distribution of each feature (such as the size of the variance). For example, features with large variance indicate high data volatility, requiring a deeper tree to capture complex patterns. Correlation strength is evaluated by combining the Pearson correlation coefficient and mutual information value. A Pearson correlation coefficient with an absolute value close to 1 indicates a strong linear correlation, and a mutual information value greater than 0.5 (based on standardized data) indicates a significant non-linear dependency. The higher the value, the greater the contribution of the storage level prediction. Combining the evaluation results of feature dimension and correlation strength, the maximum depth of the decision tree is set (e.g., 10 in the example) to match the feature complexity, and the minimum number of leaf node samples is set (e.g., in the example). Value 5) controls overfitting based on data size. For example, if 1000 file samples are used to train the decision tree, without controlling overfitting (minimum leaf node sample count = 1), the decision tree may generate an extreme rule: "Store in high-speed storage only when the file name is '2024Q1 report.pdf' and the size is exactly 15.2MB". This perfectly matches a single training sample, but cannot handle new files with slightly different file names or a size of 15.3MB, leading to generalization failure. Conversely, if the minimum leaf node sample count is set to 5, the decision tree is forced to find commonalities from at least 5 samples, forming a generalization rule: "Store in high-speed storage when the file type is PDF, the size is between 10-20MB, and the average daily access is > 5 times". The information entropy reduction before and after splitting each feature is calculated using information gain, and the feature with the largest information gain is selected as the optimal split point to generate the initialized multi-dimensional decision tree model parameters.

[0070] It should be noted that when conducting the evaluation of feature dimensions and correlation strength analysis, all feature values ​​in the multi-dimensional feature dataset are standardized to eliminate biases caused by different units and numerical ranges, so that the value distribution of each feature is on a comparable scale.

[0071] S2.3. The multi-dimensional feature dataset is divided into training and test sets using a stratified sampling method, and feature normalization is performed on the training and test sets.

[0072] Specifically, a stratified sampling method is used to divide the data based on key features (such as business domain or permission level) in the multi-dimensional feature dataset (e.g., 70% of the training set examples and 30% of the test set examples) and randomly sample to generate training and test sets. Feature normalization is performed on the training and test sets respectively, and the minimum-maximum normalization method is used to scale the numerical features to the [0,1] interval.

[0073] S2.4. The multi-dimensional decision tree model is trained using the training set. A recursive partitioning algorithm is used to construct the decision tree structure. At each node, the optimal splitting feature is selected based on the information gain value to generate a preliminary trained multi-dimensional decision tree model.

[0074] Specifically, the multi-dimensional decision tree model is trained using the training set. Feature data and labels are extracted from the training set as input data and target values. A recursive partitioning algorithm is used to construct the decision tree structure. The decision tree is constructed by iteratively splitting the dataset. The complete training set is regarded as the root node. At each node, the information gain of all features is calculated and the feature with the largest gain and the corresponding threshold (such as "access frequency > 5 times / day" as the split point) is selected. The data of the current node is divided into two subsets. The splitting operation is recursively repeated for each subset until the minimum number of samples in the leaf node and the maximum depth of the tree are met, thus generating a preliminary trained multi-dimensional decision tree model.

[0075] S2.5. Use the test set to evaluate the performance of the pre-trained multi-dimensional decision tree model after pruning and optimization, calculate the accuracy, recall, F1 score and confusion matrix, generate a performance evaluation report, and optimize and adjust the decision tree parameters to obtain the trained multi-dimensional decision tree model.

[0076] Specifically, the performance of the pre-trained multi-dimensional decision tree model after pruning and optimization is evaluated using the test set. The performance metrics of the multi-dimensional decision tree model are calculated, including accuracy, recall, F1 score and confusion matrix. A performance evaluation report is generated, and the decision tree parameters are optimized and adjusted, such as adjusting the maximum depth, minimum number of leaf node samples or split criteria, to obtain the trained multi-dimensional decision tree model.

[0077] It should be noted that the performance metrics of multi-dimensional decision tree models are used to measure the classification ability and performance of multi-dimensional decision tree models. Accuracy reflects the proportion of correct predictions made by the model, recall measures the proportion of correct identifications in the actual positive class, F1 score is the harmonic mean of accuracy and recall, and confusion matrix provides detailed classification results, including the number of true positives, false positives, true negatives, and false negatives.

[0078] S3. Using the trained multi-dimensional decision tree model, allocate the corresponding storage level to the stored object to obtain the storage level decision.

[0079] S3.1. Receive the storage object to be allocated, extract the real-time feature data of the storage object, and generate the feature vector of the current storage object.

[0080] Specifically, the system receives the storage object to be allocated, reads the metadata of the storage object through data parsing, and extracts feature data such as file size, file type, last access time, and permission level in real time. The extracted numerical features are scaled to the [0,1] interval using the min-max normalization method, and the categorical features are converted into binary vectors using one-hot encoding. All processed feature values ​​are concatenated to generate the feature vector of the current storage object.

[0081] S3.2. Input the feature vector of the currently stored object into the trained multi-dimensional decision tree model for prediction and analysis, and output the preliminary storage level recommendation results.

[0082] Specifically, the feature vector of the current stored object is input into the trained multi-dimensional decision tree model for predictive analysis. The trained multi-dimensional decision tree model recursively compares the relationship between each feature value in the feature vector and the split threshold of the decision tree node, starting from the root node. Based on the comparison result, it selects the left or right subtree path to traverse downwards until it reaches the leaf node. For example, if the "7-day access frequency" value in the feature vector is 8 times, and the split threshold of the root node is 5 times, the right subtree path is selected because 8>5. At the right subtree node, the "business importance score" value of 75 points is compared with the threshold of 80 points. Since 75≤80, the left subtree path is selected. The model reaches the leaf node and outputs the corresponding storage level label (such as the standard layer) as the preliminary storage level recommendation result.

[0083] It should be noted that during the training process, each feature (e.g., file size) of the dataset at the current node is sorted, and the midpoint between two adjacent values ​​is used as a candidate split threshold. For each candidate split threshold, the dataset at the current node is divided into two subsets, and the information gain resulting from the split is calculated. Among all candidate split thresholds for all candidate features, the feature that produces the maximum information gain and its corresponding threshold are selected as the decision tree node split threshold.

[0084] S3.3. Collect the current capacity utilization, performance load and network bandwidth status of each storage layer, and make adaptive adjustments to the preliminary storage layer recommendation results to generate storage layer decisions.

[0085] S3.3.1 Collects the current capacity utilization, performance load, and network bandwidth status of each storage layer, and reads the capacity usage percentage, computing resource load, and network transmission bandwidth utilization of each storage layer in real time through the storage management interface.

[0086] Specifically, the system connects to each storage layer in real time through the storage management interface, actively queries and collects the capacity usage percentage to quantify capacity utilization, synchronously obtains processor load indicators and input / output load indicators to assess performance load, and reads network transmission bandwidth utilization to monitor network bandwidth status, thereby obtaining real-time resource status information for each storage layer.

[0087] S3.3.2 compares and analyzes the current capacity utilization, performance load, and network bandwidth status with the preliminary storage tier recommendation results, dynamically adjusts the preliminary storage tier recommendation results according to the adaptive rules, re-evaluates the priority and applicability of each storage tier, and generates a storage tier decision.

[0088] Specifically, the system collects real-time data on capacity utilization percentage, performance load metrics (such as processor utilization and I / O operation counts), and network bandwidth utilization for each storage layer through the storage management interface. This real-time data is then compared with resource status thresholds. When the resource metrics of the storage layer corresponding to the initial storage layer recommendation result exceed the resource status threshold, the recommendation result is dynamically adjusted according to adaptive rules. Priority and applicability are reassessed based on the resource sufficiency ranking of each storage layer. Resource sufficiency includes three main dimensions: capacity utilization, performance load metrics, and network bandwidth utilization. The system then evaluates the capacity utilization, performance load metrics, and network bandwidth utilization for each storage layer. The system uses a normalization process to unify indicators of different dimensions into the same numerical range (e.g., values ​​from 0 to 1 in the example). Weight parameters are assigned to each indicator, such as a weight of 0.4 for capacity utilization, 0.4 for performance load, and 0.2 for network bandwidth. The overall resource utilization rate is calculated using a weighted average. Based on the overall resource utilization rate, each storage layer is sorted in ascending order; storage layers with lower values ​​represent more abundant resources. When the resource indicators of the storage layers corresponding to the initial storage layer recommendations exceed the threshold, the next best storage layer with the lowest overall resource utilization rate is selected according to the sorting results and used as the new target storage layer, thus generating a storage layer decision.

[0089] It should be noted that historical data such as capacity utilization, performance load indicators, and network bandwidth utilization are collected, and statistical analysis methods are used to calculate the key quantiles of each indicator (e.g., the historical 85th percentile of capacity utilization) to identify the critical point of performance degradation. At the same time, based on the service level agreement's requirements for response time and availability, business constraints are transformed into quantitative thresholds. By combining the statistical results of historical data and business demand indicators, resource status thresholds are obtained (e.g., an example value of 85% for capacity utilization and 90% for performance load).

[0090] By analyzing the correlation between capacity utilization, performance load, and response time, and transforming the business performance requirements in the service level agreement into quantitative indicators, optimization strategies are formulated based on resource cost-benefit analysis to obtain adaptive rules.

[0091] S4. Based on the storage tier decision, allocate the storage objects to different storage tiers and perform resource configuration and storage operations.

[0092] S4.1. Parse the target storage layer identifier field in the storage layer decision to obtain the target storage layer identifier, and prepare the corresponding storage resource configuration parameters according to the characteristics of the target storage layer.

[0093] Specifically, the storage tier decision content is parsed, and the target storage tier identifier is obtained by reading the target storage tier identifier field stored in the storage tier decision data structure. Based on the characteristics of the target storage tier, including storage type (e.g., high-speed storage tier or cold storage tier), performance indicators (e.g., input / output throughput requirements) and capacity limits, the corresponding storage resource configuration parameters, including access control permissions, data redundancy policies, storage path addresses and encryption settings, are queried from the storage tier configuration template to generate complete storage resource configuration parameters.

[0094] It should be noted that the performance metrics (such as IOPS, throughput, and latency) and cost data (such as unit storage cost, energy consumption cost, and maintenance cost) of different types of storage media are normalized to construct performance objective functions and cost objective functions. A weighted linear method is used to jointly optimize performance maximization and cost minimization as dual objectives, and multiple sets of performance and cost balance solutions are obtained through iteration. Based on the service level agreement requirements of the business scenario (such as response time and availability level), the solution with the lowest cost that meets the performance SLA constraints is selected from the balance solution set to generate a storage layer configuration template.

[0095] S4.2. Initialize the connection session of the target storage layer according to the storage resource configuration parameters, establish a secure communication link, verify access permissions and resource availability; and transfer the storage object from the source storage layer to the target storage layer through a block transfer mechanism.

[0096] Specifically, based on the connection protocol parameters, authentication credential parameters, and resource quota parameters corresponding to the target storage layer identifier in the storage layer decision, the integrity and validity of the parameter format are verified. Using the verified parameters, a transport layer security protocol encrypted channel is established with the target storage. In the encrypted channel, the remaining capacity and input / output load status of the target storage layer are queried to confirm resource availability. After successful verification, the storage object is divided into fixed-size data blocks (e.g., 4MB in this example), a checksum is generated for each data block, and the data blocks are sequentially transmitted from the source storage layer to the target storage layer through multi-threaded parallel transmission.

[0097] S5. Monitor the access status of stored objects in the new storage tier in real time, collect file access performance, storage cost and response time, and obtain feedback data.

[0098] S5.1. Continuously collect access data of stored objects in the new target storage layer, record performance indicators including read / write latency, throughput, and error rate, and generate raw performance indicator data; collect storage cost-related data, calculate the capacity utilization of each storage layer, and generate resource consumption details.

[0099] Specifically, monitoring programs deployed on storage nodes continuously capture read and write operations of stored objects in the new storage tier, recording in real time the response time (the time interval from receiving the request to completing the operation), data transfer rate (the amount of data successfully transferred per unit time), and operation status (success or error type) of each operation, forming raw performance indicator data containing timestamps, operation types, and performance values; at the same time, the storage management interface periodically collects the capacity utilization rate (the ratio of occupied storage space to total capacity) and resource consumption measurement (including storage space costs, IO operation costs, and network transmission costs) of each storage tier, and calculates the cost details over time in conjunction with the storage service pricing strategy, generating resource consumption details categorized by storage tier.

[0100] S5.2. Perform aggregated analysis on the original performance index data and resource consumption details, statistically analyze the average performance value, peak value and trend changes by time dimension, identify abnormal fluctuations and performance bottlenecks, and obtain feedback data.

[0101] Specifically, the aggregation analysis of the original performance index data and resource consumption details is as follows: A sliding time window method is used to statistically calculate the read / write latency, throughput, and error rate in the original performance index data at different time granularities (e.g., minute, hour, day), obtaining the average performance value, maximum value (peak value), and trend change rate based on linear regression within each time window. Simultaneously, the capacity utilization rate and cost data in the resource consumption details are aggregated along the same time dimension, calculating the cost growth rate and capacity change rate per unit time. Combined with historical baseline data, the Z-score algorithm is used to detect abnormal fluctuations in performance indicators. Performance bottlenecks are identified using the percentile method (e.g., the 95th percentile in the example). The performance statistics, cost analysis results, abnormal fluctuation markers, and performance bottleneck information are integrated into structured feedback data.

[0102] S6. Based on feedback data, optimize the multi-dimensional decision tree model through an adaptive adjustment mechanism.

[0103] S6.1. Conduct in-depth analysis of the feedback data, extract the correlation patterns between performance indicators and cost data, identify the performance characteristics of storage tier decisions under different access modes, and generate a strategy performance evaluation report.

[0104] Specifically, when conducting in-depth analysis of feedback data, the Pearson correlation coefficient is used to calculate the degree of linear correlation between performance indicators (including read / write latency, throughput, and error rate) and cost data (including storage space costs and I / O operation costs). At the same time, time series alignment analysis is used to extract the performance cost variation patterns corresponding to storage tier decisions under different access patterns (e.g., high-frequency small file access and low-frequency large file transfer). Clustering algorithms are used to identify the performance characteristics of storage tier decisions in various access patterns (e.g., the cost-effectiveness ratio of high-speed storage tiers handling high-frequency access) to determine the effectiveness of the decisions. The results of correlation pattern analysis, access pattern feature identification, and decision effectiveness judgment are integrated to generate a strategy effectiveness evaluation report.

[0105] S6.2. Based on the strategy effectiveness evaluation report and the new access pattern characteristics in the feedback data, incrementally train the multi-dimensional decision tree model and update the parameters of the multi-dimensional decision tree model.

[0106] Specifically, based on the quantitative scores and optimization suggestions for storage-level decision performance in the strategy performance evaluation report, data samples corresponding to new access pattern features (such as high-frequency small file access and low-frequency large file transfer) are extracted from the feedback data as incremental training datasets. The incremental training datasets are combined with the existing parameters of the multi-dimensional decision tree model, and an online learning algorithm is used to dynamically adjust the decision tree node splitting threshold (e.g., 0.01 for the example) and the leaf node storage-level label distribution by calculating the information gain value of the new samples. This updates the splitting rules and prediction weights of the multi-dimensional decision tree model, thus completing the optimization and adjustment of the multi-dimensional decision tree model parameters.

[0107] In summary, this invention ensures efficient response to hot data and rational resource utilization by: real-time monitoring of file access events and extracting multi-dimensional feature datasets containing basic attributes, access behaviors, and contextual information; and by continuously correcting model prediction biases through a closed-loop feedback mechanism, significantly improving the overall performance utilization and cost-effectiveness of storage, thus forming an intelligent storage hierarchical management system with adaptive evolution capabilities.

[0108] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A machine learning-based optimization method for unstructured storage hierarchical strategies, characterized in that: include, Real-time monitoring of file access events, generation of access logs, and extraction of multi-dimensional feature datasets; A multidimensional decision tree model is initialized using a multidimensional feature dataset and then trained to obtain the trained multidimensional decision tree model. Using the trained multi-dimensional decision tree model, the corresponding storage level is assigned to the stored object to obtain the storage level decision; The process of using a trained multi-dimensional decision tree model to allocate corresponding storage levels to stored objects and obtain storage level decisions involves the following steps: Receive the storage object to be allocated, extract the real-time feature data of the storage object, and generate the feature vector of the current storage object; The feature vector of the currently stored object is input into the trained multi-dimensional decision tree model for prediction and analysis, and the preliminary storage level recommendation results are output. Collect the current capacity utilization, performance load, and network bandwidth status of each storage tier, and make adaptive adjustments to the initial storage tier recommendation results to generate storage tier decisions; The process involves collecting the current capacity utilization, performance load, and network bandwidth status of each storage tier, and then adaptively adjusting the initial storage tier recommendations to generate a storage tier decision. The specific steps are as follows: Collect the current capacity utilization, performance load, and network bandwidth status of each storage layer, and read the capacity usage percentage, computing resource load, and network transmission bandwidth utilization of each storage layer in real time through the storage management interface; The current capacity utilization, performance load, and network bandwidth status are compared and analyzed with the preliminary storage tier recommendation results. The preliminary storage tier recommendation results are dynamically adjusted according to adaptive rules, the priority and applicability of each storage tier are reassessed, and a storage tier decision is generated. Based on storage tier decisions, storage objects are allocated to different storage tiers, and resource configuration and storage operations are performed. Real-time monitoring of access to stored objects in the new storage tier, collection of file access performance, storage costs and response times, and obtaining feedback data; Based on feedback data, the multi-dimensional decision tree model is optimized through an adaptive adjustment mechanism. The specific steps are as follows. Deeply analyze the feedback data, extract the correlation patterns between performance indicators and cost data, identify the performance characteristics of storage tier decisions under different access modes, and generate a strategy performance evaluation report. Based on the strategy effectiveness evaluation report and the new access pattern characteristics in the feedback data, the multi-dimensional decision tree model is incrementally trained and its parameters are updated.

2. The machine learning-based unstructured storage tiering strategy optimization method as described in claim 1, characterized in that: The access log includes file identifier, access timestamp, operation type, file size, user identifier, and business information.

3. The machine learning-based unstructured storage tiering strategy optimization method as described in claim 1, characterized in that: The real-time monitoring of file access events, generation of access logs, and extraction of multi-dimensional feature datasets are carried out through the following steps. Continuously monitor file access events, save access events as access logs, repeatedly record access logs and fill in missing values, detect and correct abnormal data, and generate preprocessed access logs. File type and file size are extracted from the preprocessed access logs and used as basic attribute features. The recent, medium and long-term access frequencies are calculated using the time window statistical method and combined with timestamp analysis to obtain access behavior characteristics. By analyzing user access patterns, file usage scenarios, business domains, and permission levels in access behavior characteristics, the business relevance and data importance level are determined and used as contextual features; By integrating and encoding basic attribute features, access behavior features, and contextual features through one-hot encoding, standardization, and numerical scoring, a multi-dimensional feature dataset is generated.

4. The machine learning-based unstructured storage tiering strategy optimization method as described in claim 1, characterized in that: The specific steps for initializing the multi-dimensional decision tree model using a multi-dimensional feature dataset are as follows. Perform statistical analysis on multi-dimensional feature datasets, calculate the mean, variance, maximum, minimum and distribution of all features, generate statistical analysis results, and calculate the Pearson correlation coefficient and mutual information value between features to generate feature correlation analysis results; Based on the statistical analysis results and feature correlation analysis results, the maximum depth and minimum number of leaf node samples of the decision tree are set. The optimal feature is selected based on information gain and splitting is performed to obtain the parameters of the initialized multidimensional decision tree model.

5. The machine learning-based unstructured storage tiering strategy optimization method as described in claim 1, characterized in that: The specific steps for training the multi-dimensional decision tree model to obtain the trained multi-dimensional decision tree model are as follows. A stratified sampling method is used to divide the multi-dimensional feature dataset into a training set and a test set, and feature normalization is performed on the training set and the test set. The multi-dimensional decision tree model is trained using the training set. A recursive partitioning algorithm is used to construct the decision tree structure. At each node, the optimal splitting feature is selected based on the information gain value to generate a preliminary trained multi-dimensional decision tree model. The performance of the pre-trained multi-dimensional decision tree model after pruning and optimization is evaluated using the test set. Accuracy, recall, F1 score and confusion matrix are calculated, a performance evaluation report is generated, and the decision tree parameters are optimized and adjusted to obtain the trained multi-dimensional decision tree model.

6. The machine learning-based unstructured storage tiering strategy optimization method as described in claim 1, characterized in that: The process of allocating storage objects to different storage layers based on storage hierarchy decisions, and performing resource configuration and storage operations, involves the following specific steps: Parse the target storage layer identifier field in the storage layer decision to obtain the target storage layer identifier, and prepare the corresponding storage resource configuration parameters according to the characteristics of the target storage layer; Initialize the connection session of the target storage layer according to the storage resource configuration parameters, establish a secure communication link, and verify access permissions and resource availability; The storage object is transferred from the source storage layer to the target storage layer through a block transfer mechanism.

7. The machine learning-based unstructured storage tiering strategy optimization method as described in claim 1, characterized in that: The process of real-time monitoring of access to stored objects in the new storage tier, collecting data on file access performance, storage costs, and response times to obtain feedback data, involves the following steps: Continuously collect access data of stored objects in the new target storage tier, record performance indicators including read / write latency, throughput, and error rate, and generate raw performance indicator data; collect storage cost-related data, calculate the capacity utilization of each storage tier, and generate resource consumption details. Aggregate and analyze the raw performance index data and resource consumption details, statistically analyze the average performance, peak performance and trend changes over time, identify abnormal fluctuations and performance bottlenecks, and obtain feedback data.

Images