Distributed big data intelligent storage management method based on AI

Abstract

The invention discloses an AI-based distributed big data intelligent storage management method, and relates to the technical field of storage management. The method comprises the steps of generating an initial feature matrix by obtaining observation data of all storage devices in a distributed storage cluster, and inputting the initial feature matrix into a pre-training life prediction model to obtain residual life; equipment with the service life lower than a first safety threshold value is judged as high-risk equipment, and data of the high-risk equipment is included in a to-be-migrated source data set; the devices with the service life higher than a second safety threshold value are included into a to-be-migrated target device set; and generating a migration plan according to the source and target data sets and executing migration. By predicting the residual life of the equipment, advanced identification and active data migration of the high-risk equipment are realized, the data loss risk is reduced, and the utilization rate of the health equipment and the overall cluster storage resource efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of storage management technology, and specifically to an AI-based distributed big data intelligent storage management method. Background Technology

[0002] In the context of the rapid development of big data and cloud computing, distributed storage systems have become a core infrastructure for enterprises and cloud platforms. Existing distributed storage technologies generally employ passive fault-tolerance mechanisms such as multiple replicas, snapshots, and checksums. These mechanisms improve reliability by redundantly storing data across multiple nodes, maintaining data availability and integrity even when a single or small number of storage devices fail. However, these traditional fault-tolerance technologies are essentially reactive. Once the number of hard drive failures exceeds the redundancy limit—for example, three replicas cannot handle the simultaneous failure of more than three hard drives—the risk of data loss and business interruption remains extremely high.

[0003] While existing technologies offer monitoring and alarm functions for equipment operation and maintenance, they primarily rely on SMART metrics and fixed thresholds, resulting in a coarse health status assessment that fails to accurately reflect the true risk level of equipment and cannot proactively identify potential failures. Regarding data migration, while existing systems support migration operations during node expansion, hardware shutdown, or load balancing, migration strategies are mostly driven by manual intervention or static rules, lacking dynamic scheduling capabilities linked to equipment lifespan prediction. This often leads to delayed migration timing, inaccurate target selection, and underutilization of healthy equipment resources. These shortcomings are particularly pronounced in environments with expanded cluster sizes and diversified storage media, making traditional distributed storage systems significantly inadequate in ensuring data reliability when multiple devices fail simultaneously, and failing to fundamentally reduce the probability of data loss. Summary of the Invention

[0004] The purpose of this invention is to address the problems mentioned in the background art, such as the reliance on passive fault tolerance mechanisms and lack of dynamic scheduling in traditional distributed storage systems, which lead to high risk of data loss and low utilization of storage resources. In response, this invention proposes an AI-based distributed big data intelligent storage management method.

[0005] A first aspect of this invention provides an AI-based distributed big data intelligent storage management method, the method comprising:

[0006] Obtain observation data from all storage devices in the distributed storage cluster;

[0007] The observation data of the target storage device are preprocessed to obtain an initial feature matrix; the target storage device can be any storage device.

[0008] The initial feature matrix is ​​input into a pre-trained lifetime prediction model to obtain the remaining lifetime of the target storage device.

[0009] Storage devices with remaining lifetime less than the first safety threshold are classified as high-risk devices, and the data stored on them is classified as data to be migrated and included in the source dataset to be migrated.

[0010] Storage devices with remaining lifetime greater than the second security threshold are classified as healthy devices and included in the set of target devices to be migrated; the second security threshold is greater than the first security threshold.

[0011] Based on the source dataset to be migrated and the target device set to be migrated, a migration plan is generated and the data migration is executed.

[0012] Optionally, the observation data includes SMART attribute data.

[0013] Optionally, the preprocessing of the observation data of the target storage device to obtain the initial feature matrix includes:

[0014] Observational data of a preset duration are extracted and converted into a time series as the first target data;

[0015] Outlier removal and missing value imputation are performed on the first target data to obtain the second target data;

[0016] The second target data is normalized to obtain an initial feature matrix; each row of the initial feature matrix represents the time feature of an attribute; each column represents the set of attribute features at a time step.

[0017] Optionally, the lifetime prediction model includes multiple cascaded feature extraction modules and a prediction module; wherein:

[0018] Multiple cascaded feature extraction modules are used to extract features from the input data using one-dimensional convolution and graph convolution to obtain high-level features;

[0019] The prediction module is used to map the high-level features to remaining lifetime values ​​using a fully connected network.

[0020] Optionally, multiple feature extraction modules have the same structure, and any one feature extraction module includes a multi-scale convolution module, a graph edge weighting module, and a graph convolution module; wherein:

[0021] The multi-scale convolution module is used to perform one-dimensional convolution operations on the temporal features of each attribute in the input using multiple convolution kernels of different scales, and to enhance the convolution results using an attention mechanism to obtain the first global feature.

[0022] The graph edge determination module is used to calculate the association strength of each pair of attributes in each time step using a neural network, so as to obtain the adjacency matrix of each time step.

[0023] The graph convolution module is used to aggregate the information of the attribute features of each time step according to the correlation strength of each pair of attributes, and obtain the deep spatial features of that time step by using graph convolution for each time step.

[0024] Optionally, the operation process of the multi-scale convolution module includes:

[0025] Let any one of the attributes be designated as the target attribute; let the temporal characteristics of the target attribute in the input be designated as the first temporal characteristic;

[0026] Multiple parallel convolutional paths are used to perform convolution operations on the first temporal features, and the convolution results of each convolutional path are concatenated according to the channel dimension to obtain the second temporal features of the target attribute; each convolutional path uses a one-dimensional convolutional kernel of a different size;

[0027] The second temporal feature of the target attribute is subjected to global average pooling, and the pooling result is mapped to channel weights through a fully connected layer;

[0028] The second temporal feature of the target attribute is weighted using the channel weights to obtain the third temporal feature of the target attribute;

[0029] The third temporal features of all attributes are integrated to obtain the first global feature, which is then used as the output of the multi-scale convolution module.

[0030] Optionally, the lifetime prediction model includes three cascaded feature extraction modules.

[0031] Optionally, the calculation process of the graph edge weighting module includes:

[0032] Designate any one of the input channels as the target channel;

[0033] A one-layer long short-term memory network is used to extract time-varying information from the features of the target channel to obtain the hidden features of the target channel;

[0034] The hidden features of all channels are integrated to obtain the second global feature;

[0035] Let any time step be denoted as the target time step; let the set of attribute features of the target time step described in the second global feature be denoted as the target attribute feature set;

[0036] For the target attribute feature set, features of two attributes are taken in a preset order, the features of the two attributes are concatenated, and then the correlation strength between the two attributes is output through two fully connected layers.

[0037] The association strength between two attributes is used as the weight of the edge, and the adjacency matrix of the target time step is constructed based on the association strength between all attributes.

[0038] Optionally, the computation process of the graph convolution module includes:

[0039] The first global feature is divided into time steps to obtain the node feature matrix for each time step;

[0040] Record any time step as the target time step;

[0041] Based on the node feature matrix and adjacency matrix of the target time step, multiple graph convolutional layers are used to perform information aggregation and feature transformation to obtain multiple graph information feature matrices;

[0042] The node feature matrix and multiple graph information feature matrices of the target time step are weighted and fused to obtain the deep spatial features of the target time step.

[0043] The deep spatial features of all time steps are concatenated along the time dimension to obtain the third global feature, which is then used as the input to the next-level module.

[0044] Optionally, the loss function of the lifespan prediction model during training is:

[0045] ;

[0046] Where y is the actual remaining lifespan. It is the predicted remaining useful life; and It is the penalty coefficient, and Less than .

[0047] The beneficial effects of this invention are:

[0048] By processing observational data from each storage device in a distributed storage cluster and combining it with a pre-trained lifetime prediction model, the remaining lifetime of the devices can be accurately assessed, enabling early identification of high-risk devices. Data migration plans generated based on the prediction results can proactively migrate risky data, reducing the probability of data loss at the source, while fully utilizing healthy devices to improve the overall storage resource utilization of the cluster. Attached Figure Description

[0049] Figure 1 A flowchart illustrating an AI-based distributed big data intelligent storage management method provided in an embodiment of the present invention;

[0050] Figure 2 A network architecture diagram of a lifetime prediction model provided in an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the structure of a feature extraction module provided in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the structure of a multi-scale convolution module provided in an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of the structure of a graph edge rights confirmation module provided in an embodiment of the present invention. Detailed Implementation

[0054] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0055] This invention provides an AI-based distributed big data intelligent storage management method. See also... Figure 1 , Figure 1 A flowchart illustrating an AI-based distributed big data intelligent storage management method provided in this embodiment of the invention. The method includes the following steps:

[0056] S101, Obtain observation data of all storage devices in the distributed storage cluster;

[0057] S102, Preprocess the observation data of the target storage device to obtain the initial feature matrix;

[0058] S103, Input the initial feature matrix into the pre-trained lifetime prediction model to obtain the remaining lifetime of the target storage device;

[0059] S104, Storage devices with remaining lifetime less than the first safety threshold are classified as high-risk devices, and the data stored on them are classified as data to be migrated and included in the source dataset to be migrated.

[0060] S105, Storage devices with remaining lifetime greater than the second safety threshold are classified as healthy devices and included in the set of target devices to be migrated;

[0061] S106. Generate a migration plan based on the source dataset to be migrated and the target device set to be migrated, and execute the data migration.

[0062] The target storage device is any storage device; the second security threshold is greater than the first security threshold.

[0063] This invention provides an AI-based distributed big data intelligent storage management method. By processing observation data from each storage device in a distributed storage cluster and combining it with a pre-trained lifetime prediction model, the remaining lifetime of the devices can be accurately assessed, enabling early identification of high-risk devices. The data migration plan generated based on the prediction results can proactively migrate risky data, reducing the probability of data loss at the source, while fully utilizing healthy devices to improve the overall storage resource utilization of the cluster.

[0064] In one implementation, the observation data includes SMART attribute data, including underlying data read error rate, start / stop count, remapped sector count, seek error rate, power-on count, write error rate, uncorrected sector count, power-off shrinkage count, and number of sectors that cannot be corrected offline.

[0065] In one implementation, step S102, preprocessing the observation data of the target storage device to obtain an initial feature matrix includes:

[0066] Step one: Extract observation data for a preset duration and convert it into a time series as the first target data. Specifically, the sampling period can be daily, extracting observation data from the first 42 days to form a time series.

[0067] Step two: Remove outliers and fill in missing values ​​in the first target data to obtain the second target data.

[0068] Step 3: Normalize the second target data to obtain the initial feature matrix. In the initial feature matrix, each row represents the time change sequence of an attribute, denoted as the time feature of that attribute; each column represents the set of attribute values ​​within a time step, denoted as the attribute feature set of that time step.

[0069] Preset time series sequences can clearly express the health trend of storage devices over time, which is beneficial for lifetime prediction models to capture patterns. Normalization processing unifies the dimensions of different attributes, making model training converge faster, more stable, and reducing bias caused by some feature values ​​being too large or too small.

[0070] In one implementation, the first security threshold can be set to 7 days, and the second security threshold can be set to 60 days. These dual thresholds ensure that data is always stored on devices with sufficient lifespan, reducing the risk of data loss after migration.

[0071] In one implementation, source-target matching is first performed based on storage space size, i.e., determining which target device each piece of data should be migrated to. Then, for high-risk devices on the same data storage node, they are sorted by their remaining lifespan from smallest to largest, and migrated sequentially when the system load is relatively light. Migrating is performed during periods of low load to avoid performance impact on running services and optimize overall cluster resource utilization efficiency.

[0072] In one embodiment, see Figure 2 , Figure 2 This is a network architecture diagram of a lifetime prediction model provided in an embodiment of the present invention. The lifetime prediction model includes three cascaded feature extraction modules and one prediction module; wherein:

[0073] Multiple cascaded feature extraction modules are used to extract features from the input data using one-dimensional convolution and graph convolution to obtain high-level features. The input to the first feature extraction module is the initial feature matrix M ( The input to the subsequent feature extraction module is the output F3 of the previous feature extraction module. R represents the set of real numbers, N represents the number of attributes, T is the number of time steps, and C is the number of channels.

[0074] The prediction module is used to map high-level features to remaining lifetime values ​​using a fully connected network.

[0075] In one implementation, multiple feature extraction modules have the same structure. See [link / reference] Figure 3 , Figure 3 This is a schematic diagram of a feature extraction module provided in an embodiment of the present invention. Any feature extraction module includes a multi-scale convolution module, a graph edge weighting module, and a graph convolution module; wherein:

[0076] Multi-scale convolution module: This module uses multiple convolution kernels of different scales to perform one-dimensional convolution operations on the temporal features of each attribute in the input, and employs an attention mechanism to enhance the convolution results, obtaining the first global feature. For details, see [link to details]. Figure 4 , Figure 4 This is a schematic diagram of a multi-scale convolution module provided in an embodiment of the present invention. The operation process of the multi-scale convolution module includes:

[0077] Let any one of the attributes be denoted as target attribute i; let the temporal features of target attribute i in the input be denoted as the first temporal feature. ( );

[0078] Multiple parallel convolutional paths are used to perform convolution operations on the first temporal features, and the convolution results of each convolutional path are concatenated according to the channel dimension to obtain the second temporal features of the target attribute. ( Each convolutional path uses a one-dimensional convolutional kernel of a different size. Specifically, there are four convolutional paths with kernel sizes of 2×1, 3×1, 6×1, and 7×1, all using the ReLU activation function.

[0079] The second temporal feature of the target attribute is subjected to global average pooling, and the pooling result is mapped to channel weights through a fully connected layer.

[0080] The second time feature of the target attribute is weighted using channel weights to obtain the third time feature of the target attribute. ( ).

[0081] Integrating the third temporal features of all attributes yields the first global feature F1. The first global feature is used as the output of the multi-scale convolution module. At this time, the number of channels C of the output feature is 4.

[0082] This multi-scale convolution module uses different convolution kernels to perform parallel convolutions on time series of the same attribute, which can simultaneously capture short-term fluctuations and long-term trends. Through channel weighting, the model automatically focuses on the features most important for remaining lifetime prediction, highlights key patterns, suppresses irrelevant or noisy information, enhances the model's ability to perceive complex time series patterns, learns the high-value degradation features contained in each attribute, and improves the accuracy of remaining lifetime prediction.

[0083] The graph-edge association module uses a neural network to calculate the association strength of each pair of attributes at each time step, obtaining the adjacency matrix for each time step. For details, see [link to module]. Figure 5 , Figure 5 This is a schematic diagram of a graph edge rights determination module provided in an embodiment of the present invention. The calculation process of the graph edge rights determination module includes:

[0084] Let any one of the input channels be the target channel k;

[0085] Features of the target channel are obtained using a single-layer Long Short-Term Memory (LSTM) network. ( Time-varying information is extracted to obtain the hidden features of the target channel. ( );

[0086] By integrating the hidden features of all channels, the second global feature F2 is obtained. );

[0087] Let any time step be denoted as the target time step t; let the set of attribute features of the target time step in the second global feature be denoted as the target attribute feature set z. t ( ) ;

[0088] For the target attribute feature set, select the features z of two attributes d and e in a preset order. t,d z t,e ( The features of the two attributes are concatenated, and then the correlation strength between the two attributes is output through two fully connected layers and a sigmoid function.

[0089] Using the association strength between two attributes as the weight of the edge, and based on the association strength between all attributes, construct the adjacency matrix A for the target time step t. t ( ).

[0090] This graph edge weighting module uses LSTM to extract time-varying information from the time-series features of each channel, capturing the dependencies between different attributes of the storage device over time and reflecting the dynamic evolution between attributes. For each pair of attributes, the association strength (edge ​​weight) is calculated, resulting in a dynamic adjacency matrix that changes over time. This accurately characterizes the influence between attributes under different states, thus solving the problem that static graph structures cannot adapt to fault evolution and avoiding information fusion bias caused by traditional graph convolution using static graphs. By dynamically calculating edge weights, graph convolution can more accurately capture the key influences between attributes, improving the accuracy of remaining lifetime prediction.

[0091] Graph Convolution Module: For each time step, graph convolution is used to aggregate the attribute features of that time step based on the correlation strength of each pair of attributes, thus obtaining the deep spatial features of that time step. Specifically, the computation process of the graph convolution module includes:

[0092] The first global feature is divided into time steps to obtain the node feature matrix for each time step;

[0093] Based on the node feature matrix P at the target time step t t ( ) and adjacency matrix A t Two graph convolutional layers are used for information aggregation and feature transformation to obtain two graph information feature matrices g. t,1 g t,2 ( The calculation expression is:

[0094] Where m=0, 1, g t,0 =P t ; It is the weight matrix of the (m+1)th graph convolutional layer; It is the ReLU activation function.

[0095] The node feature matrix and multiple graph information feature matrices at the target time step t are weighted and fused to obtain the deep spatial feature H at the target time step t. t ( The calculation expression is:

[0096] ;in, ( ) is a learnable weight matrix.

[0097] By concatenating the deep spatial features of all time steps along the time dimension, the third global feature F3 is obtained. The third global feature is the output of the graph convolution module, which is also the output of the feature extraction module, and serves as the input to the next-level module.

[0098] This graph convolution module uses a learnable weight matrix. Valid information is filtered to avoid interference from redundant graph features. Residual connections are used to prevent the gradient vanishing problem in deep networks.

[0099] In one embodiment, an improved loss function is proposed for the training process of a lifetime prediction model. The loss function is:

[0100] ;

[0101] Where y is the actual remaining lifespan. It is the predicted remaining useful life; and It is the penalty coefficient, and Less than It can Set to 1, Set it to 2.

[0102] In lifespan prediction scenarios, predicting a remaining lifespan greater than the actual remaining lifespan is generally more dangerous, leading to delayed equipment maintenance, resulting in malfunctions, data loss, and even safety incidents. Conversely, predicting a remaining lifespan less than the actual value only leads to premature maintenance, which is relatively safer. This can be achieved by setting... Less than loss function Imposing a heavier penalty on prediction errors at that time, The error penalty is lighter. This asymmetric design guides the model during training to prioritize reducing overestimation of remaining lifetime, thus avoiding high-risk predictions at the loss function level.

[0103] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall fall within the scope of the present invention.

Claims

1. An AI-based distributed big data intelligent storage management method, characterized in that, The method comprises: Obtaining observation data of all storage devices in a distributed storage cluster; Preprocessing the observation data of a target storage device to obtain an initial feature matrix; the target storage device is any one of the storage devices; Inputting the initial feature matrix into a pre-trained life prediction model to obtain the remaining life of the target storage device; Regarding a storage device with a remaining life less than a first safety threshold as a high-risk device, regarding data stored on the storage device as to-be-migrated data, and including the data in a to-be-migrated source data set; Regarding a storage device with a remaining life greater than a second safety threshold as a healthy device, and including the storage device in a to-be-migrated target device set; the second safety threshold is greater than the first safety threshold; Generating a migration plan according to the to-be-migrated source data set and the to-be-migrated target device set, and performing data migration. 2.The AI-based distributed big data intelligent storage management method of claim 1, wherein, The observation data comprises SMART attribute data. 3.The AI-based distributed big data intelligent storage management method of claim 2, wherein, The preprocessing of the observation data of the target storage device to obtain the initial feature matrix comprises: Intercepting observation data of a preset time length and converting the observation data into a time sequence as first target data; Performing outlier rejection and missing value filling on the first target data to obtain second target data; Performing normalization processing on the second target data to obtain the initial feature matrix; each row of the initial feature matrix represents the time characteristics of an attribute; and each column represents an attribute feature set of a time step. 4.The AI-based distributed big data intelligent storage management method of claim 3, wherein, The life prediction model comprises a plurality of cascaded feature extraction modules and a prediction module; wherein: The plurality of cascaded feature extraction modules are configured to perform feature extraction on input data by using one-dimensional convolution and graph convolution to obtain high-level features; The prediction module is configured to map the high-level features to a remaining life value by using a fully connected network.

5. The AI-based distributed big data intelligent storage management method of claim 4, wherein, The plurality of feature extraction modules have the same structure, and any one of the feature extraction modules comprises a multi-scale convolution module, a graph edge weight module, and a graph convolution module; wherein: The multi-scale convolution module is configured to perform one-dimensional convolution operation on the time characteristics of each attribute in the input by using a plurality of convolution kernels of different scales, and enhance the convolution results by using an attention mechanism to obtain first global features; The graph edge weight module is configured to calculate the association strength of each pair of attributes in each time step by using a neural network to obtain an adjacency matrix of each time step; The graph convolution module is configured to, for each time step, aggregate the information of the attribute features of the time step according to the association strength of each pair of attributes by using graph convolution to obtain deep spatial features of the time step.

6. The AI-based distributed big data intelligent storage management method of claim 5, wherein, The operation process of the multi-scale convolution module comprises: Regarding any one attribute as a target attribute; regarding the time characteristics of the target attribute in the input as first time characteristics; Performing convolution operation on the first time characteristics by using a plurality of parallel convolution paths, and concatenating the convolution results of each convolution path in the channel dimension to obtain second time characteristics of the target attribute; each convolution path uses a one-dimensional convolution kernel of different sizes; Performing global average pooling on the second time characteristics of the target attribute, and mapping the pooling results to channel weights through a fully connected layer; weighting the second time feature of the target attribute using the channel weight to obtain a third time feature of the target attribute; integrating the third time features of all attributes to obtain a first global feature, and taking the first global feature as an output of the multi-scale convolution module.

7. The AI-based distributed big data intelligent storage management method of claim 6, wherein, The life prediction model includes three cascaded feature extraction modules. 8.The AI-based distributed big data intelligent storage management method of claim 5, wherein, The operation process of the graph edge weight module includes: Taking any one channel in the input as a target channel; extracting time-varying information of the target channel by using a one-layer long short-term memory network to obtain hidden features of the target channel; integrating the hidden features of all channels to obtain a second global feature; taking any one time step as a target time step, and taking an attribute feature set of the target time step in the second global feature as a target attribute feature set; for the target attribute feature set, taking the features of two attributes in a preset order, splicing the features of the two attributes, and then outputting the correlation strength between the two attributes through two fully connected layers; taking the correlation strength between the two attributes as the weight of the edge, and constructing an adjacency matrix of the target time step according to the correlation strength between all attributes. 9.The AI-based distributed big data intelligent storage management method of claim 5, wherein, The operation process of the graph convolution module includes: segmenting the first global feature according to time steps to obtain a node feature matrix of each time step; taking any one time step as a target time step; adopting multiple graph convolution layers to aggregate information and transform features according to the node feature matrix and the adjacency matrix of the target time step to obtain multiple graph information feature matrices; weighting and fusing the node feature matrix and the multiple graph information feature matrices of the target time step to obtain a deep spatial feature of the target time step; splicing the deep spatial features of all time steps along the time dimension to obtain a third global feature, and taking the third global feature as an input of a lower-level module.

10. The AI-based distributed big data intelligent storage management method of claim 4, wherein, The loss function loss of the life prediction model in the training process is: ； where y is the true remaining lifetime, is the remaining lifetime prediction value; and is a penalty coefficient, and is less than .

Images