Data optimization storage method and system based on artificial intelligence

Abstract

The invention is suitable for the field of data storage, and provides a data optimization storage method and system based on artificial intelligence, and the method comprises the steps: collecting and fusing real-time performance data of a storage node and logic relation data of a storage system, and forming a multi-dimensional monitoring data stream; identifying candidate fault equipment presenting early degradation characteristics through a pre-trained fault prediction model, and generating a fault early warning signal; constructing a physical topology and logic dependency graph, and deducing a data availability risk chain and an associated service influence chain caused by a fault; and formulating and executing an active defense strategy for data migration and access path adjustment according to the risk chain. According to the method, the storage fault is pre-judged in advance and the full-link risk prevention and control are realized, the data availability and stability of the storage system are improved, and the intelligent and active transformation of the storage operation and maintenance is promoted.

Description

Technical Field

[0001] This invention belongs to the field of data storage, and in particular relates to a data optimization storage method and system based on artificial intelligence. Background Technology

[0002] With the deepening of digital transformation, the scale of data has exploded, leading to the widespread application of architectures such as distributed storage and cloud storage in the field of computer storage technology. The scale and complexity of storage systems continue to increase. As the core infrastructure for data storage and management, the stability and data availability of storage systems directly affect the normal operation of various businesses, thus placing higher demands on the fault prevention and data protection capabilities of storage systems. In recent years, artificial intelligence technology has gradually penetrated the storage field, enabling intelligent operation and maintenance of storage systems. Analyzing storage performance data to identify and warn of faults has become an important trend in storage technology development.

[0003] Currently, the closest existing technologies in the storage field are mainly fault alarm and data protection solutions based on traditional monitoring tools. These solutions collect some performance indicators of storage nodes, set fixed thresholds to trigger anomaly alarms, and trigger preset failover or data recovery mechanisms when hardware failures or abnormal data access are detected. Some solutions use regular data backups to ensure data security, rely on manual intervention or simple automated scripts to complete fault handling, and lack in-depth mining and comprehensive analysis of the physical topology and logical dependencies of the storage system.

[0004] Existing technologies generally lack the ability to accurately identify early degradation characteristics of storage hardware, and rely heavily on passive fault response mechanisms. They cannot predict hardware failure risks in advance, nor can they comprehensively and systematically analyze the chain reaction of failures on data entities and upper-layer application services. This results in defense strategies that lack foresight and specificity, making it difficult to effectively avoid the risk of data loss or service interruption, and failing to meet the high availability and intelligent operation and maintenance requirements of complex storage systems. Summary of the Invention

[0005] The purpose of this invention is to provide a data optimization storage method based on artificial intelligence, which aims to solve the technical problems existing in the prior art as identified in the background art.

[0006] This invention is implemented as follows: a data optimization storage method based on artificial intelligence, the method comprising:

[0007] Collect and integrate real-time performance data of storage nodes and logical relationship data of storage system describing the relationship between data entities to form a multi-dimensional monitoring data stream;

[0008] Based on the multi-dimensional monitoring data stream, a fault prediction model is used to identify candidate fault devices in the storage hardware that exhibit early degradation characteristics, and a fault warning signal containing the candidate fault device identifier and degradation characteristics is generated.

[0009] Based on the fault warning signal and the logical relationship data of the storage system, a physical topology and logical dependency graph starting from the candidate fault device is constructed and analyzed, and the data availability risk chain and related service impact chain caused by the failure of the candidate fault device are deduced.

[0010] Based on the data availability risk chain and the related service impact chain, a prioritized proactive defense strategy is formulated and implemented.

[0011] As a further embodiment of the present invention, the formation of a multi-dimensional monitoring data stream specifically includes:

[0012] Continuously collect timing performance metrics from each storage node of the storage system. These timing performance metrics include input / output operation latency, read / write operations per second, read / write error count, bad sector growth rate, and storage media health parameters.

[0013] Synchronously obtain storage system logical relationship data from the storage system's configuration management library and metadata service. The storage system logical relationship data includes a data redundancy mapping table describing the distribution location of data replicas, a resource mapping table defining the correspondence between storage pools and physical hardware, and a service dependency table recording the binding relationship between applications and the data volumes they use.

[0014] The time-series performance metrics are aggregated and calculated according to a preset time window to obtain aggregated performance metrics. The aggregated performance metrics are then timestamped and matched with the logical relationship data of the storage system at the current moment to generate the structured multi-dimensional monitoring data stream.

[0015] As a further embodiment of the present invention, the generation of a fault warning signal including candidate faulty device identifiers and degradation characteristics specifically includes:

[0016] Collect the time-series performance metrics generated by the storage system during its historical operating cycle, and associate them with maintenance records of whether the corresponding storage nodes eventually failed, to construct a historical performance metric dataset and labeled fault samples;

[0017] The time-series performance indicators in the historical performance indicator dataset are aggregated and feature-engineered using a preset time window to form a historical feature sequence for model training.

[0018] Using the historical feature sequence and the labeled fault samples, supervised training is performed on the initial fault prediction model to obtain a pre-trained fault prediction model.

[0019] The aggregated performance metrics are extracted from the multi-dimensional monitoring data stream and used as the input feature sequence.

[0020] The input feature sequence is input into a pre-trained fault prediction model. The fault prediction model calculates a comprehensive early degradation probability score by analyzing the trend and correlation of each indicator in the input feature sequence deviating from the corresponding historical baseline pattern.

[0021] When the early degradation probability score exceeds a preset threshold, the corresponding storage hardware is determined to be a candidate fault device, and a fault warning signal is generated. The fault warning signal encapsulates the candidate fault device identifier, the early degradation probability score exceeding the threshold, and the key indicator features that cause the exceeding the threshold.

[0022] As a further aspect of the present invention, the deduction of the data availability risk chain and related service impact chain caused by the failure of the candidate faulty device specifically includes:

[0023] The candidate faulty device identifier is parsed from the fault warning signal;

[0024] Starting with the candidate fault device identifier, and combining the resource mapping table in the logical relationship data of the storage system, the associated hardware devices of the shared physical infrastructure are traversed level by level to construct a physical topology impact subgraph;

[0025] Starting with the data entities stored on the associated storage nodes in the candidate fault device and physical topology impact subgraph, the data redundancy mapping table and service dependency table in the logical relationship data of the storage system are used to traverse the data replicas, redundancy groups and upper-layer application services to construct the logical dependency impact subgraph.

[0026] Simulate scenarios where the candidate faulty devices experience faults of varying severity levels, perform propagation analysis on the physical topology influence subgraph and the logical dependency influence subgraph, enumerate the systemic risk paths caused by the fault scenarios, and form a data availability risk chain and an associated service influence chain.

[0027] As a further aspect of the present invention, the formulation and execution of a prioritized proactive defense strategy includes a strategy to pre-migrate data entities on the risk chain to a secure storage area before the candidate faulty device actually fails, and a strategy to adjust the data access paths of related services.

[0028] As a further aspect of the present invention, the strategy of pre-migrating critical data entities on the risk chain to a secure storage area before the candidate faulty device actually fails specifically includes:

[0029] Based on the physical topology influence subgraph and the logical dependency influence subgraph, all data entities stored on the associated hardware devices included in the physical topology influence subgraph are selected as the set of data entities to be migrated.

[0030] From the resource mapping table in the logical relationship data of the storage system, select a set of storage devices whose physical topology and power supply are independent of the physical topology influence subgraph and whose current health status is stable, and mark the storage pool to which the set of storage devices belongs as a safe storage area;

[0031] For each data entity in the set of data entities to be migrated, the source storage location is determined from the data redundancy mapping table according to the identifier of the data entity, and a data migration task from the source storage location to the secure storage area is initiated.

[0032] After all data replication or migration tasks are completed, the new copy location records of all data entities in the set of data entities to be migrated are updated in batches in the data redundancy mapping table.

[0033] As a further aspect of the present invention, the strategy for adjusting the data access path of associated services specifically includes:

[0034] After the data redundancy mapping table is updated in batches, all related services that depend on any one of the data entities in the set of data entities to be migrated are queried according to the service dependency table.

[0035] Send a path update notification to the application orchestrator corresponding to the associated service. The path update notification includes the identifiers of all data entities in the set of data entities to be migrated and the new access endpoint of each data entity in the secure storage area.

[0036] Receive confirmation feedback from the application orchestrator to confirm that the configuration of all associated services has been updated to point to the corresponding new access endpoint.

[0037] Another object of the present invention is to provide a data optimization storage system based on artificial intelligence, the system comprising:

[0038] The real-time performance data acquisition and fusion module is used to collect and fuse real-time performance data of storage nodes and logical relationship data of storage system describing the relationship between data entities to form a multi-dimensional monitoring data stream.

[0039] The fault prediction model module is used to identify candidate fault devices exhibiting early degradation characteristics in the storage hardware based on the multi-dimensional monitoring data stream, and generate a fault warning signal containing the candidate fault device identifier and degradation characteristics.

[0040] The data availability risk chain deduction module is used to construct and analyze the physical topology and logical dependency graph starting from the candidate faulty device based on the fault warning signal and the logical relationship data of the storage system, and to deduce the data availability risk chain and related service impact chain that will result if the candidate faulty device fails.

[0041] The proactive defense strategy formulation and execution module is used to formulate and execute prioritized proactive defense strategies based on the data availability risk chain and the related service impact chain.

[0042] The beneficial effects of this invention are:

[0043] This invention constructs a fully intelligent storage optimization system encompassing data acquisition and fusion, early fault prediction, risk chain deduction, and proactive defense execution. It fully leverages the advantages of artificial intelligence in data analysis and pattern recognition to achieve early prediction and precise prevention of storage system failures. By integrating multi-dimensional monitoring data, it accurately identifies early degradation characteristics of storage hardware, identifies candidate failure devices in advance, and then deduces the full-chain impact of the failure through physical topology and logical dependency graphs, making risk prediction more comprehensive and accurate. Based on a priority-based proactive defense strategy developed according to the risk chain, it can complete data security migration and service access path adjustment before hardware failure, blocking the impact of failures on data and services at the source. This significantly improves the data availability and operational stability of the storage system, reduces business losses caused by failures, and promotes the transformation of storage systems from passive operation and maintenance to a proactive and intelligent optimization model, adapting to the high-reliability operation requirements of complex storage environments. Attached Figure Description

[0044] Figure 1 A flowchart illustrating an artificial intelligence-based data optimization storage method provided in an embodiment of the present invention;

[0045] Figure 2 A flowchart illustrating the collection, fusion, and storage of real-time performance data from storage nodes, provided in an embodiment of the present invention;

[0046] Figure 3 A flowchart for generating a fault warning signal including candidate faulty device identifiers and degradation characteristics, provided for embodiments of the present invention;

[0047] Figure 4 A flowchart illustrating the data availability risk chain and related service impact chain that will result from the failure of a candidate faulty device, provided for embodiments of the present invention;

[0048] Figure 5 A flowchart for formulating and executing a prioritized proactive defense strategy provided in an embodiment of the present invention;

[0049] Figure 6This is a structural block diagram of an artificial intelligence-based data optimization storage system provided in an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0051] Figure 1 A flowchart of an artificial intelligence-based data optimization storage method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0052] S100 collects and integrates real-time performance data of storage nodes and logical relationship data of storage system describing the relationship between data entities to form a multi-dimensional monitoring data stream;

[0053] Real-time performance data collection needs to cover the core operational status dimensions of all storage nodes in the storage system. These time-series performance indicators directly reflect the physical health and service capabilities of the storage hardware. Continuous collection can capture subtle fluctuations and trends in these indicators over time, and these characteristics are key to the identification of early degradation signals by artificial intelligence models. Acquiring logical relationship data of the storage system focuses on sorting out the intrinsic connections between data, hardware, and services. This relational information serves as a bridge between physical state and logical impact, enabling isolated performance data to generate more valuable relational meanings.

[0054] Aggregation calculations of time-series performance indicators within a preset time window are a way to refine and organize the original, scattered data. By extracting statistical features from different time dimensions, noise interference caused by instantaneous fluctuations can be filtered out, highlighting regular state change trends and making the data more in line with the input requirements of artificial intelligence models.

[0055] Aligning the aggregated performance metrics with the current storage system logical relationship data using timestamps and matching association keys is the core step in achieving data fusion. Synchronization in the time dimension ensures data consistency within the same time slice, while binding association keys enables precise mapping between performance data and logical relationships. The resulting structured, multi-dimensional monitoring data stream can fully present the physical operating status and logical relationship structure of the storage system at a specific point in time.

[0056] In large-scale distributed storage scenarios, massive storage nodes continuously generate performance data, and there are complex logical bindings between different nodes and data volume application services. This process can integrate data scattered across various nodes and system components into a unified analytical foundation.

[0057] S200, based on the multi-dimensional monitoring data stream, the fault prediction model identifies candidate fault devices in the storage hardware that exhibit early degradation characteristics, and generates a fault warning signal containing the candidate fault device identifier and degradation characteristics.

[0058] The construction of historical performance index datasets and labeled fault samples needs to cover the full amount of time-series performance data under different operating conditions and different hardware types during the long-term operation of the storage system. Through precise association with maintenance records, clear fault labels are provided for the model, enabling the model to learn the complete evolution path from normal operation characteristics to early degradation characteristics to fault occurrence.

[0059] The pre-defined time window aggregation and feature engineering of historical time series performance indicators not only regularize the original data, but also highlight the changing trends and abnormal fluctuations of performance indicators through operations such as time series differencing and normalization, remove irrelevant noise, and transform scattered indicator data into a highly identifiable historical feature sequence, enabling the model to efficiently capture subtle signs before a failure occurs.

[0060] Supervised training of the initial fault prediction model utilizes the correspondence between historical feature sequences and labeled fault samples to allow the model to gradually grasp the intrinsic relationship between different combinations of performance indicators, different trends of change, and hardware faults, forming a stable fault identification logic and ensuring that the model has the ability to generalize to adapt to the operating characteristics of different storage hardware.

[0061] By extracting aggregated performance metrics from multi-dimensional monitoring data streams as input feature sequences, the consistency between the model's input data and training data is ensured, laying the foundation for accurate model output results. During computation, the fault prediction model does not analyze individual metrics in isolation, but rather deeply mines the magnitude and duration of deviations of each metric from historical baselines, as well as the mutual influence between metrics. A weighted calculation yields a comprehensive early degradation probability score, fully quantifying the hardware's degradation state. When the score exceeds a preset threshold, the generated fault warning signal not only identifies candidate faulty devices but also includes key degradation indicators, providing precise guidance for subsequent risk analysis.

[0062] In large-scale storage cluster scenarios, the long-term operation of massive storage nodes generates a large amount of time-series performance data. The rich sample library formed by linking maintenance records allows the model to fully learn the degradation patterns of different types of storage hardware. Whether it is degradation caused by the growth of bad sectors in mechanical hard drives or anomalies caused by the decline in the health of solid-state drive storage media, the model can accurately identify and lock high-risk devices in advance through the differences in feature sequences.

[0063] S300, based on the fault warning signal and the logical relationship data of the storage system, construct and analyze the physical topology and logical dependency graph starting from the candidate fault device, and deduce the data availability risk chain and related service impact chain that will result if the candidate fault device fails.

[0064] Extracting candidate faulty device identifiers from fault warning signals is the starting point of the entire deduction process. Only by accurately identifying the warning devices can we ensure that all subsequent topology construction and risk analysis revolve around the real risk source, avoiding deviations in the deduction direction. When constructing the physical topology impact subgraph, candidate faulty devices are used as the core. The resource mapping table is used to traverse the associated hardware of the shared physical infrastructure. The underlying logic is that the propagation of physical hardware faults often stems from the correlation of shared resources. For example, devices sharing power supply modules, storage controllers, or transmission buses can trigger a chain reaction due to the failure of a single device. By depicting this correlation, the subgraph accurately defines the physical fault propagation boundary, making hidden physical correlation risks explicit.

[0065] The construction of the logical dependency impact subgraph focuses on the relationship between data and services. Starting from the data entities on the candidate faulty devices and associated storage nodes, the data redundancy mapping table clarifies the data replica distribution, and the service dependency table sorts out the binding relationship between applications and data. This design not only considers the buffering effect of the data redundancy mechanism on risks, but also takes into account the data dependency requirements of upper-layer services, and opens up a complete logical link from physical hardware failure to changes in data availability, and then to the impact on service operation.

[0066] Simulating different severity levels of fault scenarios is necessary because the impact of faults varies significantly. Mild faults may only affect hardware performance, moderate faults may cause partial functional failures, and severe faults may cause complete hardware downtime. The scope and speed of fault propagation are drastically different in different scenarios. Based on the propagation analysis of the physical topology and logical dependency impact subgraph, following the objective laws from physical fault triggering to logical correlation propagation, the impact status of faults on related hardware, data entities, and application services is determined layer by layer. All possible systemic risk paths are enumerated, and finally a structured data availability risk chain and related service impact chain are formed.

[0067] In large-scale distributed storage environments, there are complex physical relationships and logical bindings between storage nodes, data entities, and application services. When a storage node exhibits early degradation characteristics, it may physically affect other nodes belonging to the same storage cluster and sharing the core controller. Logically, it may affect the business data stored on these nodes, as well as transaction systems, data analysis services, and other services that rely on this data. By deducing this process, the complete risk path can be clearly presented, from increased data access latency under minor failures to partial service degradation under moderate failures, and core business interruption under severe failures, providing a comprehensive reference for subsequent targeted defenses.

[0068] S400, based on the data availability risk chain and the related service impact chain, formulate and execute a prioritized proactive defense strategy.

[0069] When screening the set of data entities to be migrated, the physical topology influence subgraph and the logical dependency influence subgraph are deeply combined to fully cover all data stored on related hardware devices, ensuring that no potentially risky data is missed. The underlying logic is that the transmission of risk chains is related, and migrating only part of the data will cause the remaining risky data to face the risk of loss when a failure occurs.

[0070] The selection of secure storage areas strictly follows the principle of dual independence of physical topology and power supply, while emphasizing the healthy and stable state of storage devices. This dual standard cuts off the fault propagation path at the hardware infrastructure level and ensures the storage security of data after migration at the device operation level, avoiding secondary risks caused by problems in the secure storage area itself.

[0071] During data migration, the source storage location is accurately determined based on the data redundancy mapping table to ensure the accuracy of the migrated objects. Batch updates of the new replica location records enable real-time synchronization of storage system metadata, ensuring consistency in subsequent data access and management operations. In the access path adjustment phase, all related services are first locked through the service dependency table to ensure that no service that depends on the migrated data is missed. A path update notification containing the data identifier and the new access endpoint is sent to provide application services with a clear basis for adjustment. Receiving confirmation feedback forms a closed-loop verification to ensure that all related services have completed configuration updates, avoiding data access failures or service anomalies due to some services not adapting to the new path.

[0072] In large-scale storage cluster scenarios, when the risk chain involves massive data entities and numerous upper-layer application services, this step can systematically screen the data to be migrated, accurately locate secure storage resources, and efficiently complete data migration and path adjustment. Even if the candidate faulty device subsequently experiences a severe failure, it can ensure that the data is intact and available and the service operation is unaffected.

[0073] like Figure 2 As shown, the formation of the multi-dimensional monitoring data stream specifically includes:

[0074] S110, continuously collect timing performance metrics from each storage node of the storage system. The timing performance metrics include input / output operation latency, read / write operations per second, read / write error count, bad sector growth rate, and storage medium health parameters.

[0075] S120, synchronously obtain storage system logical relationship data from the configuration management library and metadata service of the storage system. The storage system logical relationship data includes a data redundancy mapping table describing the distribution location of data replicas, a resource mapping table defining the correspondence between storage pools and physical hardware, and a service dependency table recording the binding relationship between applications and the data volumes they use.

[0076] S130, the time-series performance indicators are aggregated and calculated according to a preset time window to obtain aggregated performance indicators, and the aggregated performance indicators are timestamped and matched with the storage system logical relationship data at the current time to generate the structured multi-dimensional monitoring data stream.

[0077] For the time-series performance metrics of storage nodes, a pre-defined time window is used to perform aggregated calculations of a single metric within the window. The core calculation formula is as follows:

[0078] ;

[0079] in:

[0080] Indicates the first The storage node's first The time-series performance index is in the first The aggregated performance metrics values ​​within a preset time window;

[0081] This indicates the preset aggregation calculation function, which can be selected according to the type of performance indicator, such as mean, maximum, minimum, cumulative value, variance, growth rate, etc. For example, the cumulative value is used for read / write error count, and the mean value is used for input / output operation delay.

[0082] Indicates the first The storage node's first Time-series performance metrics at timestamps The original data collected at that moment;

[0083] A unique identifier for a storage node, corresponding to a different physical storage hardware device in the storage system;

[0084] A unique identifier representing timing performance metrics, such as the sequence number of metrics like input / output operation latency and read / write operations per second;

[0085] Indicates the timestamp of the original time-series performance metrics collection;

[0086] Indicates the first The set of timestamps corresponding to each preset time window, i.e. This includes the collection timestamps of all raw metrics within the window;

[0087] A unique identifier representing a preset time window, used to distinguish window aggregation results for different time periods.

[0088] The aggregated performance metrics are timestamped and matched with the logical relationship data of the storage system at the current moment. The core purpose is to ensure that the performance metrics data reflecting the operating status of the storage hardware and the logical relationship data describing the relationship between data entities, hardware, and services maintain strict consistency in the time dimension, while achieving precise association binding in the entity identifier dimension. This avoids data fragmentation and association errors in the multi-dimensional monitoring data stream caused by time misalignment or identifier mismatch, thereby generating a structured, spatiotemporally consistent, and effectively associated multi-dimensional monitoring data stream.

[0089] like Figure 3 As shown, the generation of a fault warning signal containing candidate faulty device identifiers and degradation characteristics specifically includes:

[0090] S210, collect the time-series performance indicators generated by the storage system during the historical operating cycle, and associate them with the maintenance records of whether the corresponding storage nodes eventually failed, to construct a historical performance indicator dataset and labeled fault samples.

[0091] S220, perform preset time window aggregation calculation and feature engineering on the time-series performance indicators in the historical performance indicator dataset to form a historical feature sequence for model training;

[0092] Specifically:

[0093] ;

[0094] in:

[0095] Indicates the first The first historical fault sample corresponding to the first The storage node at the ... The historical feature sequence formed within a preset time window after aggregation and feature engineering;

[0096] This represents the feature engineering processing function, which includes operations such as normalization, standardization, temporal difference, feature concatenation, and outlier removal. It is a processing and transformation of the aggregated performance indicators.

[0097] Indicates the first The storage node's first The time-series performance index is in the first The aggregated performance index values ​​within a preset time window serve as the input basis for feature engineering.

[0098] A unique identifier representing a historical fault sample, corresponding to a single storage node fault case bound to maintenance records within the historical operating cycle;

[0099] A set of identifiers representing all timing performance metrics, including all metrics such as input / output operation latency and bad sector growth rate;

[0100] This represents the set of hyperparameters for feature engineering, including key parameters such as normalized mean and standard deviation, standardized scaling factor, and step size for temporal differencing.

[0101] S230, using the historical feature sequence and the labeled fault samples, supervised training is performed on the initial fault prediction model to obtain a pre-trained fault prediction model;

[0102] S240, extract the aggregated performance indicators from the multi-dimensional monitoring data stream as the input feature sequence;

[0103] S250, The input feature sequence is input into a pre-trained fault prediction model. The fault prediction model calculates a comprehensive early degradation probability score by analyzing the trend and correlation of each indicator in the input feature sequence deviating from the corresponding historical baseline pattern.

[0104] The fault prediction model calculates a comprehensive early degradation probability score based on the input feature sequence. The core calculation formula is as follows:

[0105] ;

[0106] in:

[0107] Indicates the first The overall early degradation probability score of each storage hardware unit, with a value range of [value missing]. A higher value indicates a higher probability of early hardware degradation.

[0108] Indicates the first The weighting coefficients of key performance indicators on storage hardware degradation are set according to the correlation between the indicator and hardware failure. For example, the weight of bad sector growth rate and storage media health parameters is higher than the number of read / write operations per second.

[0109] Indicates the first The first storage hardware The probability value of early degradation of a single key performance indicator deviating from the historical baseline pattern is output by the fault prediction model through analysis of the indicator's trend and correlation, and the value range is [value range missing]. ;

[0110] This represents the total number of key performance indicators involved in the comprehensive probability calculation.

[0111] S260, when the early degradation probability score exceeds a preset threshold, the corresponding storage hardware is determined to be a candidate fault device, and a fault warning signal is generated. The fault warning signal encapsulates the candidate fault device identifier, the early degradation probability score exceeding the standard, and the key indicator features that cause the exceeding of the standard.

[0112] The core criterion for determining candidate faulty devices is the comprehensive early degradation probability score output by the pre-trained fault prediction model. This score is a quantitative assessment of the current early degradation state of the hardware based on the input feature sequence of the storage hardware. It is obtained by analyzing the trend of each key performance indicator deviating from the historical baseline pattern and the correlation between the indicators. It is a quantitative representation of the probability of hardware failure. Its calculation process is based on supervised training of historical fault samples and has statistically effective fault prediction. Therefore, this score is used as the core criterion for determination.

[0113] Exceeding the first preset threshold indicates that the degradation characteristics of the key performance indicators of the storage hardware have significantly deviated from the historical baseline level of normal operation. The early degradation trend exhibited by the hardware is no longer a normal operational fluctuation, but has statistically significant characteristics of a precursor to failure. The probability of actual hardware failure has increased to a critical level that requires the activation of early warning and proactive defense. If intervention measures are not taken in time, the hardware is likely to experience failures of varying degrees in the short term, thereby triggering a chain reaction of problems in data storage or upper-layer services.

[0114] The preset threshold is set based on historical performance index datasets and labeled fault samples. ROC curves and precision-recall curves are plotted, and the area under the curve is calculated to determine the optimal critical point, resulting in an initial threshold at the statistical level. This threshold can balance the precision and recall of fault prediction, reducing false negatives and false positives. Secondly, it is adapted to the business scenarios of the storage system. For example, the threshold for storage hardware carrying core business data is set lower (more sensitive), while the threshold for hardware carrying non-core data can be appropriately increased. At the same time, it is fine-tuned based on the enterprise's operation and maintenance costs, fault tolerance, and data recovery capabilities. Finally, during the actual system operation, the threshold is continuously iterated and optimized based on the actual effect of fault warnings and the false negatives / false positives reported by operation and maintenance, so that the threshold always adapts to the operating status of the storage system and business needs.

[0115] like Figure 4As shown, the deduced data availability risk chain and related service impact chain that would result from a failure of a candidate faulty device specifically include:

[0116] S310, parse the candidate faulty device identifier from the fault warning signal;

[0117] S320, starting with the candidate fault device identifier as the starting node, combines the resource mapping table in the storage system logical relationship data, and traverses the associated hardware devices of the shared physical infrastructure level by level to construct a physical topology impact subgraph;

[0118] The physical topology impact subgraph is a graph structure that reflects the physical relationships between storage hardware. It starts with the candidate faulty device identifier and combines it with the resource mapping table in the storage system's logical relationship data. It traverses the associated hardware devices sharing the physical infrastructure level by level. The nodes of this subgraph are various physical hardware devices in the storage system, including the candidate faulty device and all storage nodes, controllers, buses, power supply modules, etc. that are physically associated with it. The edges of the subgraph represent the physical relationships between hardware devices, such as shared cabinets, shared power supply, connection through the same storage controller, shared data transmission bus, etc. The weight of the edges can be set according to the tightness of the physical relationship. The tighter the relationship, the higher the possibility of fault propagation.

[0119] The core function of the physical topology influence subgraph is to clearly and intuitively depict the direct and indirect impact range that may occur at the physical hardware level when a candidate faulty device fails, accurately locate which physical hardware devices will be affected by the failure of the candidate faulty device, or even be triggered by secondary faults, and at the same time clarify the fault propagation path between physical hardware.

[0120] S330 starts with the data entities stored on the associated storage nodes in the candidate fault device and physical topology influence subgraph, and uses the data redundancy mapping table and service dependency table in the logical relationship data of the storage system to traverse the data replicas, redundancy groups and upper-layer application services to construct the logical dependency influence subgraph.

[0121] The logical dependency impact subgraph starts with the data entities on the associated storage nodes in the candidate fault device and physical topology impact subgraph. It uses the data redundancy mapping table and service dependency table in the logical relationship data of the storage system to traverse the data replicas, data redundancy groups, storage pools and upper-layer application services layer by layer to construct a directed graph structure that reflects the logical relationship between data entities and application services. The nodes of this subgraph contain multiple types of logical entities, specifically data entities such as data volumes, data blocks and files stored on hardware devices, as well as storage pools, applications, business services, etc. The edges of the subgraph represent the logical dependency relationships between nodes, such as the replica mapping relationship of data entities, the binding and usage relationship between application services and data volumes, the ownership relationship between storage pools and data entities, etc. The direction of the edges represents the direction of data access or failure impact propagation.

[0122] The core function of the logical dependency impact subgraph is to comprehensively analyze, from the logical level of data and services, how the physical hardware failure will propagate to data entities after the failure of a candidate faulty device, thereby affecting the operation status of upper-layer application services. It accurately identifies the propagation path of availability risks at the data level and the scope of affected application services, making up for the limitation of the physical topology impact subgraph, which only focuses on the hardware level and cannot relate data and services.

[0123] S340, Simulate scenarios where the candidate faulty devices experience faults of different severity levels, perform propagation analysis on the physical topology influence subgraph and the logical dependency influence subgraph, enumerate the systemic risk paths caused by the fault scenarios, and form a data availability risk chain and an associated service influence chain.

[0124] The specific steps for simulating different severity levels of failure scenarios for candidate faulty devices and forming a data availability risk chain and related service impact chain are as follows: First, based on the device type, hardware structure, and actual operation and maintenance experience of the storage hardware, define multiple severity levels of failure scenarios for candidate faulty devices, which are usually divided into three levels: minor failure, moderate failure, and severe failure. At the same time, clarify the initial failure characteristics and impact of each failure scenario.

[0125] Subsequently, each level of fault scenario was used as the initial fault condition. Risk propagation analysis was carried out sequentially on the physical topology impact subgraph and the logical dependency impact subgraph. First, in the physical topology impact subgraph, starting from the candidate faulty device, based on the tightness of the physical association edges between hardware and the preset fault propagation rules, it was determined whether the fault would propagate from the current node to the associated hardware, and the attenuation of the fault severity level after propagation, to determine the scope and state of the physical fault impact. Then, based on the physical fault results, in the logical dependency impact subgraph, based on the redundancy guarantee rules such as the number of replicas and the distribution of redundancy groups defined in the data redundancy mapping table, the availability status of the data entity on each affected hardware was determined. Then, based on the binding dependency relationship between the application service and the data entity recorded in the service dependency table, it was determined whether the application service would be affected by service degradation, service lag, or complete service interruption due to data unavailability or decreased availability, and the degree of service impact was also clarified.

[0126] Based on the fault propagation results at the physical and logical levels, we enumerate all complete and continuous systemic risk paths from the candidate faulty device, through physical hardware nodes and data entity nodes to the application service nodes, and sort out the fault propagation level, affected nodes and severity of each path.

[0127] Finally, these risk paths are integrated and categorized according to the logical relationships and hierarchical connections of fault propagation, forming a structured data availability risk chain and related service impact chain. The judgment throughout the process is based on the objective laws of physical hardware fault propagation in the storage field, the clearly defined data redundancy mapping rules and service dependency rules in the logical relationship data of the storage system, as well as the fault propagation experience accumulated by enterprises in long-term operation and maintenance and the fault impact judgment business rules formulated for business scenarios. At the same time, the fault tolerance capability of the data redundancy mechanism and the disaster recovery configuration of application services are used as auxiliary judgments.

[0128] The core function of the data availability risk chain and related service impact chain is to comprehensively, clearly, and structurally demonstrate the complete risk transmission path, impact scope, and degree of impact of each node of candidate faulty devices at different fault severity levels, from the physical hardware level to the data storage level and then to the upper-layer application service level. It quantifies the systemic impact that a fault may cause, providing accurate, specific, and implementable risk basis for the subsequent formulation of prioritized proactive defense strategies. This allows the formulation of defense strategies to revolve around the key nodes and core impact paths of the risk chain, achieving targeted and differentiated risk prevention and control, avoiding blind defense measures, and enabling operations and maintenance personnel to quickly identify the core data and critical services affected by the fault, prioritizing defense measures for high-risk nodes and paths, and minimizing the potential losses caused by the fault.

[0129] Furthermore, the formulation and execution of prioritized proactive defense strategies include strategies for pre-migrating data entities on the risk chain to a secure storage area before the candidate faulty device actually fails, and strategies for adjusting the data access paths of related services.

[0130] like Figure 5 As shown, the strategy of pre-migrating critical data entities on the risk chain to a secure storage area before the candidate faulty device actually fails specifically includes:

[0131] S410, based on the physical topology influence subgraph and the logical dependency influence subgraph, select all data entities stored on the associated hardware devices contained in the physical topology influence subgraph as a set of data entities to be migrated;

[0132] S420, from the resource mapping table in the logical relationship data of the storage system, select a set of storage devices whose physical topology and power supply are independent of the physical topology influence subgraph and whose current health status is stable, and mark the storage pool to which the set of storage devices belongs as a safe storage area;

[0133] S430, for each data entity in the set of data entities to be migrated, determine the source storage location from the data redundancy mapping table according to the identifier of the data entity, and start the data migration task from the source storage location to the secure storage area.

[0134] S440, after all data copying or migration tasks are completed, update the new copy location records of all data entities in the set of data entities to be migrated in the data redundancy mapping table in batches.

[0135] The strategy for adjusting the data access path of related services specifically includes:

[0136] S450, after the data redundancy mapping table is updated in batches, query all related services that depend on any data entity in the set of data entities to be migrated according to the service dependency table;

[0137] S460, Send a path update notification to the application orchestrator corresponding to the associated service. The path update notification includes the identifiers of all data entities in the set of data entities to be migrated and the new access endpoint of each data entity in the secure storage area.

[0138] S470 receives confirmation feedback from the application orchestrator, confirming that the configuration of all associated services has been updated to point to the corresponding new access endpoint.

[0139] Figure 6 A structural block diagram of an artificial intelligence-based data optimization storage system provided in an embodiment of the present invention is shown below. Figure 6As shown, the system includes:

[0140] The real-time performance data acquisition and fusion module 100 is used to collect and fuse real-time performance data of storage nodes and storage system logical relationship data describing the relationship between data entities to form a multi-dimensional monitoring data stream.

[0141] The fault prediction model module 200 is used to identify candidate fault devices exhibiting early degradation characteristics in the storage hardware based on the multi-dimensional monitoring data stream, and generate a fault warning signal containing the candidate fault device identifier and degradation characteristics.

[0142] The data availability risk chain deduction module 300 is used to construct and analyze the physical topology and logical dependency graph starting from the candidate faulty device based on the fault warning signal and the logical relationship data of the storage system, and to deduce the data availability risk chain and related service impact chain that will result if the candidate faulty device fails.

[0143] The proactive defense strategy formulation and execution module 400 is used to formulate and execute prioritized proactive defense strategies based on the data availability risk chain and the related service impact chain.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

[0146] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A data optimization storage method based on artificial intelligence, characterized in that, The method includes: Collect and integrate real-time performance data of storage nodes and logical relationship data of storage system describing the relationship between data entities to form a multi-dimensional monitoring data stream; Based on the multi-dimensional monitoring data stream, a fault prediction model is used to identify candidate fault devices in the storage hardware that exhibit early degradation characteristics, and a fault warning signal containing the candidate fault device identifier and degradation characteristics is generated. Based on the fault warning signal and the logical relationship data of the storage system, a physical topology and logical dependency graph starting from the candidate fault device is constructed and analyzed, and the data availability risk chain and related service impact chain caused by the failure of the candidate fault device are deduced. Based on the data availability risk chain and the related service impact chain, a prioritized proactive defense strategy is formulated and implemented.

2. The method according to claim 1, characterized in that, The formation of the multi-dimensional monitoring data stream specifically includes: Continuously collect timing performance metrics from each storage node of the storage system. These timing performance metrics include input / output operation latency, read / write operations per second, read / write error count, bad sector growth rate, and storage media health parameters. Synchronously obtain storage system logical relationship data from the storage system's configuration management library and metadata service. The storage system logical relationship data includes a data redundancy mapping table describing the distribution location of data replicas, a resource mapping table defining the correspondence between storage pools and physical hardware, and a service dependency table recording the binding relationship between applications and the data volumes they use. The time-series performance metrics are aggregated and calculated according to a preset time window to obtain aggregated performance metrics. The aggregated performance metrics are then timestamped and matched with the logical relationship data of the storage system at the current moment to generate the structured multi-dimensional monitoring data stream.

3. The method according to claim 2, characterized in that, The generation of a fault warning signal containing candidate faulty device identifiers and degradation characteristics specifically includes: Collect the time-series performance metrics generated by the storage system during its historical operating cycle, and associate them with maintenance records of whether the corresponding storage nodes eventually failed, to construct a historical performance metric dataset and labeled fault samples; The time-series performance indicators in the historical performance indicator dataset are aggregated and feature-engineered using a preset time window to form a historical feature sequence for model training. Using the historical feature sequence and the labeled fault samples, supervised training is performed on the initial fault prediction model to obtain a pre-trained fault prediction model. The aggregated performance metrics are extracted from the multi-dimensional monitoring data stream and used as the input feature sequence. The input feature sequence is input into a pre-trained fault prediction model. The fault prediction model calculates a comprehensive early degradation probability score by analyzing the trend and correlation of each indicator in the input feature sequence deviating from the corresponding historical baseline pattern. When the early degradation probability score exceeds a preset threshold, the corresponding storage hardware is determined to be a candidate fault device, and a fault warning signal is generated. The fault warning signal encapsulates the candidate fault device identifier, the early degradation probability score exceeding the threshold, and the key indicator features that cause the exceeding the threshold.

4. The method according to claim 3, characterized in that, The deduced data availability risk chain and related service impact chain that would result from a failure of a candidate faulty device specifically include: The candidate faulty device identifier is parsed from the fault warning signal; Starting with the candidate fault device identifier, and combining the resource mapping table in the logical relationship data of the storage system, the associated hardware devices of the shared physical infrastructure are traversed level by level to construct a physical topology impact subgraph; Starting with the data entities stored on the associated storage nodes in the candidate fault device and physical topology impact subgraph, the data redundancy mapping table and service dependency table in the logical relationship data of the storage system are used to traverse the data replicas, redundancy groups and upper-layer application services to construct the logical dependency impact subgraph. Simulate different severity levels of failure scenarios for the candidate faulty devices, perform propagation analysis on the physical topology influence subgraph and the logical dependency influence subgraph, enumerate the systemic risk paths caused by the failure scenarios, and form a data availability risk chain and an associated service influence chain.

5. The method according to claim 4, characterized in that, The formulation and implementation of prioritized proactive defense strategies include strategies for pre-migrating data entities on the risk chain to a secure storage area before the candidate faulty device actually fails, and strategies for adjusting the data access paths of related services.

6. The method according to claim 5, characterized in that, The strategy of pre-migrating data entities on the risk chain to a secure storage area before the candidate faulty device actually fails specifically includes: Based on the physical topology influence subgraph and the logical dependency influence subgraph, all data entities stored on the associated hardware devices included in the physical topology influence subgraph are selected as the set of data entities to be migrated. From the resource mapping table in the logical relationship data of the storage system, select a set of storage devices whose physical topology and power supply are independent of the physical topology influence subgraph and whose current health status is stable, and mark the storage pool to which the set of storage devices belongs as a safe storage area; For each data entity in the set of data entities to be migrated, the source storage location is determined from the data redundancy mapping table according to the identifier of the data entity, and a data migration task from the source storage location to the secure storage area is initiated. After all data replication or migration tasks are completed, the new copy location records of all data entities in the set of data entities to be migrated are updated in batches in the data redundancy mapping table.

7. The method according to claim 6, characterized in that, The strategy for adjusting the data access path of related services specifically includes: After the data redundancy mapping table is updated in batches, all related services that depend on any one of the data entities in the set of data entities to be migrated are queried according to the service dependency table. Send a path update notification to the application orchestrator corresponding to the associated service. The path update notification includes the identifiers of all data entities in the set of data entities to be migrated and the new access endpoint of each data entity in the secure storage area. Receive confirmation feedback from the application orchestrator to confirm that the configuration of all associated services has been updated to point to the corresponding new access endpoint.

8. A data optimization storage system based on artificial intelligence, characterized in that, The system includes: The real-time performance data acquisition and fusion module is used to collect and fuse real-time performance data of storage nodes and logical relationship data of storage system describing the relationship between data entities to form a multi-dimensional monitoring data stream. The fault prediction model module is used to identify candidate fault devices exhibiting early degradation characteristics in the storage hardware based on the multi-dimensional monitoring data stream, and generate a fault warning signal containing the candidate fault device identifier and degradation characteristics. The data availability risk chain deduction module is used to construct and analyze the physical topology and logical dependency graph starting from the candidate faulty device based on the fault warning signal and the logical relationship data of the storage system, and to deduce the data availability risk chain and related service impact chain that will result if the candidate faulty device fails. The proactive defense strategy formulation and execution module is used to formulate and execute prioritized proactive defense strategies based on the data availability risk chain and the related service impact chain.

Images