Distributed storage data active migration method and system based on intelligent health prediction

Abstract

The application discloses a kind of based on intelligent health prediction's distributed storage data active migration method and system, it is related to distributed storage system operation and maintenance technical field, wherein the method includes: periodically collecting the multidimensional health state index of each node in distributed storage cluster;Based on health state index, the health score of each node is calculated by multi-time scale dynamic model;When the health score of any node is continuously lower than the preset active migration threshold, mark the node as high-risk node, and trigger the active migration process for all data replicas on the node;Construct multi-objective optimization cost function for the data to be migrated, and select the candidate node with the lowest comprehensive cost from the health node pool as the migration target according to the cost function;With resource-aware dynamic rate, migrate the data on the high-risk node to the selected target node, and atomically update the metadata after the data migration is successful, then asynchronously clean up the old data replicas on the high-risk node.

Description

Technical Field

[0001] This invention relates to the field of distributed storage system operation and maintenance technology, and in particular to a method and system for proactive migration of distributed storage data based on intelligent health prediction. Background Technology

[0002] Distributed storage systems, as a core supporting technology for cloud computing and big data architectures, are widely used in enterprise-level data centers and cloud service platforms. Among related technologies, a passive recovery system centered on node fault detection and data reconstruction is constructed through the collaborative operation of multi-replica redundancy and erasure coding techniques. Specifically, this system covers the entire process from hardware status monitoring to data redundancy recovery, including key stages such as heartbeat detection, fault determination, and data reconstruction. Traditional solutions rely on periodic heartbeat timeout mechanisms to trigger data reconstruction, restoring faulty node data from remaining replicas or erasure coding blocks to maintain redundancy; however, its technical architecture is essentially a "post-fault response" mode.

[0003] However, existing passive recovery methods rely on single-dimensional assessment models that directly use threshold judgments, lacking a multi-timescale health prediction system. This can lead to delayed system response to soft faults. Specifically, when nodes experience progressive degradation such as slow bad sectors on hard drives (SMART anomalies) or controller performance degradation (IOPS fluctuations), current technologies typically use static threshold judgments, failing to capture the synergistic effect of long-term trends and short-term fluctuations. Based on this, traditional solutions suffer from three systemic flaws: First, centralized data reconstruction triggers resource contention, leading to a 30%-50% decrease in business IOPS and a 2-3 order of magnitude increase in latency; second, redundancy during fault recovery is lower than preset values, increasing the probability of secondary failures by 5-10 times; third, target node selection strategies only consider capacity or random allocation, easily creating new hotspots (increasing load unevenness by 40%) or low migration efficiency (increasing network latency by 30%). These flaws are particularly pronounced as cluster size increases, necessitating a proactive operational paradigm to reconstruct the technology. Summary of the Invention

[0004] The main objective of this invention is to provide a method for proactive migration of distributed storage data based on intelligent health prediction.

[0005] Another objective of this invention is to propose a distributed storage data proactive migration system based on intelligent health prediction.

[0006] The third objective of this invention is to provide a computer device.

[0007] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0008] To achieve the above objectives, a first aspect of the present invention proposes a method for proactive migration of distributed storage data based on intelligent health prediction, comprising:

[0009] S1 periodically collects multi-dimensional health status indicators of each node in the distributed storage cluster. The health status indicators include hard disk subsystem indicators, network subsystem indicators, computing and memory subsystem indicators, and software system indicators.

[0010] S2, based on health status indicators, calculates the health score of each node through a multi-timescale dynamic model. The multi-timescale dynamic model includes long-term trend analysis of chronic indicators, short-term fluctuation assessment of acute indicators, and exponential decay penalty processing of event-type indicators. It also introduces a collaborative penalty factor to quantify and correct the composite risk of multiple abnormal indicators.

[0011] S3, when the health score of any node continues to be lower than the preset active migration threshold, the node is marked as a high-risk node and an active migration process for all data replicas on the node is triggered;

[0012] S4. Construct a multi-objective optimization cost function for the data to be migrated. The cost function includes load balancing cost, data locality cost, migration efficiency cost, and storage lifetime balancing cost. Based on the cost function, select the candidate node with the lowest comprehensive cost from the healthy node pool as the migration target.

[0013] S5 migrates data from high-risk nodes to selected target nodes at a resource-aware dynamic rate, atomically updates metadata after successful data migration, and then asynchronously cleans up old data copies on high-risk nodes.

[0014] In one embodiment of the present invention, multi-dimensional health status indicators of each node in the distributed storage cluster are collected periodically, including:

[0015] S11, the hard disk subsystem metrics include the reallocation sector count, read / write error rate, instruction timeout count, seek error rate in the SMART information, as well as real-time hard disk read / write latency and IOPS;

[0016] S12, the network subsystem metrics include packet loss rate, packet error rate, bandwidth utilization, and TCP retransmission rate of the network interface, and the computing and memory subsystem metrics include CPU temperature, CPU core voltage, fan speed, and memory ECC error count.

[0017] In one embodiment of the present invention, based on health status indicators, the health score of each node is calculated using a multi-timescale dynamic model, including:

[0018] S21, set the observation time window for chronic indicators to 90 days, the observation time window for acute indicators to 24 hours, and the observation time window for event-type indicators to instantaneous.

[0019] S22, Calculate the collaborative penalty factor:

[0020]

[0021] in This is a preset amplification factor used to quantify the combined risk of multiple abnormal indicators.

[0022] In one embodiment of the present invention, a multi-objective optimization cost function is constructed for the data to be migrated, including:

[0023] S41, the cost function for multi-objective optimization is as follows:

[0024]

[0025] in For the cost of load balancing, As a cost of data locality, For the sake of migration efficiency, To balance the costs of storage lifetime;

[0026] S42, Weighting coefficient to The system dynamically adjusts based on the priorities configured by the administrator to balance the optimization goals of load balancing, data reliability, migration efficiency, and hardware lifespan.

[0027] In one embodiment of the present invention, migrating data from high-risk nodes to selected target nodes at a resource-aware dynamic rate includes:

[0028] S51 monitors the cluster’s network bandwidth utilization and disk IOPS in real time. When the network bandwidth utilization is higher than the first threshold or the disk IOPS is higher than the second threshold, the migration rate is reduced.

[0029] S52 increases the migration rate when network bandwidth utilization is below the third threshold and disk IOPS are below the fourth threshold, in order to accelerate data migration during off-peak periods.

[0030] In one embodiment of the present invention, it further includes:

[0031] S6: After all data migration and cleanup is completed on a high-risk node, the administrator is notified to take the node offline for maintenance. When the node is brought back online after maintenance, it is treated as a brand new empty node and added to the cluster to avoid the risk of failure due to hardware aging.

[0032] To achieve the above objectives, a second aspect of the present invention proposes a distributed storage data proactive migration system based on intelligent health prediction, comprising:

[0033] The multi-dimensional health status indicator collection module is used to periodically collect multi-dimensional health status indicators of each node in the distributed storage cluster. The health status indicators include hard disk subsystem indicators, network subsystem indicators, computing and memory subsystem indicators, and software system indicators.

[0034] The multi-timescale dynamic model calculation module is used to calculate the health score of each node based on health status indicators through a multi-timescale dynamic model. The multi-timescale dynamic model includes long-term trend analysis of chronic indicators, short-term fluctuation assessment of acute indicators, and exponential decay penalty processing of event-type indicators. It also introduces a collaborative penalty factor to quantify and correct the composite risk of multiple abnormal indicators.

[0035] The high-risk node marking and migration triggering module is used to mark any node as a high-risk node and trigger an active migration process for all data replicas on that node when the health score of any node continues to fall below a preset active migration threshold.

[0036] The multi-objective optimization cost function construction module is used to construct a multi-objective optimization cost function for the data to be migrated. The cost function includes load balancing cost, data locality cost, migration efficiency cost, and storage lifetime balancing cost. Based on the cost function, the candidate node with the lowest comprehensive cost is selected from the healthy node pool as the migration target.

[0037] The resource-aware dynamic migration and metadata update module is used to migrate data on high-risk nodes to selected target nodes at a resource-aware dynamic rate, and atomically update metadata after successful data migration, and then asynchronously clean up old data copies on high-risk nodes.

[0038] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, for implementing a method for proactive migration of distributed storage data based on intelligent health prediction as described in the first aspect embodiment.

[0039] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for proactive migration of distributed storage data based on intelligent health prediction, as described in the first aspect embodiment.

[0040] The embodiments of the present invention have the following beneficial effects:

[0041] The embodiments of the present invention can effectively avoid rebuild storms in distributed storage systems, eliminate reliability windows, and achieve continuous assurance of data redundancy and efficient utilization of system resources through intelligent health prediction and multi-objective optimization migration strategies. Attached Figure Description

[0042] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0043] Figure 1 A flowchart of a distributed storage data active migration method based on intelligent health prediction provided in an embodiment of the present invention;

[0044] Figure 2 The overall architecture diagram of a distributed storage data proactive migration method based on intelligent health prediction provided by an embodiment of the present invention;

[0045] Figure 3 A detailed calculation flowchart of a health score model based on a distributed storage data active migration method for intelligent health prediction, provided in an embodiment of the present invention;

[0046] Figure 4 A detailed flowchart of a multi-objective optimization target node selection strategy for a distributed storage data active migration method based on intelligent health prediction, provided in an embodiment of the present invention;

[0047] Figure 5 This invention provides a schematic diagram of the overall architecture of a distributed storage data proactive migration system based on intelligent health prediction, as shown in an embodiment of the present invention.

[0048] Figure 6 This is a structural diagram of a distributed storage data active migration system based on intelligent health prediction, provided in an embodiment of the present invention. Detailed Implementation

[0049] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0050] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0051] The following description, with reference to the accompanying drawings, describes a method and system for proactive migration of distributed storage data based on intelligent health prediction, according to an embodiment of the present invention.

[0052] Example 1

[0053] This embodiment provides a method for proactive migration of distributed storage data based on intelligent health prediction. For example... Figure 1 As shown, the method includes the following steps:

[0054] S1 periodically collects multi-dimensional health status indicators for each node in the distributed storage cluster. These health status indicators include indicators for the hard disk subsystem, network subsystem, computing and memory subsystem, and software system.

[0055] Specifically, the system periodically collects multi-dimensional health status indicators from all nodes in the distributed storage cluster to achieve comprehensive monitoring of node operating status. This step is technically implemented based on a distributed monitoring architecture, where each storage node is equipped with a health monitoring agent responsible for periodically polling and collecting local hardware and software status information. The collection frequency can be dynamically configured according to the cluster size and business load, typically set to once every 5 to 15 minutes to ensure a balance between data timeliness and system overhead.

[0056] In this embodiment of the invention, health status indicators cover four dimensions: hard disk subsystem, network subsystem, computing and memory subsystem, and software system. Hard disk subsystem indicators are primarily obtained through the SMART protocol, including reallocation sector count, read / write error rate, instruction timeout count, etc., while also collecting real-time IOPS and read / write latency. Network subsystem indicators are obtained through system network interfaces (such as Linux's ifconfig or ethtool), including packet loss rate, packet error rate, bandwidth utilization, and TCP retransmission rate. Computing and memory subsystem indicators include CPU temperature, core voltage, fan speed, and memory ECC error count, used to assess hardware stability. Software system indicators focus on the number of crashes and restarts of the storage service daemon, as well as hardware-related error information recorded in the kernel logs, to identify potential software-hardware interaction problems.

[0057] Furthermore, the system standardizes the collected raw data. For example, SMART metrics are mapped to a scoring range of 0-100, network packet loss rate is expressed as a percentage, CPU temperature is expressed in degrees Celsius, and memory ECC error counts are based on the number of errors per unit time. The collection and reporting of these metrics must comply with standard monitoring protocols such as SNMP or Prometheus to ensure data integrability and scalability.

[0058] This step is applicable to large-scale distributed storage systems (such as Ceph and HDFS), especially in scenarios with high availability and high reliability requirements, such as core data storage systems in industries like finance, healthcare, and telecommunications. Through periodic data collection, the system can identify potential health risks before nodes experience substantial failures, providing data support for subsequent health scoring and migration decisions.

[0059] The technical advantage of this step lies in providing real-time, comprehensive, and quantifiable node health status input for the entire proactive migration mechanism, which forms the foundation for fault prediction and intelligent decision-making. By collecting multi-dimensional indicators, the system can more accurately identify "soft faults" and "performance degradation," thereby taking preventative measures before faults occur and significantly improving system stability and data reliability.

[0060] Furthermore, S1 includes:

[0061] S11, the hard disk subsystem metrics include the reallocation sector count, read / write error rate, instruction timeout count, seek error rate in the SMART information, as well as real-time hard disk read / write latency and IOPS.

[0062] Specifically, in step S1, the system periodically collects health status indicators of the hard disk subsystem from all nodes in the distributed storage cluster to achieve early identification of potential node failures. Specifically, the hard disk subsystem indicators include the reallocation sector count, read / write error rate, instruction timeout count, seek error rate from SMART information, as well as real-time hard disk read / write latency and IOPS.

[0063] In this embodiment of the invention, a health monitoring agent is deployed on each storage node, periodically polling the SMART attribute of the hard drive through the kernel interface or hardware driver layer to collect the aforementioned metrics. The reallocation sector count reflects the damage and remapping of physical sectors on the hard drive; a higher value indicates more severe hard drive wear. Read / write error rate and seek error rate measure the frequency of anomalies during read / write operations and head positioning, respectively, typically measured in terms of the number of errors per billion operations. The instruction timeout count detects the number of times the hard drive responds to timeouts and is a key indicator for assessing controller or interface performance degradation. Real-time read / write latency and IOPS are obtained through system-level monitoring tools (such as iostat, smartmontools, etc.) to reflect the current performance of the hard drive.

[0064] Furthermore, the normal range for reallocation sector counts is typically less than 100; exceeding this value indicates that the hard drive has entered a degradation phase. The baseline values ​​for read / write error rate and seek error rate are derived from historical cluster data. If the current value exceeds the baseline by a certain percentage (e.g., 50%), an anomaly flag is triggered. The threshold for instruction timeout counts can be set to no more than 5 times per hour to avoid system unreliability due to frequent timeouts. A read / write latency P95 value exceeding 1.5 times the cluster's average latency is also considered abnormal. IOPS is set within a reasonable range based on the hard drive type (e.g., HDD or SSD) and workload; for example, the normal IOPS range for HDDs is 50-150, while for SSDs it is 1000-5000.

[0065] In this embodiment of the invention, this step is applicable to large-scale distributed storage systems (such as Ceph, HDFS, etc.), especially in scenarios with high availability and high reliability requirements, such as core data storage systems in industries like finance, healthcare, and telecommunications. By collecting hard disk subsystem metrics in real time, the system can identify performance degradation or potential failures before nodes completely fail, providing a data foundation for subsequent health scoring and proactive migration decisions.

[0066] This step enables refined monitoring of hard drive health, improving the system's sensitivity to "soft failures." By introducing multi-dimensional indicators and a dynamic data collection mechanism, the system can more accurately assess the health status of nodes, thus providing a scientific basis for subsequent proactive migration and effectively avoiding the risk of rebuilding storms and data loss caused by hard drive failures.

[0067] S12, the network subsystem metrics include packet loss rate, packet error rate, bandwidth utilization, and TCP retransmission rate of the network interface, and the computing and memory subsystem metrics include CPU temperature, CPU core voltage, fan speed, and memory ECC error count.

[0068] Specifically, in the node health assessment system of this invention, the network subsystem and the computing and memory subsystem are key monitoring dimensions. Network subsystem metrics include packet loss rate, packet error rate, bandwidth utilization, and TCP retransmission rate of the network interface. These metrics reflect the stability and performance of the node at the network communication layer. The packet loss rate is usually expressed as a percentage, defined as the ratio of the number of data packets lost per unit time to the total number of data packets sent, i.e. ,in For the number of lost packets, This represents the total number of packets sent. The packet error rate measures the proportion of data packets that encounter checksum errors during transmission; it is calculated as follows: ,in This represents the number of error packets. Bandwidth utilization is used to assess the load status of network links, and is typically expressed as the ratio of currently used bandwidth to maximum bandwidth. The TCP retransmission rate reflects the reliability of the network layer, and it is defined as follows: ,in To the number of retransmitted packets, This represents the number of packets sent.

[0069] The computing and memory subsystem metrics include CPU temperature ( ), CPU core voltage ( ), fan speed ( ) and memory ECC error count ( CPU temperature is typically measured by a hardware sensor, in degrees Celsius (°C), and its threshold range is generally set at... Exceeding this range may affect computational stability. The CPU core voltage must be maintained within the design specifications, such as... Abnormal voltage can lead to calculation errors or hardware damage. Fan speed is used to assess the operating status of the cooling system, measured in RPM (revolutions per minute), and its normal range is typically [range missing]. The memory ECC error count reflects the memory's error correction capability. The higher the value, the worse the memory health. The system needs to set an error threshold. When the count exceeds this threshold, it is considered a potential fault.

[0070] In this embodiment of the invention, the aforementioned metrics are collected periodically by a health monitoring agent deployed on each storage node, typically every 5 minutes, ensuring the system can promptly detect performance degradation trends. These metrics correspond to different time windows and processing logics in the health scoring model. For example, network packet loss rate and TCP retransmission rate are acute metrics with a 24-hour evaluation window, while memory ECC error count is an event-driven metric processed using an exponential decay penalty model. Through comprehensive analysis of these metrics, the system can accurately identify potential risks to nodes, providing a reliable basis for subsequent proactive migration decisions.

[0071] S2, based on health status indicators, calculates the health score of each node through a multi-timescale dynamic model. The multi-timescale dynamic model includes long-term trend analysis of chronic indicators, short-term fluctuation assessment of acute indicators, and exponential decay penalty processing of event-type indicators. It also introduces a collaborative penalty factor to quantify and correct the composite risk of multiple abnormal indicators.

[0072] Specifically, based on the collected health status indicators, the system calculates the health score of each node through a multi-timescale dynamic model. This model comprehensively considers the long-term trend analysis of chronic indicators, the short-term fluctuation assessment of acute indicators, and the exponential decay penalty processing of event-type indicators. It also introduces a collaborative penalty factor to quantify and correct the composite risk of multiple abnormal indicators, thereby achieving an accurate assessment of the node's health status.

[0073] In this embodiment of the invention, the calculation of the health score first relies on the classification of health indicators and the division of time windows. Chronic indicators (such as hard disk reallocation sector count and power-on time) reflect the long-term wear and tear of the equipment, and their evaluation window is typically 90 days. The system uses an S-shaped function to map the original indicator values ​​to initial sub-scores and further calculates their trend (slope) within the time window. If the trend is deteriorating, a trend penalty factor is applied, for example, multiplying the initial score of the chronic indicator by 0.8 to reflect its potential risk. Acute indicators (such as hard disk read / write latency and network packet loss rate) focus on short-term fluctuations, with an evaluation window of 24 hours. The system corrects the sub-scores by calculating the deviation of their P95 value from the cluster baseline and the standard deviation. Event-related indicators (such as process crashes and ECC errors) use an exponential decay penalty model. Whenever an event occurs, a fixed penalty is applied, which decays exponentially over time. ,in The time following the event (in hours) is used to avoid the excessive impact of short-term events on long-term health.

[0074] Furthermore, the system introduces a collaborative penalty factor. ,in A magnification factor (e.g., 0.5) is used to quantify the combined risk of multiple abnormal indicators. The final health score is calculated through weighted fusion, using the following formula:

[0075]

[0076] in The basic weights for each indicator are (e.g., 0.4 for chronic indicators, 0.4 for acute indicators, and 0.2 for event-related indicators). This is the corrected sub-fraction.

[0077] In practical applications, this step operates within the prediction and decision engine, serving as a core module for node health status assessment. Its output directly impacts the triggering of subsequent proactive migration decisions and the selection of target nodes. By introducing multi-timescale analysis and a collaborative penalty mechanism, the system can identify potential faulty nodes earlier and more accurately, thereby taking preventative measures before nodes completely fail, effectively avoiding rebuild storms, and improving the overall reliability and stability of the system.

[0078] Furthermore, S2 includes:

[0079] S21, set the observation time window for chronic indicators to 90 days, the observation time window for acute indicators to 24 hours, and the observation time window for event-type indicators to instantaneous.

[0080] Specifically, in the dynamic health scoring step of this invention, setting the observation time window for chronic indicators to 90 days, the observation time window for acute indicators to 24 hours, and the observation time window for event-type indicators to instantaneous is one of the key technical means to achieve accurate assessment of node health status. This step, through multi-timescale indicator classification and time window division, constructs a comprehensive health model that can reflect long-term node loss, short-term anomalies, and instantaneous events.

[0081] In this embodiment of the invention, chronic indicators (such as hard disk reallocation sector count, power-on time, etc.) typically reflect the long-term wear trend of the device. Therefore, a 90-day observation window is used to capture their slowly changing characteristics. In implementation, the system continuously records and updates the historical data of the indicators through a sliding time window mechanism, calculates the slope of change within the window, and applies a trend penalty factor accordingly. Acute indicators (such as hard disk read / write latency, network packet loss rate, etc.) are used to detect short-term performance degradation of nodes, and their observation window is set to 24 hours to ensure rapid response to sudden anomalies. Event-type indicators (such as process crashes, ECC errors, etc.) are discrete events with instantaneous impact. Therefore, an instantaneous window is used, that is, the event is immediately included in the scoring model when it occurs, and then its impact weight is gradually reduced according to an exponential decay model.

[0082] Furthermore, the window length for chronic indicators is 90 days, for acute indicators it is 24 hours, and for event-type indicators it is instantaneous. In the scoring model, the sub-score correction formula for chronic indicators is: ,in , representing the penalty coefficient for trend deterioration. The correction formula for the acute indicator is: ,in It is related to the degree of deviation of the indicator from the baseline and volatility. The correction formula for event-driven indicators is: ,in The attenuation coefficient is... The time interval after the event occurs.

[0083] This step applies to node health assessment modules in distributed storage systems, particularly in large-scale clusters, to identify potentially faulty nodes and trigger proactive migration. By setting different time windows, the system can differentiate between long-term aging, short-term performance degradation, and sudden events, thus enabling differentiated responses under different failure modes. For example, if the system detects a continuous increase in the hard drive reallocation sector count over 90 days, it can still consider it a potential risk and take proactive measures even if the current value has not reached the threshold.

[0084] This step significantly improves the sensitivity and accuracy of the health scoring model by introducing a multi-timescale indicator evaluation mechanism. Long-term evaluation of chronic indicators helps identify equipment aging trends and avoids misjudgments due to short-term fluctuations; short-term evaluation of acute indicators enhances the response capability to sudden failures; and the instantaneous processing mechanism for event-based indicators ensures immediate response to severe events. The synergistic analysis of these three types of indicators further amplifies the ability to identify complex failures, thereby enabling proactive data migration before substantial node failures occur, effectively avoiding rebuild storms, and improving the overall reliability and stability of the system.

[0085] S22, Calculate the collaborative penalty factor ,in This is a preset amplification factor used to quantify the combined risk of multiple abnormal indicators.

[0086] Specifically, in the dynamic health scoring step of this invention, the collaborative penalty factor... This is a key quantification mechanism used to enhance the system's sensitivity to complex faults. This factor is achieved by introducing an amplification factor. The combined effects of multiple abnormal indicators are nonlinearly amplified to more accurately reflect the degree of deterioration in the overall health status of the node.

[0087] In this embodiment of the invention, the calculation of the collaborative penalty factor is based on the status of all health indicators of a node within a certain evaluation period. Specifically, the system first classifies the collected indicators, including chronic indicators (such as hard disk reallocation sector count), acute indicators (such as network latency P95), and event-related indicators (such as the number of process crashes). Within each category of indicators, the system identifies the number of indicators currently in an abnormal state and calculates their proportion in the total number of indicators. This proportion is multiplied by a preset amplification factor. Adding the baseline value of 1 yields the collaborative penalty factor. Amplification factor The value range is usually in The size is configured by the system administrator based on the cluster's fault tolerance and risk preference.

[0088] Furthermore, the collaborative penalty factor The calculation depends on two core parameters: And the number of indicators that are simultaneously abnormal / the total number of indicators. Among them, The settings need to be tailored to the specific application scenario, such as in financial or medical systems where reliability requirements are extremely high. It can be set to 0.5 or higher to enhance the response to anomalies in multiple metrics; however, in performance-sensitive real-time business systems, The penalty can be appropriately reduced to avoid excessive punishment leading to misjudgment. Meanwhile, the determination of abnormal indicators should be based on preset baseline thresholds and statistical models, such as P95 deviation from the threshold, standard deviation exceeding the set range, or the exponential decay penalty score of event-type indicators reaching the warning value.

[0089] Specifically, the collaborative penalty factor It is mainly used in the weighted fusion stage of the health scoring model. When multiple health indicators of a node show abnormalities simultaneously, This will amplify the decline in its overall health score, thus triggering the active migration mechanism earlier. For example, in a node containing 8 health indicators, if 3 indicators are abnormal simultaneously, and ,but This factor will be weighted and fused with the modified sub-scores of various indicators to ultimately generate the node's... This serves as the basis for migration decisions.

[0090] In this embodiment of the invention, the collaborative penalty factor The introduction of this factor significantly improves the system's ability to identify complex faults. Compared to traditional single-indicator threshold judgment methods, this factor can more comprehensively reflect the potential risks of nodes, avoiding misjudgments or omissions caused by fluctuations in a single indicator. Furthermore, by amplifying the factor... With its flexible configuration, the system can adapt to risk preferences under different business scenarios, achieving adaptive optimization of health scores. This step, as a core component of the health scoring model, provides a scientific and reliable basis for subsequent proactive migration decisions and is a key technical support for the "fault prevention" achieved by this invention.

[0091] S3. When the health score of any node continues to fall below the preset active migration threshold, the node is marked as a high-risk node, and an active migration process for all data replicas on that node is triggered.

[0092] Specifically, when the health score of any node continuously falls below a preset proactive migration threshold, the system marks that node as a high-risk node and triggers a proactive migration process for all data replicas on that node. This step is a key component in realizing the fault prevention mechanism in this invention. Its technical implementation is based on the output of a dynamic health score model, combined with a preset migration threshold and time window strategy, to ensure the accuracy and stability of migration decisions.

[0093] In this embodiment of the invention, the health score is calculated using a dynamic scoring model with multiple time scales and trend analysis. The health status of a node is divided into three categories: chronic, acute, and event-related indicators, and sub-scores are calculated for each category. When a node's final health score... When the score remains below the preset active migration threshold (e.g., 70 points), the system will initiate the migration process. To prevent misjudgment due to short-term fluctuations, this judgment must remain valid for a certain time window (e.g., 24 hours) before migration can be triggered.

[0094] Furthermore, the calculation of the health score relies on multi-dimensional indicator adjustments and collaborative penalty factors. Its value is calculated by multiplying the ratio of the number of simultaneously abnormal indicators to the total number of indicators by an amplification factor. Therefore, the migration threshold needs to be set in conjunction with the system redundancy level, business load tolerance, and hardware health baseline, and is usually dynamically adjusted through operation and maintenance configuration or adaptive algorithms. Furthermore, triggering the migration process also needs to consider the number of replicas, data distribution strategies, and migration priority rules to ensure the efficiency and controllability of the migration process.

[0095] This step is applicable to scenarios in distributed storage systems where the health of nodes is gradually deteriorating but not yet completely failing, such as bad sectors on hard drives, continuously increasing network latency, or frequent process crashes. By identifying potentially faulty nodes in advance, the system migrates data replicas to healthy nodes, thereby avoiding rebuild storms and reliability window issues caused by sudden node failures.

[0096] The technical advantage of this step is that, through a proactive migration mechanism, the system can adjust data redundancy before node failure, ensuring high data availability and system stability. At the same time, this mechanism significantly reduces the rebuilding overhead caused by sudden failures, improving overall operational efficiency and resource utilization.

[0097] S4. Construct a multi-objective optimization cost function for the data to be migrated. The cost function includes load balancing cost, data locality cost, migration efficiency cost, and storage lifetime balancing cost. Based on the cost function, select the candidate node with the lowest overall cost from the healthy node pool as the migration target.

[0098] Specifically, the system constructs a multi-objective optimization cost function for the data to be migrated, and selects the candidate node with the lowest overall cost from the healthy node pool as the migration target based on this function. This step is one of the core innovations of this invention, aiming to achieve optimal resource allocation during the data migration process and improve the overall reliability, load balancing, and migration efficiency of the system.

[0099] In this embodiment of the invention, the cost function consists of four key dimensions: load balancing cost, data locality cost, migration efficiency cost, and storage lifetime balancing cost. Each dimension corresponds to a different system performance metric, and the suitability of candidate nodes is comprehensively evaluated through a weighted summation. Specifically, the system evaluates each candidate node... Calculate its overall cost The formula is:

[0100]

[0101] in, to These are configurable weighting coefficients used to reflect the priority of different cost dimensions in migration decisions. For example, during peak business periods, the weighting can be appropriately increased. Prioritizing migration efficiency; and in clusters with severely aging hardware, improving... To extend the overall storage lifespan.

[0102] Furthermore, Typically, the load of a candidate node is calculated based on metrics such as CPU utilization, memory utilization, disk IOPS, and network bandwidth utilization, after normalization, to determine the degree to which its load deviates from the cluster average. Then consider the physical location relationship between the candidate node and the source node, such as whether they are located in the same rack or the same server, to ensure that data replicas are distributed across fault domains. Based on the network latency and available bandwidth between the source node and the candidate node, quantization is usually performed using RTT (Round-Trip Time) and the reciprocal of the available bandwidth. The health score of the candidate node's hard drive reflects its remaining lifespan and reliability; the lower the health score, the higher the cost.

[0103] This step is performed by the target node selector in the resource optimization scheduler. When the system identifies a high-risk node, this module filters nodes from the healthy node pool that meet hard conditions such as capacity and availability, and calculates the aforementioned cost function for each node. Finally, the system selects the N nodes (N being the number of replicas) with the lowest overall cost as migration targets, ensuring that the data still meets redundancy requirements after migration.

[0104] The technical advantage of this step lies in the fact that, through a multi-objective optimization model, the system can balance multiple key performance indicators such as load balancing, data reliability, migration efficiency, and hardware lifespan during the migration process, avoiding the introduction of new performance bottlenecks or reliability risks due to migration operations. Simultaneously, through a configurable weighting mechanism, the system possesses strong adaptive capabilities, dynamically adjusting migration strategies according to different business scenarios, thereby achieving efficient, intelligent, and reliable proactive data migration.

[0105] Furthermore, S4 includes:

[0106] S41, the cost function for multi-objective optimization is as follows:

[0107]

[0108] in For the cost of load balancing, As a cost of data locality, For the sake of migration efficiency, This comes at the cost of balancing storage lifetime.

[0109] Specifically, in the proactive migration decision-making process of this invention, target node selection based on multi-objective optimization is a key step in achieving efficient and reliable data migration. This step involves constructing a comprehensive cost function to quantitatively evaluate candidate nodes, thereby selecting the optimal migration target node. The comprehensive cost function is:

[0110]

[0111] In this embodiment of the invention, the cost function consists of four dimensions: load balancing cost. The cost of data locality Migration efficiency cost Balance the cost with storage lifetime .in, By evaluating metrics such as CPU utilization, memory usage, disk IOPS, and network bandwidth utilization of candidate nodes, the deviation of their current load status from the system average load is calculated to ensure that no new hotspot issues will be triggered after migration. Based on the distribution of fault domains (such as racks, servers, and network switches), nodes that are not in the same fault domain as the source node replica are prioritized to improve data fault tolerance.

[0112] Furthermore, migration efficiency costs The expected migration time is calculated using a weighted average method by measuring network latency and bandwidth utilization between the source node and candidate nodes. Storage lifetime leveling cost. This is one of the innovative aspects of the present invention. Its calculation is based on the health score of the candidate node's hard disk, and prioritizes the migration of data to nodes with high health and short usage time, thereby realizing the "metabolism" of hardware resources and extending the overall lifespan of the cluster.

[0113] In this embodiment of the invention, the weighting coefficient to The system administrator can configure this according to actual business needs; for example, the system speed can be appropriately increased during peak business periods. Prioritize migration efficiency, while increasing [efficiency] during the maintenance window. This optimizes hardware lifetime distribution. The adjustability of these parameters enhances the system's adaptability.

[0114] In practical applications, this step runs within the resource optimization scheduler module. Its inputs are the replica information of the data blocks to be migrated and the real-time status of candidate nodes. The output is a list of target nodes for each data block. Through multi-objective optimization, the system can achieve an optimal balance among multiple dimensions such as load balancing, data reliability, migration efficiency, and hardware lifespan, thereby avoiding system performance degradation or reliability reduction caused by inappropriate migration strategies.

[0115] In summary, this step, by introducing a multi-objective cost function, enables the intelligent selection of data migration target nodes. It is one of the core supporting technologies for the transformation of this invention from "passive response" to "active prevention," and has significant engineering practical value and system optimization effect.

[0116] S42, Weighting coefficient to The system dynamically adjusts based on the priorities configured by the administrator to balance the optimization goals of load balancing, data reliability, migration efficiency, and hardware lifespan.

[0117] Specifically, in some implementations, the weighting coefficients to The selection process dynamically adjusts based on the administrator's configured priorities to balance the optimization goals of load balancing, data reliability, migration efficiency, and hardware lifespan. This step is a core component of the target node selection strategy in this invention. Its technical implementation is based on a multi-objective optimization model, which integrates multiple cost indicators into a comprehensive cost function through weighted summation, thereby achieving quantitative evaluation and ranking of candidate nodes.

[0118] Specifically, when executing proactive migration decisions, the system first selects candidate nodes from the healthy node pool that meet hard criteria such as capacity and availability. Then, for each candidate node... Calculate the cost in four dimensions: load balancing cost The cost of data locality Migration efficiency cost Balance the cost with storage lifetime These cost metrics are typically normalized to ensure comparability across different dimensions. Finally, the system calculates the comprehensive cost of candidate nodes using the following formula:

[0119]

[0120] in, to These are configurable weighting coefficients, each corresponding to the priority of one of the four cost terms. The values ​​of these weighting coefficients typically range from [value range missing]. And satisfy This is to ensure the rationality and interpretability of the cost function. For example, in scenarios where business performance is prioritized, and It can be set to a higher value (such as 0.35 and 0.30), and in scenarios where hardware lifespan optimization is prioritized, It can be set to 0.40 or higher.

[0121] Furthermore, the weighting coefficients can be adjusted through the system management interface or configuration files, supporting dynamic configuration based on various dimensions such as cluster load status, service type, and hardware aging. For example, when the overall cluster load is high, the weighting coefficients can be appropriately reduced. To avoid further intensifying resource competition; when nodes are severely aging, it can improve... This is to guide data to migrate to healthier nodes.

[0122] This step is widely applicable to the operation and maintenance management of distributed storage systems, especially in large-scale clusters. By flexibly configuring weight coefficients, the system can achieve optimal migration decisions under different business needs and hardware conditions. Its technical advantage lies in the fact that by introducing a configurable weight mechanism, the system can adaptively balance multiple optimization objectives, avoiding suboptimal decisions caused by prioritizing a single objective, thereby improving overall migration efficiency, data reliability, and hardware resource utilization.

[0123] S5 migrates data from high-risk nodes to selected target nodes at a resource-aware dynamic rate, atomically updates metadata after successful data migration, and then asynchronously cleans up old data copies on high-risk nodes.

[0124] Specifically, in some implementations, resource-aware controlled-rate migration is one of the key execution steps of this invention. Its core lies in migrating data replicas from high-risk nodes to selected target nodes at a dynamically adjusted rate, thereby ensuring migration efficiency while avoiding significant interference with the normal business performance of the cluster. The technical implementation of this step is based on a real-time perception and feedback control mechanism for the cluster resource status.

[0125] Specifically, the migration controller constructs a resource load model by periodically collecting key performance indicators such as network bandwidth utilization and disk IOPS from each node in the cluster. During data migration, the controller dynamically adjusts the migration rate based on the current resource load status. For example, when network bandwidth utilization exceeds a preset first threshold (e.g., 80%) or disk IOPS exceeds the upper limit for normal business operation (e.g., 8000 IOPS), the migration rate will be automatically reduced to prevent resource contention from increasing business latency. Conversely, when resource utilization is below a second threshold (e.g., 30%), the migration rate can be increased to accelerate data migration and shorten the risk window for high-risk nodes.

[0126] Furthermore, the migration process employs a streaming mechanism, dividing large files into multiple data blocks and transmitting them block by block in a pipeline manner. This approach not only improves transmission efficiency but also enhances the system's fault tolerance. Even if some data blocks fail to be transmitted, there is no need to retransmit the entire file, thereby reducing performance fluctuations caused by migration interruptions.

[0127] In this embodiment of the invention, the migration rate adjustment is typically based on a sliding window algorithm, using a 10-second evaluation period to calculate the weighted average of the current resource load and generate rate control commands accordingly. Furthermore, in the migration priority strategy, the system assigns higher migration priority to data with only a single replica or critical blocks in the EC (Extremely Concentrated Area) to ensure that data redundancy always meets system reliability requirements.

[0128] This step plays a crucial role in the overall technical solution, and its resource awareness mechanism ensures the smoothness and controllability of the migration process, avoiding the traditional "rebuild storm" problem. At the same time, it provides a stable data migration foundation for the atomic updates of subsequent metadata and the asynchronous cleanup of old data, thereby ensuring the high availability and data consistency of the system during the migration process.

[0129] The distributed storage data proactive migration method based on intelligent health prediction in this invention can identify storage node failure risks in advance and proactively migrate data, effectively avoiding rebuild storms and reliability windows, and improving system stability and data security.

[0130] Furthermore, S5 includes:

[0131] S51 monitors the cluster's network bandwidth utilization and disk IOPS in real time. When the network bandwidth utilization exceeds the first threshold or the disk IOPS exceeds the second threshold, the migration rate is reduced.

[0132] Specifically, in the resource-aware controlled-rate migration step, the system dynamically adjusts the data migration rate by monitoring the cluster's network bandwidth utilization and disk IOPS in real time. This ensures migration efficiency while avoiding significant impact on normal business performance. This step is one of the core implementations of the smooth migration mechanism in this invention, ensuring that the migration process automatically slows down when resource load is high and accelerates when load is low, thus achieving resource coordination between business and migration tasks.

[0133] In this embodiment of the invention, the system periodically collects the bandwidth utilization rate (in percentage, denoted as ) of the network interface through health monitoring agents deployed on each node. ) and disk input / output operations per second (IOPS, denoted as These metrics are reported to the prediction and decision engine via the monitoring agent, and are analyzed and responded to in real time by the migration rate controller in the resource optimization scheduler.

[0134] The specific operation method is as follows: The migration rate controller is set with two configurable thresholds, denoted as follows: and When detected or When this happens, the system will automatically reduce the data transfer rate of the current migration task. For example, a sliding window rate control algorithm (Token Bucket or Leaky Bucket) can be used to reduce the migration rate from the maximum value. Gradually reduce to In order to alleviate competition for resources.

[0135] Furthermore, the system can employ a tiered rate control strategy based on the severity of the current resource load. For example, when bandwidth utilization exceeds... But not exceeding At that time, the migration rate decreased to When exceeding At that time, it was further reduced to Similarly, the same tiered strategy can be used to control disk IOPS.

[0136] In practical applications, this step is particularly suitable for scenarios where large-scale distributed storage systems migrate data during peak business periods. For example, during peak hours in financial trading systems or online video platforms, the system can reduce the migration rate to avoid network congestion or disk I / O bottlenecks caused by migration tasks, thereby maintaining high throughput and low latency for the business.

[0137] This step effectively mitigates resource conflicts between migration tasks and normal business operations, reducing the impact of the migration process on the overall cluster performance. Through dynamic rate control, the system can automatically "give way" when resources are scarce and "accelerate" when resources are idle, thereby achieving smooth execution of migration tasks and improving system stability and availability.

[0138] S52 increases the migration rate when network bandwidth utilization is below the third threshold and disk IOPS are below the fourth threshold, in order to accelerate data migration during off-peak periods.

[0139] Specifically, in the resource-aware controlled rate migration step, when the network bandwidth utilization is lower than the third threshold (denoted as...), And disk IOPS are below the fourth threshold (denoted as ) When data migration occurs, the migration controller will automatically increase the data migration rate to accelerate data migration during periods of low business activity. This step is a key component in realizing the "smooth migration" and "resource awareness" strategies in this invention, and its technical implementation is based on a real-time monitoring and dynamic adjustment mechanism for cluster resources.

[0140] In this embodiment of the invention, the adjustment of the migration rate relies on a quantitative assessment of network and disk resources. Specifically, the system periodically collects the bandwidth utilization rate of the network interface (in percentage) and the disk IOPS (in times per second), and compares them with a preset... and The system compares both data points. If both are below their respective thresholds, the system determines that the current period is a low point in business activity and has a high degree of resource idleness, thereby triggering a migration rate increase mechanism.

[0141] Furthermore, the increase in migration rate is dynamically calculated by a rate control algorithm. This algorithm can be based on linear interpolation or an exponential growth model, adjusting the upper limit of the migration rate according to the gap between the current resource utilization and a threshold. For example, when bandwidth utilization is lower than... And IOPS is lower than At this time, the system can increase the migration rate limit from the default value. Upgraded to ,in Usually 1.5 to 2 times that of idle resources, to make full use of idle resources.

[0142] In practical applications, this step is typically deployed in the background scheduling module of a distributed storage system, complementing the rate suppression strategy during peak business traffic periods. Its purpose is to maximize migration efficiency when resources are idle, shorten the migration cycle of data on high-risk nodes, thereby restoring system redundancy more quickly and reducing the risk window for data loss. Simultaneously, by avoiding large-scale data migration during peak business periods, it effectively mitigates the impact of a "rebuild storm" on normal business operations.

[0143] In summary, this step, by dynamically sensing the status of network and disk resources and combining it with rate control strategies, accelerates data migration during periods of low business activity. It is one of the core technologies of this invention for achieving proactive migration in a subtle and seamless manner, and has significant effects on optimizing resource utilization and improving system stability.

[0144] S6: After all data migration and cleanup is completed on a high-risk node, the administrator is notified to take the node offline for maintenance. When the node is brought back online after maintenance, it is treated as a brand new empty node and added to the cluster to avoid the risk of failure due to hardware aging.

[0145] Specifically, in some implementations, once all data on a high-risk node has been migrated and cleaned up, the system will notify the administrator to perform offline maintenance on that node through an operation and maintenance linkage mechanism. This step is a key link in realizing planned maintenance and cluster self-healing capabilities in this invention, and its technical implementation is based on the integration of the state synchronization mechanism of the distributed storage system with the operation and maintenance interface.

[0146] The specific operation is as follows: First, the system continuously reports the node status through the health monitoring agent. After confirming that all data replicas on the node have been successfully migrated and verified, the prediction and decision engine sends a migration completion signal to the metadata manager. Upon receiving this signal, the metadata manager performs an atomic metadata update operation to ensure that the data redundancy always meets the system's preset replica count or erasure coding configuration requirements. Subsequently, the system sends an alarm notification to the administrator through a control plane interface (such as a REST API or a dedicated operation and maintenance channel), indicating that the node is in a "data clearing" state and can be safely taken offline for hardware maintenance.

[0147] After a node is repaired, the system treats it as a "brand new empty node" and rejoins it to the cluster. This operation is implemented through cluster management interfaces (such as Kubernetes API, Ceph MON, HDFS NameNode, etc.). The node is assigned an initial state of "empty node" during registration and re-participates in the health scoring and data scheduling process. This process avoids the risk of recurring failures due to hardware aging and prevents potential faulty data remaining in the old node from causing secondary impacts on the cluster.

[0148] In this embodiment of the invention, this step relies on a mechanism to confirm the migration completion status, including a data integrity threshold (e.g., an MD5 checksum pass rate of 100%), atomicity guarantees for metadata updates (e.g., using two-phase commit or the Raft protocol), and a latency tolerance time for maintenance notifications (e.g., a maximum waiting time of no more than 5 minutes). Furthermore, the initial configuration when a node comes back online must meet the cluster node admission criteria, such as a "healthy" SMART status for the hard drive, a connectivity test for the network interface, and a normal startup of the storage service daemon.

[0149] This step is widely applicable in practical applications to large-scale distributed storage systems, such as cloud storage platforms, enterprise-grade NAS, and HDFS clusters. Its technical value lies in achieving "controlled exit and controlled return of failed nodes," thereby improving cluster availability and operational efficiency, while reducing the probability of recurrence of failures due to hardware aging and extending the overall cluster lifecycle.

[0150] The distributed storage data proactive migration method based on intelligent health prediction in this invention further reduces the probability of repeated failures caused by hardware aging by taking high-risk nodes offline for maintenance after data migration and cleanup, and adding them to the cluster as brand new empty nodes when they come back online. This improves the long-term reliability and maintenance efficiency of the cluster.

[0151] Example 2

[0152] This invention relates to a method for proactive migration of distributed storage data based on intelligent health prediction, such as... Figure 1 As shown, the method includes the following steps:

[0153] 1. Collection and monitoring of node health indicators.

[0154] Specifically, the system periodically collects multi-dimensional health status indicators from all storage nodes in the cluster. These health status indicators include:

[0155] Hard disk subsystem metrics: hard disk reallocation sector count, read / write error rate, instruction timeout count, seek error rate based on SMART information; and real-time hard disk read / write latency and IOPS.

[0156] Network subsystem metrics: packet loss rate, packet error rate, bandwidth utilization, and TCP retransmission rate of network interfaces.

[0157] Computing and memory subsystem metrics: CPU temperature, CPU core voltage, fan speed, and memory ECC error count.

[0158] Software system metrics: number of crashes and restarts of the storage service daemon, and hardware-related error information in the kernel logs.

[0159] 2. Dynamic health score based on multiple time scales and trend analysis.

[0160] This step is one of the core innovations of this invention. It is not a simple weighted average, but a comprehensive evaluation model that considers both the long-term trend of the indicator and short-term sudden changes.

[0161] Specifically, the steps include:

[0162] (1) Indicator classification and time window division:

[0163] Chronic indicators: These reflect long-term wear and tear on equipment, such as hard drive reallocation sector count and power-on time. The observation window is a long period (e.g., 90 days).

[0164] Acute indicators: These reflect short-term equipment anomalies, such as hard drive read / write latency and network packet loss rate. The observation window is a short period (e.g., 24 hours).

[0165] Event-based metrics: Reflect discrete, critical events, such as process crashes or ECC errors. The observation window is instantaneous.

[0166] (2) Calculation of index sub-scores:

[0167] Chronic metrics: These use a sigmoid function to map raw values ​​to scores and calculate their trend (slope) over a period of time. If the trend is deteriorating, a penalty factor is applied to the current score. For example, a hard drive reallocation sector count that is slowly but steadily increasing will have its score reduced due to the trend penalty, even if the current absolute value is not high.

[0168] Acute indicators: Calculate the percentile (e.g., P95) within the short-term window and compare it to the cluster baseline. Also calculate its volatility (standard deviation). The further it deviates from the baseline, the greater the volatility, and the lower the score.

[0169] Event-based metrics: Employ an exponentially decaying penalty model. A fixed penalty is applied each time an event occurs, and this penalty decays exponentially over time. Frequent events within a short period will lead to an accumulation of penalties and a sharp drop in the score.

[0170] (3) Integration of dynamic weights and total score:

[0171] Set a basic weight for each type of indicator.

[0172] Introducing a "synergistic penalty factor": When multiple different types of indicators for a node simultaneously show abnormalities, its risk is considered significantly higher than that of a single abnormal indicator. A synergistic coefficient is calculated. (α is the amplification factor).

[0173] Final Health Score Synergistic effects amplify the overall score decline, making the model more sensitive to complex faults.

[0174] In this embodiment of the invention, the health assessment of node A is assumed to be as follows:

[0175] Chronic indicator: The redistribution sector count was 50 (60 points according to the S-shaped function), but it increased from 10 to 50 in the past week (trend deterioration, subject to a 0.8 penalty), resulting in a final score of 48 points.

[0176] Acute indicator: Network latency P95 exceeded the baseline by 50% (score 67 points) and fluctuated greatly (penalty of 0.9 was imposed), resulting in a final score of 60.3 points.

[0177] Event-driven metric: A process crash occurred within 24 hours (initial penalty -30, now reduced to a 24-hour decay period as 12 hours have passed). ), which is mapped to a score of 81.8.

[0178] Collaborative penalty: When all three indicators are abnormal, the collaborative coefficient is expressed as:

[0179] .

[0180] Basic weights: C:0.4, A:0.4, E:0.2.

[0181] Final health score = 1.1875 × (0.4×48 + 0.4×60.3 + 0.2×81.8)≈ 1.1875×58.36≈69.3 points.

[0182] 3. Proactive migration decision trigger.

[0183] The system presets an "active migration threshold". When the health score of any node remains below this threshold for a certain period of time (to prevent false alarms due to jitter), the system marks the node as a "high-risk node" and immediately triggers an active migration process for all data replicas on that node.

[0184] In this embodiment of the invention, node A has a health score of 69.3. If the threshold is 70, then the node is triggered to actively migrate.

[0185] 4. Target node selection based on multi-objective optimization.

[0186] This step is another core innovation of the present invention. It selects an optimal "new home" from the pool of healthy nodes for each piece of data to be migrated.

[0187] Specifically, this step includes:

[0188] (1) Establish the cost objective function:

[0189] Load balancing costs: Evaluate the utilization of CPU, memory, disk I / O, and network bandwidth of candidate nodes to ensure that the cluster load remains balanced after migration.

[0190] Data locality cost: Based on fault domain awareness, nodes located on different racks and servers than other data replicas are prioritized to improve data reliability.

[0191] Migration efficiency cost: Evaluate the network bandwidth and latency between the source node and candidate nodes, and prioritize nodes with better network paths to accelerate migration.

[0192] Storage lifespan balancing cost: Introducing the health score of the candidate node's own hard drive, tending to migrate data to younger, healthier hard drives, achieving "metabolism" of cluster hardware.

[0193] (2) Multi-objective decision-making:

[0194] First, filter out nodes that do not meet the hard requirements, such as insufficient capacity or being offline.

[0195] A weighted summation method is used to calculate a comprehensive cost for each candidate node j:

[0196]

[0197] in, For the cost of load balancing, As a cost of data locality, For the sake of migration efficiency, To balance the cost of storage lifetime based on the health of the candidate nodes themselves. to These are configurable weighting coefficients that reflect the priority of different dimensions.

[0198] Select the N nodes (where N is the number of replicas) with the lowest overall cost as the migration targets.

[0199] In this embodiment of the invention, when selecting a target node, it is assumed that there are two candidate nodes:

[0200] Node X: Load cost: 80; Locality cost: 20; Efficiency cost: 30; Lifetime cost: 40. Total cost .

[0201] Node Y: Load cost: 60; Locality cost: 50; Efficiency cost: 50; Lifetime cost: 90. Overall cost .

[0202] Therefore, the system will select node X, which has a lower overall cost.

[0203] 5. Controlled rate migration of resource awareness.

[0204] The migration controller does not move all data at once, but includes:

[0205] Rate control: Monitor cluster load in real time and dynamically adjust migration rate. Reduce rate during peak business hours and increase rate during off-peak hours.

[0206] Priority scheduling: Set higher migration priority for data that has only a single replica or critical blocks in EC.

[0207] Streaming: Large files are divided into chunks and transmitted in a pipeline manner to improve efficiency.

[0208] 6. Atomized metadata switching and old data cleanup.

[0209] Once a piece of data is successfully written to disk and verified on the target node, the metadata server atomically updates the data distribution mapping, adding the new node to ensure that data redundancy always meets the requirements.

[0210] After the metadata update is successful, the system asynchronously sends instructions to the "high-risk node" to delete the migrated data copy and free up space.

[0211] 7. Node offline and operation and maintenance coordination.

[0212] Once all data on the high-risk node has been migrated, the system will issue an alert to the administrator that the node can be safely taken offline for maintenance.

[0213] Once a node is repaired and brought back online, it is treated as a brand new empty node and added to the cluster.

[0214] Furthermore, such as Figure 2 As shown, the specific implementation steps of this embodiment of the invention include:

[0215] First, starting with step S10, the system continues to run.

[0216] S20 periodically collects health indicators from all nodes.

[0217] S30 calls the health scoring model to calculate a health score for each node.

[0218] Specifically, such as Figure 3 As shown, step S30 includes:

[0219] S301 retrieves all raw indicator data for a node.

[0220] S302 classifies the indicators (chronic C, acute A, event-related E) and places them into different time windows.

[0221] S303, calculate the initial sub-scores for the three categories of indicators respectively.

[0222] S304 calculates the trend penalty for chronic indicators and corrects their sub-scores.

[0223] S305 calculates the fluctuation penalty for acute indicators and corrects their sub-scores.

[0224] S306 calculates the exponential decay penalty for event-type indicators and corrects their sub-scores.

[0225] S307, calculate the co-punishment factor K among all abnormal indicators.

[0226] S308, based on the basic weights and the synergistic factor K, is used to weight and fuse the data to obtain the final health score.

[0227] S309, output the health score.

[0228] S40: Determine if any node's health score is below the threshold. If not, return to continue monitoring; if yes, proceed to S50.

[0229] S50 marks the node as a "high-risk node".

[0230] S60: For each piece of data on this node, execute the target node selection process of multi-objective optimization to select the best target node.

[0231] Specifically, such as Figure 4 As shown, step S60 includes:

[0232] S601, Input the data block to be migrated and its copy location information.

[0233] S602 selects all healthy, sufficiently large candidate nodes from the cluster.

[0234] S603 calculates four costs for each candidate node: load balancing cost, data locality cost, migration efficiency cost, and storage lifetime cost.

[0235] S604, based on the weight configured by the administrator ( ~ ), calculate the comprehensive cost for each candidate node.

[0236] S605 sorts all candidate nodes from low to high based on their overall cost.

[0237] S606: Select the node with the lowest overall cost as the target node for this data block.

[0238] S607, Output the list of target nodes.

[0239] S70 initiates resource-aware controlled data migration.

[0240] S80: Determine whether a data migration was successful. If the migration was successful, proceed to step S90.

[0241] S90 atomically updates metadata and adds new copies.

[0242] S100 asynchronously cleans up old data copies on high-risk nodes.

[0243] S110: Determine whether all data on the high-risk node has been migrated and cleaned up. If it has been migrated and cleaned up, proceed to step S120.

[0244] S120 Notifies the administrator that the node can be safely taken offline, and the process ends.

[0245] Example 3

[0246] This invention relates to a distributed storage data proactive migration system based on intelligent health prediction, the system specifically comprising:

[0247] Health monitoring agent: Deployed on each storage node, responsible for collecting and reporting metrics.

[0248] Prediction and Decision Engine: This is the core control unit, which includes a health scoring module and a migration triggering module.

[0249] Resource optimization scheduler: includes target node selector and migration rate controller.

[0250] Metadata Manager: Responsible for maintaining and updating data location information.

[0251] Data migration executor: responsible for efficiently and reliably transferring data blocks between nodes.

[0252] Furthermore, such as Figure 5 As shown, in this embodiment of the invention, the system mainly includes:

[0253] Monitoring and Decision Center: This is the brain of the system. It contains a node health collector, responsible for receiving metric data from all storage nodes; and a fault prediction and decision engine, which has a built-in health scoring model and migration triggering logic.

[0254] Distributed storage cluster: Consists of a large number of storage nodes. Each node runs a health monitoring agent, which collects local metrics and reports them.

[0255] Control Flow and Data Flow: The decision center directs the cluster's actions through control commands (such as initiating migration or updating metadata). Storage nodes then perform the actual data migration through background data migration traffic. As shown in the diagram, when the decision engine determines that node B is "high-risk," it will instruct node B to migrate its data to the pre-selected healthy node C.

[0256] Example 4

[0257] This invention also provides a distributed storage data proactive migration system based on intelligent health prediction, such as... Figure 6 As shown, a distributed storage data proactive migration system 10 based on intelligent health prediction includes:

[0258] The multi-dimensional health status indicator collection module 100 is used to periodically collect multi-dimensional health status indicators of each node in the distributed storage cluster. The health status indicators include hard disk subsystem indicators, network subsystem indicators, computing and memory subsystem indicators, and software system indicators.

[0259] The multi-timescale dynamic model calculation module 200 is used to calculate the health score of each node based on health status indicators through a multi-timescale dynamic model. The multi-timescale dynamic model includes long-term trend analysis of chronic indicators, short-term fluctuation assessment of acute indicators, and exponential decay penalty processing of event-type indicators. It also introduces a collaborative penalty factor to quantify and correct the composite risk of multiple abnormal indicators.

[0260] The high-risk node marking and migration triggering module 300 is used to mark any node as a high-risk node and trigger an active migration process for all data replicas on that node when the health score of any node is continuously lower than a preset active migration threshold.

[0261] The multi-objective optimization cost function construction module 400 is used to construct a multi-objective optimization cost function for the data to be migrated. The cost function includes load balancing cost, data locality cost, migration efficiency cost, and storage lifetime balancing cost. Based on the cost function, the candidate node with the lowest comprehensive cost is selected from the healthy node pool as the migration target.

[0262] The resource-aware dynamic migration and metadata update module 500 is used to migrate data on high-risk nodes to selected target nodes at a resource-aware dynamic rate, and atomically update metadata after successful data migration, and then asynchronously clean up old data copies on high-risk nodes.

[0263] Example 5

[0264] To implement the methods of the above embodiments, the present invention also provides a computer device, which includes a memory and a processor; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, so as to implement the various steps of the above methods.

[0265] Example 6

[0266] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method as described in the foregoing embodiments.

[0267] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0268] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0269] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for active migration of distributed storage data based on intelligent health prediction, characterized in that, Includes the following steps: S1 periodically collects multi-dimensional health status indicators of each node in the distributed storage cluster. The health status indicators include hard disk subsystem indicators, network subsystem indicators, computing and memory subsystem indicators, and software system indicators. S2, based on health status indicators, calculates health scores for each node using a multi-timescale dynamic model. The observation window for chronic indicators is set to 90 days, for acute indicators to 24 hours, and for event-related indicators to instantaneous. The multi-timescale dynamic model includes long-term trend analysis of chronic indicators, short-term fluctuation assessment of acute indicators, and exponential decay penalty processing for event-related indicators. A synergistic penalty factor is introduced to quantify and correct the composite risk of multiple abnormal indicators. The formula for calculating the synergistic penalty factor is as follows: wherein is a preset amplification factor for quantifying the composite risk of the plurality of abnormal indicators; S3, when the health score of any node continues to be lower than the preset active migration threshold, the node is marked as a high-risk node and an active migration process for all data replicas on the node is triggered; S4. Construct a multi-objective optimization cost function for the data to be migrated. The cost function includes load balancing cost, data locality cost, migration efficiency cost, and storage lifetime balancing cost. Based on the cost function, select the candidate node with the lowest comprehensive cost from the healthy node pool as the migration target. S5 migrates data from high-risk nodes to selected target nodes at a resource-aware dynamic rate, atomically updates metadata after successful data migration, and then asynchronously cleans up old data copies on high-risk nodes.

2. The method according to claim 1, characterized in that, Periodically collect multi-dimensional health status indicators for each node in the distributed storage cluster, including: S11, the hard disk subsystem metrics include the reallocation sector count, read / write error rate, instruction timeout count, seek error rate in the SMART information, as well as real-time hard disk read / write latency and IOPS; S12, the network subsystem metrics include packet loss rate, packet error rate, bandwidth utilization, and TCP retransmission rate of the network interface, and the computing and memory subsystem metrics include CPU temperature, CPU core voltage, fan speed, and memory ECC error count.

3. The method according to claim 1, characterized in that, Construct a multi-objective optimization cost function for the data to be migrated, including: S41, the cost function for multi-objective optimization is as follows: wherein is a load balancing cost, is a data locality cost, is a migration efficiency cost, is a storage lifetime balancing cost; S42, Weighting coefficient to The system dynamically adjusts based on the priorities configured by the administrator to balance the optimization goals of load balancing, data reliability, migration efficiency, and hardware lifespan.

4. The method according to claim 1, characterized by, Migrate data from high-risk nodes to selected target nodes at a resource-aware dynamic rate, including: S51 monitors the cluster’s network bandwidth utilization and disk IOPS in real time. When the network bandwidth utilization is higher than the first threshold or the disk IOPS is higher than the second threshold, the migration rate is reduced. S52 increases the migration rate when network bandwidth utilization is below the third threshold and disk IOPS are below the fourth threshold, in order to accelerate data migration during off-peak periods.

5. The method of claim 1, characterized in that, Also includes: S6: After all data migration and cleanup is completed on a high-risk node, the administrator is notified to take the node offline for maintenance. When the node is brought back online after maintenance, it is treated as a brand new empty node and added to the cluster to avoid the risk of failure due to hardware aging.

6. A distributed storage data active migration system based on intelligent health prediction, characterized in that, include: The multi-dimensional health status indicator collection module is used to periodically collect multi-dimensional health status indicators of each node in the distributed storage cluster. The health status indicators include hard disk subsystem indicators, network subsystem indicators, computing and memory subsystem indicators, and software system indicators. The multi-timescale dynamic model calculation module is used to calculate the health score of each node based on health status indicators through a multi-timescale dynamic model. The observation time window for chronic indicators is set to 90 days, for acute indicators to 24 hours, and for event-related indicators to instantaneous. The multi-timescale dynamic model includes long-term trend analysis of chronic indicators, short-term fluctuation assessment of acute indicators, and exponential decay penalty processing for event-related indicators. A synergistic penalty factor is introduced to quantitatively correct the composite risk of multiple abnormal indicators; the calculation formula for the synergistic penalty factor is as follows: wherein is a preset amplification factor for quantifying the composite risk of the plurality of abnormal indicators; The high-risk node marking and migration triggering module is used to mark any node as a high-risk node and trigger an active migration process for all data replicas on that node when the health score of any node continues to fall below a preset active migration threshold. The multi-objective optimization cost function construction module is used to construct a multi-objective optimization cost function for the data to be migrated. The cost function includes load balancing cost, data locality cost, migration efficiency cost, and storage lifetime balancing cost. Based on the cost function, the candidate node with the lowest comprehensive cost is selected from the healthy node pool as the migration target. The resource-aware dynamic migration and metadata update module is used to migrate data on high-risk nodes to selected target nodes at a resource-aware dynamic rate, and atomically update metadata after successful data migration, and then asynchronously clean up old data copies on high-risk nodes.

7. A computer device, comprising: Including processor and memory; The processor reads executable program code stored in memory to run a program corresponding to the executable program code, so as to implement a method for proactive migration of distributed storage data based on intelligent health prediction as described in any one of claims 1-5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements a method for proactive migration of distributed storage data based on intelligent health prediction as described in any one of claims 1-5.

Images