A high-reliability distributed storage data processing method

Abstract

The application discloses a high-reliability distributed storage data processing method, relates to the technical field of distributed storage, and comprises the following steps: collecting hardware health indexes of storage nodes in a distributed storage cluster, real-time available bandwidths among the nodes and repair bandwidth states, and obtaining a heterogeneous state vector; inputting the heterogeneous state vector into a heterogeneous data loss probability model coupled with real-time failure rates of the nodes and the repair bandwidth states for calculation, and obtaining instantaneous data loss probability values of data objects; and through construction of the heterogeneous data loss probability model coupled with the real-time failure rates of the nodes and the repair bandwidth states, dynamic and accurate quantification of data loss risks of a distributed storage system is realized, defects of traditional static evaluation in neglecting fluctuations of network repair capabilities are effectively overcome, and on this basis, in combination with a predictive data migration and a model self-learning feedback mechanism, potential faults can be actively avoided in a sub-health stage, and evaluation accuracy can be continuously optimized.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to a highly reliable distributed storage data processing method. Background Technology

[0002] As the core infrastructure for carrying massive amounts of data, distributed storage systems have always had a focus of industry research on data reliability assurance mechanisms. With the continuous expansion of storage clusters and the increasing heterogeneity of hardware, the failure modes faced by storage nodes have evolved from simple hardware failures to complex risk scenarios including hardware sub-optimal performance, network bandwidth fluctuations, and competition for repair resources. Traditional distributed storage data processing methods mainly rely on static redundancy strategies and passive fault response mechanisms based on heartbeat detection. While these methods ensure data persistence to a certain extent, they have significant limitations when facing complex and dynamic environments.

[0003] However, existing technologies struggle to meet the requirements for synergistic optimization of high reliability and high performance in the following aspects: Traditional risk assessment models typically assume that node failure rates are constant or only related to a single hardware metric, ignoring the nonlinear amplification effect of limited repair bandwidth on the probability of data loss. This leads to a significant lag in risk perception when network congestion or repair channels are abnormal. Secondly, existing data migration and repair strategies often employ fixed threshold triggering or reactive remediation models, lacking the ability to predictively intervene in sub-optimal states. Data reconstruction is often initiated only after a failure actually occurs, resulting in long repair windows, high risk of business interruption, and performance avalanche caused by centralized repair traffic crowding out normal business bandwidth. Furthermore, traditional redundancy strategy adjustments are usually based on globally unified rules or manual experience, failing to jointly model the access frequency, business value, and real-time risk status of data objects. This leads to insufficient protection of high-value data and excessive redundancy of low-value data, making it difficult to achieve a dynamic balance between storage resource utilization and reliability. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a highly reliable distributed storage data processing method, comprising: Collect hardware health indicators, real-time available bandwidth between nodes, and repair bandwidth status of storage nodes in the distributed storage cluster to obtain a heterogeneous state vector; The heterogeneous state vector is input into a heterogeneous data loss probability model that couples the real-time failure rate of nodes with the repair bandwidth status to calculate the instantaneous data loss probability value of the data object. The risk level is determined by comparing the instantaneous data loss probability value with the preset risk threshold. When the risk level is the sub-health warning level, the safety target node is screened based on the heterogeneous state vector and predictive data migration is performed to obtain a migration completion confirmation signal. The real-time failure rate parameters of the corresponding source node are updated using the migration completion confirmation signal to obtain the corrected heterogeneous data loss probability model; When the risk level triggers a data repair task, the adaptive minimum storage regeneration coding parameters are dynamically solved based on the link-level topology data in the heterogeneous state vector, and the repair traffic quota is adjusted in combination with the business delay feedback signal to obtain a controlled repair data flow. According to the preset evaluation cycle, the access popularity and data value tags of the data objects are obtained. Combined with the latest instantaneous data loss probability value output by the modified heterogeneous data loss probability model, the optimal redundancy strategy is calculated through the value entropy joint decision function and asynchronous elastic scaling operation is executed to obtain the redundancy change completion status.

[0006] As a preferred embodiment of the highly reliable distributed storage data processing method of the present invention, the step of inputting the heterogeneous state vector into a heterogeneous data loss probability model coupled with the real-time failure rate of nodes and the repair bandwidth status to calculate the instantaneous data loss probability value of the data object is as follows: Extract the hardware health indicators and repair bandwidth status of each storage node in the set of nodes where the data object is located from the heterogeneous state vector; For each storage node in the node set, the conditional failure rate is calculated based on hardware health indicators, and a logical failure indication value is generated based on the repair bandwidth status. The conditional failure rate and the logical failure indicator value are nonlinearly coupled to obtain the instantaneous failure probability of a single node. The instantaneous data loss probability value of the data object is obtained by performing joint probability calculation on the instantaneous failure probability of all storage nodes in the node set.

[0007] As a preferred embodiment of the highly reliable distributed storage data processing method of the present invention, the specific steps of calculating the conditional failure rate based on hardware health indicators and generating a logical failure indication value based on the repair bandwidth status are as follows: Extract disk reallocation sector count, command timeout count, and temperature fluctuation range from hardware health indicators; The disk reallocation sector count, command timeout count, and temperature fluctuation amplitude are input into a preset hardware degradation mapping function to obtain the hardware health correction factor. Obtain the current read / write load rate of the storage node, and input the read / write load rate into the load pressure mapping function to obtain the load pressure correction factor; The conditional failure rate is obtained by multiplying the base failure rate, hardware health correction factor, and load pressure correction factor. The actual repair bandwidth of the storage node is continuously monitored within the sliding time window. When the actual repair bandwidth is continuously lower than the preset repair threshold within the sliding time window, the logical failure indicator value is set to an effective state. When the actual repair bandwidth recovers to above the preset repair threshold within the sliding time window, the logical failure indicator value is set to an invalid state.

[0008] As a preferred embodiment of the highly reliable distributed storage data processing method of the present invention, the step of performing a nonlinear coupling operation between the conditional failure rate and the logical failure indicator value to obtain the instantaneous failure probability of a single node includes the following steps: Obtain conditional failure rate and logic failure indication values; A nonlinear amplification function containing a logical failure indicator value is constructed. The nonlinear amplification function applies an exponential amplification to the conditional failure rate when the logical failure indicator value is valid, and keeps the conditional failure rate unchanged when the logical failure indicator value is invalid. The conditional failure rate is substituted into the nonlinear amplification function for calculation to obtain the instantaneous failure probability of a single node.

[0009] As a preferred embodiment of the highly reliable distributed storage data processing method of the present invention, the specific steps of filtering secure target nodes based on heterogeneous state vectors and performing predictive data migration to obtain a migration completion confirmation signal are as follows: Nodes with instantaneous data loss probability values ​​higher than the sub-health warning level threshold are removed from the heterogeneous state vectors to obtain an initial candidate node set; Remove nodes from the initial candidate node set that are in the same rack as the source node to obtain a cross-rack candidate node set; For each node in the cross-rack candidate node set, the comprehensive migration cost is calculated based on the real-time available bandwidth between nodes, the conditional failure rate, and the current load rate. The node with the lowest overall migration cost is selected as the safe target node, and the data object is migrated from the source node to the safe target node through rate-limited copying, and a migration completion confirmation signal is obtained.

[0010] As a preferred embodiment of the highly reliable distributed storage data processing method of the present invention, the specific steps of updating the real-time failure rate parameters of the corresponding source node using the migration completion confirmation signal to obtain the corrected heterogeneous data loss probability model are as follows: Record the historical timestamp and current timestamp of the last time the predictive data migration was triggered at the source node, and calculate the time difference between the historical timestamp and the current timestamp; The reciprocal of the time difference is used as the estimate of the soft fault frequency. An exponentially weighted moving average algorithm is used to fuse the estimated soft failure frequency with the original conditional failure rate of the source node to obtain the updated conditional failure rate. The updated conditional failure rate is used to replace the original conditional failure rate parameter of the corresponding source node in the heterogeneous data loss probability model to obtain the corrected heterogeneous data loss probability model.

[0011] As a preferred embodiment of the highly reliable distributed storage data processing method of the present invention, the steps of dynamically solving the adaptive minimum storage regeneration coding parameters based on the link-level topology data in the heterogeneous state vector, and adjusting the repair traffic quota in conjunction with the service delay feedback signal to obtain a controlled repair data stream are as follows: Identify the number of currently available auxiliary nodes and cross-rack bandwidth margin based on link-level topology data; Calculate the network congestion tolerance coefficient based on the service delay feedback signal; Substitute the cross-rack bandwidth margin and network congestion tolerance coefficient into the minimum storage regeneration code boundary constraint equation to solve for the balance between storage overhead and repair bandwidth that satisfies the current network state. Based on the trade-off points, the group size and number of check blocks of the adaptive minimum storage regeneration code are determined to obtain a repair coding configuration that adapts to the current network topology. A repair data stream is generated based on the repair coding configuration, and the sending rate of the repair data stream is dynamically adjusted according to the service delay feedback signal to obtain a controlled repair data stream.

[0012] As a preferred embodiment of the highly reliable distributed storage data processing method of the present invention, the specific steps of calculating the optimal redundancy strategy through the value entropy joint decision function and performing asynchronous elastic scaling operation to obtain the redundancy change completion status are as follows: Obtain the access popularity and data value tags of data objects, and normalize the access popularity and data value tags into value weight factors; The latest instantaneous data loss probability value output by the modified heterogeneous data loss probability model is converted into a risk penalty term; By substituting the value weighting factor and risk penalty term into the value entropy joint decision function, the optimal redundancy strategy is obtained by solving the redundancy configuration that minimizes the sum of storage cost and reliability loss. When the optimal redundancy strategy requires an upgrade in redundancy, the new redundant copy is written first and its integrity is verified before the old check block is deleted. When the optimal redundancy strategy requires downgrading redundancy, a new check block is first written and its integrity is verified before the old redundant copy is deleted, thus obtaining a redundancy change completion status.

[0013] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein the computer program, when executed by the processor, implements any step of the highly reliable distributed storage data processing method as described in the first aspect of the present invention.

[0014] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the highly reliable distributed storage data processing method as described in the first aspect of the present invention.

[0015] The beneficial effects of this invention are as follows: By constructing a heterogeneous data loss probability model that combines the real-time failure rate of coupled nodes with the repair bandwidth status, the invention achieves dynamic and accurate quantification of data loss risk in distributed storage systems, effectively overcoming the shortcomings of traditional static assessments that ignore fluctuations in network repair capabilities. Based on this, combined with predictive data migration and model self-learning feedback mechanisms, the invention can proactively avoid potential faults and continuously optimize assessment accuracy during the sub-healthy phase. At the same time, it adaptively adjusts repair coding parameters and traffic quotas based on link topology and service latency, ensuring zero interference to services during the repair process. The invention also introduces a value entropy joint decision function to drive asynchronous elastic scaling of redundancy strategies, ensuring that storage resource allocation always aligns with data value and real-time risk status. Ultimately, this reduces storage costs and improves overall system operation and maintenance efficiency while ensuring high reliability. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the highly reliable distributed storage data processing method in Example 1. Detailed Implementation

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0019] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0020] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0021] Example 1, referring to Figure 1 This is one embodiment of the present invention, which provides a highly reliable distributed storage data processing method, including the following steps: S1. Collect hardware health indicators, real-time available bandwidth between nodes, and repair bandwidth status of storage nodes in the distributed storage cluster to obtain heterogeneous state vectors.

[0022] Furthermore, the distributed storage cluster's management interface periodically polls each storage node to obtain disk reallocation sector counts, command timeout counts, temperature fluctuations, and read / write load rates, and encapsulates these data into hardware health indicator sub-vectors.

[0023] For any two storage nodes in a distributed storage cluster, bandwidth probe data packets are sent within a preset probe period. The real-time available bandwidth between nodes is calculated based on the sending and receiving timestamps of the bandwidth probe data packets, and the real-time available bandwidth between all node pairs is encapsulated into a bandwidth status sub-vector. The actual repair traffic throughput of each storage node within a sliding time window is continuously monitored, and the comparison result between the actual repair traffic throughput and a preset repair threshold is encapsulated into a repair bandwidth status sub-vector. The hardware health indicator sub-vector, bandwidth status sub-vector, and repair bandwidth status sub-vector are concatenated according to a preset field order to obtain a heterogeneous state vector representing the current operating state of the distributed storage cluster.

[0024] It should be noted that by collecting hardware health, inter-node bandwidth, and repair bandwidth status from multiple dimensions and encapsulating them into structured heterogeneous state vectors, the real-time operating status of the distributed storage cluster can be comprehensively depicted. This provides a complete and timely data foundation for subsequent risk assessment and decision-making, effectively avoiding misjudgments or omissions caused by evaluation based on a single indicator.

[0025] Specifically, the calculation of real-time available bandwidth between nodes employs an active probing mechanism based on timestamp differences rather than relying on switch traffic statistics. This is because switch statistics only reflect the load of aggregated ports and cannot perceive the actual effective throughput of the storage protocol layer, and suffer from coarse sampling granularity. Active probing, by sending fixed-size bandwidth probe packets and recording precise send and receive timestamps, can directly measure the available bandwidth at the application layer between storage nodes, eliminating interference from protocol stack overhead and queuing delays. After applying this invention, the obtained real-time available bandwidth between nodes accurately reflects the actual transmission capacity that data migration and repair tasks can utilize, providing an accurate physical layer constraint basis for subsequent migration cost calculation and repair traffic quota adjustment. The expression for calculating real-time available bandwidth between nodes is: ; In the formula, This indicates the real-time available bandwidth between nodes. Indicates the payload length of the bandwidth probe data packet. This indicates the timestamp of the received bandwidth probe data packet. This indicates the timestamp of the bandwidth probe data packet being sent.

[0026] To generate the repair bandwidth state subvector, a mechanism is adopted that continuously compares the actual repair traffic throughput with the preset repair threshold within a sliding time window. The motivation is that instantaneous bandwidth fluctuations are normal network jitter. If the repair capability is judged by single-point sampling, the logical failure indicator value will frequently flip, thus causing oscillations in risk assessment. The sliding time window mechanism filters short-term spikes by continuously monitoring, and only confirms that the repair capability is substantially damaged when the repair bandwidth is continuously lower than the threshold. After being applied to this invention, the obtained repair bandwidth state subvector can distinguish between transient network congestion and persistent repair channel failure, so that the logical failure indicator value in subsequent nonlinear coupling operations has time stability and avoids the false triggering of predictive migration and repair strategies.

[0027] S2. Input the heterogeneous state vector into the heterogeneous data loss probability model that couples the real-time failure rate of the node with the repair bandwidth state, and calculate to obtain the instantaneous data loss probability value of the data object.

[0028] Furthermore, the hardware health indicators and repair bandwidth status of each storage node in the node set where the data object resides are extracted from the heterogeneous state vector; for each storage node in the node set, the conditional failure rate is calculated based on the hardware health indicators, and a logical failure indication value is generated based on the repair bandwidth status; the conditional failure rate and the logical failure indication value are nonlinearly coupled to obtain the instantaneous failure probability of a single node; the instantaneous failure probabilities of all storage nodes in the node set are jointly calculated to obtain the instantaneous data loss probability value of the data object.

[0029] Extract disk reallocation sector count, command timeout count, and temperature fluctuation amplitude from hardware health indicators; input the disk reallocation sector count, command timeout count, and temperature fluctuation amplitude into a preset hardware degradation mapping function to obtain a hardware health correction factor; obtain the current read / write load rate of the storage node and input the read / write load rate into a load pressure mapping function to obtain a load pressure correction factor.

[0030] The conditional failure rate is obtained by multiplying the base failure rate, hardware health correction factor, and load pressure correction factor. The actual repair bandwidth of the storage node is continuously monitored within the sliding time window. When the actual repair bandwidth is continuously lower than the preset repair threshold within the sliding time window, the logical failure indicator value is set to an active state. When the actual repair bandwidth recovers to above the preset repair threshold within the sliding time window, the logical failure indicator value is set to an inactive state.

[0031] Obtain the conditional failure rate and the logical failure indicator value; construct a nonlinear amplification function that incorporates the logical failure indicator value. This function exponentially amplifies the conditional failure rate when the logical failure indicator value is valid, and keeps the conditional failure rate unchanged when the logical failure indicator value is invalid; substitute the conditional failure rate into the nonlinear amplification function to calculate the instantaneous failure probability of a single node; the expression for the nonlinear amplification function is: ; in, This represents the instantaneous failure probability of a single node. Indicates the conditional failure rate. Indicates a logical failure indicator value. This represents the preset failure amplification factor. This represents the natural exponential function.

[0032] It should be noted that by nonlinearly coupling the real-time failure rate of nodes with the repair bandwidth status, the failure risk weight can be automatically amplified when the repair capability is limited. This allows the instantaneous data loss probability value to truly reflect the dynamic impact of network fluctuations on data reliability, improves the sensitivity and accuracy of risk perception, and overcomes the shortcomings of traditional static models that cannot adapt to changes in the network environment.

[0033] The hardware degradation mapping function employs a normalized Sigmoid monotonically increasing function. This is because the physical dimensions and numerical ranges of the three indicators—disk reallocation sector count, command timeout count, and temperature fluctuation amplitude—differ significantly. Direct weighting would lead to large numerical indicators dominating the results while small numerical indicators are overwhelmed. The Sigmoid function compresses these three heterogeneous indicators into a unified bounded interval while maintaining monotonicity, making different hardware anomaly signals comparable on the same scale. When applied to this invention, the resulting hardware health correction factor eliminates dimensional bias, enabling the conditional failure rate to reflect the combined impact of various hardware degradation symptoms in a balanced manner. The calculation expression for the hardware health correction factor is: ; In the formula, This indicates the hardware health correction factor. This represents the raw value of the input hardware health indicator. This represents the baseline reference value for the corresponding hardware health indicator. This indicates the preset sensitivity adjustment coefficient. This represents the natural exponential function.

[0034] The load stress mapping function uses a piecewise linear saturation function. The motivation is that the impact of read and write operations on hardware wear of storage nodes is approximately constant in the low load range. Only when the load exceeds the critical point does the wear rate increase linearly with the load, and there is a physical upper limit. The piecewise linear saturation function accurately describes this nonlinear load-wear relationship, avoiding excessive amplification of the failure rate under low load and exceeding reasonable boundaries under high load. After being applied to this invention, the resulting load stress correction factor ensures that the conditional failure rate only increases under true high stress conditions, which is consistent with the actual physical degradation law of storage devices.

[0035] The joint probability calculation employs a product model under the independent failure assumption. This is motivated by the fact that in distributed storage, multiple replicas or checksum blocks of the same data object are distributed across different physical nodes, and hardware and network failures between nodes are statistically independent. The product model strictly follows the rules for calculating the joint probability of independent events in probability theory, avoiding probability overflow caused by the additive model or additional errors introduced by the correlation assumption. When applied to this invention, the resulting instantaneous data loss probability is mathematically strictly equal to the probability of simultaneous failure of all bearer nodes, providing an unbiased quantitative benchmark for risk level assessment. The expression for calculating the instantaneous data loss probability is: ; In the formula, This represents the instantaneous probability of data loss for a data object. This represents the total number of storage nodes that hold the data objects. Indicates the first The instantaneous failure probability of each storage node. This represents the chain multiplication operator.

[0036] S3. Determine the risk level based on the comparison between the instantaneous data loss probability value and the preset risk threshold. When the risk level is the sub-health warning level, select safe target nodes based on heterogeneous state vectors and perform predictive data migration to obtain a migration completion confirmation signal.

[0037] Furthermore, nodes with instantaneous data loss probability values ​​higher than the sub-health warning level threshold are removed from the heterogeneous state vectors to obtain an initial candidate node set; nodes in the same rack as the source node are removed from the initial candidate node set to obtain a cross-rack candidate node set; for each node in the cross-rack candidate node set, the comprehensive migration cost is calculated based on the real-time available bandwidth between nodes, the conditional failure rate, and the current load rate; the node with the minimum comprehensive migration cost is selected as the safe target node, and the data object is migrated from the source node to the safe target node through rate-limited copying, obtaining a migration completion confirmation signal.

[0038] It should be noted that during the sub-health warning stage, based on cross-rack topology and comprehensive migration costs, safe target nodes are screened and predictive migration is performed. High-risk data can be proactively transferred before the actual failure occurs. This avoids business interruption and bandwidth contention caused by post-failure repair, and ensures that normal read and write performance is not disturbed through the rate-limiting copy mechanism, thus achieving synergistic optimization of reliability and availability.

[0039] The calculation of the comprehensive migration cost employs a multi-objective weighted normalization method. This is motivated by the fact that the three factors—real-time available bandwidth, conditional failure rate, and current load rate—have inconsistent influence directions on migration decisions, and their dimensions cannot be directly added. Weighted normalization converts these three heterogeneous indicators into dimensionless, unidirectional cost values, enabling migration decisions to balance transmission efficiency, target node reliability, and service interference within a unified framework. When applied to this invention, the resulting comprehensive migration cost ensures that the selection of secure target nodes simultaneously considers migration speed, long-term survival probability, and online service protection, avoiding the deterioration of other dimensions due to the optimization of a single indicator. The expression for calculating the comprehensive migration cost is: ; In the formula, Indicates the overall migration cost. This indicates the real-time available bandwidth between candidate nodes. This represents the maximum observed real-time available bandwidth among nodes in the cluster. This indicates the conditional failure rate of the candidate node. This indicates the current load rate of the candidate nodes. , , These represent the preset weighting coefficients for bandwidth, failure rate, and load rate, respectively.

[0040] The rate-limited copy method employs a token bucket rate control algorithm. The motivation is that predictive migration is a background preventative operation that should not preempt front-end business I / O resources, but it should also avoid increasing the source node failure exposure time during migration due to excessively low rate limits leading to an excessively long migration window. The token bucket algorithm allows for bursts of small amounts of data to cope with brief periods of idle bandwidth, while strictly controlling long-term average bandwidth usage through the token generation rate. When applied to this invention, the migration process corresponding to the migration completion confirmation signal ensures that the business latency SLA is not violated, maximizes migration throughput within the available bandwidth window, and shortens the exposure time of data in a sub-healthy state.

[0041] S4. Use the migration completion confirmation signal to update the real-time failure rate parameters of the corresponding source node to obtain the corrected heterogeneous data loss probability model.

[0042] Furthermore, the historical timestamp and current timestamp of the last time the predictive data migration was triggered by the source node are recorded, and the time difference between the historical timestamp and the current timestamp is calculated. The reciprocal of the time difference is used as the soft failure frequency estimate. The soft failure frequency estimate is fused with the original conditional failure rate of the source node using an exponentially weighted moving average algorithm to obtain the updated conditional failure rate.

[0043] The updated conditional failure rate parameter is used to replace the original conditional failure rate parameter of the corresponding source node in the heterogeneous data loss probability model to obtain the corrected heterogeneous data loss probability model; the expression for the exponentially weighted moving average algorithm is: ; in, This indicates the failure rate of the updated conditions. This represents the estimated frequency of soft faults. This indicates the original failure rate of the source node's conditions. This represents the preset learning rate weighting coefficient.

[0044] It should be noted that by using the migration completion confirmation signal to feed back the source node failure rate parameter update, the model has the ability to learn about the frequency of soft faults. It can continuously calibrate the evaluation benchmark based on historical migration behavior, avoid model drift caused by hardware aging or environmental changes, and ensure that risk prediction always fits the actual system state under long-term operation.

[0045] S5. When the risk level triggers the data repair task, the adaptive minimum storage regeneration coding parameters are dynamically solved based on the link-level topology data in the heterogeneous state vector, and the repair traffic quota is adjusted in combination with the service delay feedback signal to obtain a controlled repair data flow.

[0046] Furthermore, based on link-level topology data, the number of currently available auxiliary nodes and cross-rack bandwidth margin are identified; the network congestion tolerance coefficient is calculated based on service delay feedback signals; the cross-rack bandwidth margin and network congestion tolerance coefficient are substituted into the minimum storage regeneration code boundary constraint equation to solve for the trade-off point between storage overhead and repair bandwidth that satisfies the current network state; based on the trade-off point, the packet size and check block number of the adaptive minimum storage regeneration code are determined to obtain a repair coding configuration adapted to the current network topology state; a repair data stream is generated based on the repair coding configuration, and the sending rate of the repair data stream is dynamically adjusted based on service delay feedback signals to obtain a controlled repair data stream.

[0047] It should be noted that the adaptive coding parameters are dynamically solved and the repair traffic quota is adjusted based on the link topology and service delay feedback. This allows the repair process to flexibly adjust resource consumption according to the degree of network congestion and service priority. While ensuring repair efficiency, it strictly limits the interference with online services and achieves an intelligent balance between repair tasks and production load.

[0048] The network congestion tolerance coefficient employs a negative correlation mapping of service delay feedback signals. This is motivated by the fact that increased service delay directly indicates that the network path is approaching a congestion threshold, at which point repair tasks must relinquish more bandwidth resources. The negative correlation mapping causes the tolerance coefficient to decrease monotonically with increasing delay, automatically shrinking the upper limit of the repair traffic quota. When applied to this invention, the resulting network congestion tolerance coefficient transforms application-layer service quality perception into constraints for solving coding parameters, allowing the adjustment of repair strategies to be driven by underlying network indicators rather than manual experience. The calculation expression for the network congestion tolerance coefficient is: ; In the formula, This represents the network congestion tolerance coefficient. This indicates the current service delay feedback signal value. Indicates the baseline value of service latency. This indicates the upper limit of business latency tolerance. This represents the function that takes the maximum value.

[0049] The Lagrange multiplier method is used to solve the boundary constraint equation of the minimum storage regeneration code. The motivation lies in the theoretical lower bound constraint between storage overhead and repair bandwidth. The cross-rack bandwidth margin and the network congestion tolerance coefficient jointly define the current feasible region, and the Pareto optimal point needs to be found within the feasible region. The Lagrange multiplier method transforms the constrained optimization problem into an unconstrained stationary point solution, which can analytically obtain the optimal trade-off between storage overhead and repair bandwidth without traversal search. When applied to this invention, the obtained trade-off point mathematically guarantees that the repair bandwidth occupation is minimized under the current network condition while the storage redundancy does not exceed the theoretical lower bound, so that the adaptive coding configuration has both theoretical optimality and engineering feasibility.

[0050] S6. Obtain the access popularity and data value tags of data objects according to the preset evaluation cycle, combine the latest instantaneous data loss probability value output by the modified heterogeneous data loss probability model, calculate the optimal redundancy strategy through the value entropy joint decision function and execute asynchronous elastic scaling operation to obtain the redundancy change completion status.

[0051] Furthermore, the access popularity and data value tags of data objects are obtained, and the access popularity and data value tags are normalized into value weight factors; the latest instantaneous data loss probability value output by the modified heterogeneous data loss probability model is converted into a risk penalty term; the value weight factors and risk penalty term are substituted into the value entropy joint decision function to solve for the redundancy configuration that minimizes the sum of storage cost and reliability loss, thus obtaining the optimal redundancy strategy.

[0052] When the optimal redundancy strategy requires an upgrade in redundancy, a new redundant copy is written and its integrity verified before the old check block is deleted; when the optimal redundancy strategy requires a downgrade in redundancy, a new check block is written and its integrity verified before the old redundant copy is deleted, resulting in a redundancy change completion status; the expression for the value entropy joint decision function is: ; in, This represents the combined cost of storage and reliability loss. Indicates the value weighting factor. Indicates the redundancy of the candidate redundant configuration. This represents the probability value of the latest instantaneous data being lost. This represents the preset risk penalty coefficient. This represents the natural logarithm function.

[0053] It should be noted that by using the value entropy joint decision function to integrate and evaluate data access popularity, value tags and real-time risks, the redundancy strategy is driven to scale asynchronously on demand. This allows high-value, high-frequency data to receive stronger protection while low-value data has a moderate reduction in redundancy overhead. Under the premise of meeting overall reliability standards, the utilization rate of storage resources is optimized, achieving a globally optimal match between cost and reliability.

[0054] The value weight factor normalization employs a Min-Max linear scaling method. This is motivated by the fact that the original distribution of access popularity and data value tags may be highly skewed, with a few hot data points occupying the vast majority of the numerical range. Directly substituting these into the decision function would cause the weight of cold data to approach zero and be ignored. Min-Max scaling maps both to a unified interval while maintaining their relative order, ensuring that all data objects have non-zero weight in redundant decisions. When applied to this invention, the resulting value weight factor ensures that redundant strategy adjustments cover the entire dataset rather than only serving top-tier hot data, preventing long-tail data from being underprotected due to excessively low weight. The calculation expression for the value weight factor is: ; In the formula, Indicates the value weighting factor. This represents the weighted sum of the visit popularity and the raw value of the data value tag. Indicates the current evaluation period The minimum observed value, Indicates the current evaluation period The maximum observed value.

[0055] The transformation of the risk penalty term uses a linear proportional mapping. The motivation is that the probability value of the latest instantaneous data loss is already a normalized probability, but its magnitude is usually much smaller than the value weight factor. Direct addition would cause the risk term to be submerged in the joint cost. The linear proportional mapping amplifies the probability value to a magnitude comparable to the storage cost term through the risk penalty coefficient, making the two terms comparable in influence during the optimization process. After being applied to this invention, the resulting risk penalty term ensures that the redundancy degradation decision will not be overly aggressive due to the small risk cost, and the upgrade decision will not be overly conservative due to the large risk cost, so that the minimization of the joint cost value truly reflects the balance between cost and reliability.

[0056] The sequential execution strategy of writing before deleting in asynchronous elastic scaling operations is motivated by the fact that distributed storage systems are in a temporarily vulnerable state within the redundancy change window. If deletion is performed before writing, the data redundancy will fall below the safety threshold before the writing is completed, and any single point of failure may lead to permanent data loss. Writing before deleting ensures that the data redundancy is never lower than the level before the change throughout the entire change process, exchanging the atomic safety of the change process for a short-term additional storage overhead. After applying this invention, the operation process corresponding to the redundancy change completion state satisfies the minimum reliability constraint at any time, eliminating the data loss risk exposure introduced by the elastic scaling operation itself.

[0057] This embodiment also provides a computer device suitable for a highly reliable distributed storage data processing method, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the highly reliable distributed storage data processing method proposed in the above embodiment.

[0058] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0059] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the highly reliable distributed storage data processing method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0060] In summary, this invention achieves dynamic and accurate quantification of data loss risk in distributed storage systems by constructing a heterogeneous data loss probability model that combines the real-time failure rate of coupled nodes with the repair bandwidth status. This effectively overcomes the shortcomings of traditional static assessments that ignore fluctuations in network repair capabilities. Based on this, combined with predictive data migration and model self-learning feedback mechanisms, it can proactively avoid potential faults and continuously optimize assessment accuracy during the sub-health stage. At the same time, it adaptively adjusts repair coding parameters and traffic quotas according to link topology and service latency, ensuring zero interference to services during the repair process. The introduction of a value entropy joint decision function drives the asynchronous elastic scaling of redundancy strategies, ensuring that storage resource allocation always aligns with data value and real-time risk status. Ultimately, it reduces storage costs and improves overall system operation and maintenance efficiency while ensuring high reliability.

[0061] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A highly reliable distributed storage data processing method, characterized in that: include: Collect hardware health indicators, real-time available bandwidth between nodes, and repair bandwidth status of storage nodes in the distributed storage cluster to obtain a heterogeneous state vector; The heterogeneous state vector is input into a heterogeneous data loss probability model that couples the real-time failure rate of nodes with the repair bandwidth status to calculate the instantaneous data loss probability value of the data object. The risk level is determined by comparing the instantaneous data loss probability value with the preset risk threshold. When the risk level is the sub-health warning level, the safety target node is screened based on the heterogeneous state vector and predictive data migration is performed to obtain a migration completion confirmation signal. The real-time failure rate parameters of the corresponding source node are updated using the migration completion confirmation signal to obtain the corrected heterogeneous data loss probability model; When the risk level triggers the data repair task, the adaptive minimum storage regeneration coding parameters are dynamically solved based on the link-level topology data in the heterogeneous state vector, and the repair traffic quota is adjusted in combination with the business delay feedback signal to obtain a controlled repair data flow. According to the preset evaluation cycle, the access popularity and data value tags of the data objects are obtained. Combined with the latest instantaneous data loss probability value output by the modified heterogeneous data loss probability model, the optimal redundancy strategy is calculated through the value entropy joint decision function and asynchronous elastic scaling operation is executed to obtain the redundancy change completion status.

2. The highly reliable distributed storage data processing method as described in claim 1, characterized in that: The process involves inputting a heterogeneous state vector into a heterogeneous data loss probability model that couples the node's real-time failure rate with the repair bandwidth state to calculate the instantaneous data loss probability value of the data object. The specific steps are as follows: Extract the hardware health indicators and repair bandwidth status of each storage node in the set of nodes where the data object is located from the heterogeneous state vector; For each storage node in the node set, the conditional failure rate is calculated based on hardware health indicators, and a logical failure indication value is generated based on the repair bandwidth status. The conditional failure rate and the logical failure indicator value are nonlinearly coupled to obtain the instantaneous failure probability of a single node. The instantaneous data loss probability value of the data object is obtained by performing joint probability calculation on the instantaneous failure probability of all storage nodes in the node set.

3. The highly reliable distributed storage data processing method as described in claim 2, characterized in that: The specific steps for calculating the conditional failure rate based on hardware health indicators and generating a logical failure indication value based on the repair bandwidth status are as follows: Extract disk reallocation sector count, command timeout count, and temperature fluctuation range from hardware health indicators; The disk reallocation sector count, command timeout count, and temperature fluctuation amplitude are input into a preset hardware degradation mapping function to obtain the hardware health correction factor. Obtain the current read / write load rate of the storage node, and input the read / write load rate into the load pressure mapping function to obtain the load pressure correction factor; The conditional failure rate is obtained by multiplying the base failure rate, hardware health correction factor, and load pressure correction factor. The actual repair bandwidth of the storage node is continuously monitored within the sliding time window. When the actual repair bandwidth is continuously lower than the preset repair threshold within the sliding time window, the logical failure indicator value is set to an effective state. When the actual repair bandwidth recovers to above the preset repair threshold within the sliding time window, the logical failure indicator value is set to an invalid state.

4. The highly reliable distributed storage data processing method as described in claim 3, characterized in that: The specific steps for performing a nonlinear coupling operation between the conditional failure rate and the logical failure indicator value to obtain the instantaneous failure probability of a single node are as follows: Obtain conditional failure rate and logic failure indication values; A nonlinear amplification function containing a logical failure indicator value is constructed. The nonlinear amplification function applies an exponential amplification to the conditional failure rate when the logical failure indicator value is valid, and keeps the conditional failure rate unchanged when the logical failure indicator value is invalid. The conditional failure rate is substituted into the nonlinear amplification function for calculation to obtain the instantaneous failure probability of a single node.

5. The highly reliable distributed storage data processing method as described in claim 4, characterized in that: The specific steps for selecting safe target nodes based on heterogeneous state vectors and performing predictive data migration to obtain a migration completion confirmation signal are as follows: Nodes with instantaneous data loss probability values ​​higher than the sub-health warning level threshold are removed from the heterogeneous state vectors to obtain an initial candidate node set; Remove nodes from the initial candidate node set that are in the same rack as the source node to obtain a cross-rack candidate node set; For each node in the cross-rack candidate node set, the comprehensive migration cost is calculated based on the real-time available bandwidth between nodes, the conditional failure rate, and the current load rate. The node with the lowest overall migration cost is selected as the safe target node, and the data object is migrated from the source node to the safe target node through rate-limited copying, and a migration completion confirmation signal is obtained.

6. The highly reliable distributed storage data processing method as described in claim 5, characterized in that: The steps for updating the real-time failure rate parameters of the corresponding source node using the migration completion confirmation signal to obtain the corrected heterogeneous data loss probability model are as follows: Record the historical timestamp and current timestamp of the last time the predictive data migration was triggered at the source node, and calculate the time difference between the historical timestamp and the current timestamp; The reciprocal of the time difference is used as the estimate of the soft fault frequency. An exponentially weighted moving average algorithm is used to fuse the estimated soft failure frequency with the original conditional failure rate of the source node to obtain the updated conditional failure rate. The updated conditional failure rate is used to replace the original conditional failure rate parameter of the corresponding source node in the heterogeneous data loss probability model to obtain the corrected heterogeneous data loss probability model.

7. The highly reliable distributed storage data processing method as described in claim 6, characterized in that: The process of dynamically solving the adaptive minimum storage regeneration coding parameters based on the link-level topology data in the heterogeneous state vector, and adjusting the repair traffic quota in conjunction with the service delay feedback signal to obtain a controlled repair data stream, includes the following steps: Identify the number of currently available auxiliary nodes and cross-rack bandwidth margin based on link-level topology data; Calculate the network congestion tolerance coefficient based on the service delay feedback signal; Substitute the cross-rack bandwidth margin and network congestion tolerance coefficient into the minimum storage regeneration code boundary constraint equation to solve for the balance between storage overhead and repair bandwidth that satisfies the current network state. Based on the trade-off points, the group size and number of check blocks of the adaptive minimum storage regeneration code are determined to obtain a repair coding configuration that adapts to the current network topology. A repair data stream is generated based on the repair coding configuration, and the sending rate of the repair data stream is dynamically adjusted according to the service delay feedback signal to obtain a controlled repair data stream.

8. The highly reliable distributed storage data processing method as described in claim 7, characterized in that: The specific steps for calculating the optimal redundancy strategy using the value entropy joint decision function and executing asynchronous elastic scaling operations to obtain the redundancy change completion status are as follows: Obtain the access popularity and data value tags of data objects, and normalize the access popularity and data value tags into value weight factors; The latest instantaneous data loss probability value output by the modified heterogeneous data loss probability model is converted into a risk penalty term; By substituting the value weighting factor and risk penalty term into the value entropy joint decision function, the optimal redundancy strategy is obtained by solving the redundancy configuration that minimizes the sum of storage cost and reliability loss. When the optimal redundancy strategy requires an upgrade in redundancy, the new redundant copy is written first and its integrity is verified before the old check block is deleted. When the optimal redundancy strategy requires downgrading redundancy, a new check block is first written and its integrity is verified before the old redundant copy is deleted, thus obtaining a redundancy change completion status.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the highly reliable distributed storage data processing method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the highly reliable distributed storage data processing method according to any one of claims 1 to 8.

Images