A computation-aware erasure code repair method and system for analytics storage clusters

Abstract

The application belongs to the technical field of distributed storage, and specifically discloses a calculation-aware erasure code repair method and system for an analysis storage cluster. The method comprises the following steps: detecting a data block fault in a target encoding strip, determining a candidate node set and obtaining the network resource state and the calculation resource state of each candidate node; estimating the decoding throughput rate of the candidate nodes for data decoding; constructing a repair time model for representing network transmission time and decoding calculation time according to the network resource state and the decoding throughput rate of each candidate node; determining the repair task allocation proportion of each candidate node based on the repair time model, so as to minimize the maximum value in the repair operation time of all candidate nodes; generating a repair plan for data transmission and decoding according to the repair task allocation proportion, and scheduling the corresponding candidate nodes to perform data transmission and decoding operations, so as to recover the failed data block. Through the application, joint perception of calculation resources and network resources is realized.

Description

Technical Field

[0001] This application belongs to the field of distributed storage technology, and more specifically, relates to a computation-aware erasure coding repair method and system for analytical storage clusters. Background Technology

[0002] In distributed data analytics storage clusters, data reliability is the cornerstone of business continuity. With the explosive growth of data volume, erasure coding technology is widely used to replace traditional multi-replica mechanisms due to its high storage efficiency. When a data block or node fails in the cluster, timely and efficient execution of erasure coding repair operations is crucial for preventing permanent data loss, maintaining system service level agreements (SLAs), and ensuring the stability of upper-layer data analytics tasks.

[0003] Existing erasure coding repair techniques primarily focus on network-level optimization, aiming to reduce repair bandwidth consumption or network transmission time. These approaches typically assume that the main bottleneck to repair performance lies in network I / O (input / output), and therefore accelerate the data recovery process by designing optimal repair topologies, selecting the nearest network node, or employing pipelined transmission strategies. These methods have indeed significantly improved repair efficiency and achieved good application results in traditional storage architectures with limited network resources but relatively abundant computing resources.

[0004] However, in modern analytics storage clusters, data analytics executors and storage nodes are often deployed on the same physical node to achieve data locality. In this co-location architecture, the distributed data analytics workloads running in the background periodically compete for computing resources such as CPU and memory bandwidth. Existing remediation solutions, lacking awareness of node computing resource status, still rely solely on network bandwidth for scheduling, easily assigning decoding tasks to nodes currently experiencing computational intensive phases. This causes the remediation pipeline to be blocked by computational bottlenecks, resulting in actual remediation times far exceeding theoretical expectations, severely impacting the timeliness of fault recovery and the overall availability of the cluster. Summary of the Invention

[0005] To address the problem that existing technologies fail to detect dynamic changes in node computing resources under analytical loads, resulting in repair tasks being scheduled to computational bottleneck nodes and low repair efficiency, this application proposes a computation-aware erasure coding repair method and system for analytical storage clusters. This method achieves joint awareness and collaborative optimization of computing and network resources, thereby significantly reducing erasure coding repair time under complex load environments.

[0006] The first aspect of this application relates to a computation-aware erasure coding repair method for analytical storage clusters. The method includes: detecting a data block failure in a target coding stripe; determining the coding parameters of the target coding stripe, the failed data block, and a set of candidate nodes storing available data blocks; and obtaining the network resource status and computational resource status of each candidate node in the candidate node set; estimating the decoding throughput of the candidate nodes for data decoding based on the periodic and phased computational patterns of the distributed data analysis load running by the data analysis executor, combined with the computational resource status; constructing a repair time model to characterize network transmission time and decoding computation time based on the network resource status and decoding throughput of each candidate node; determining the repair task allocation ratio for each candidate node based on the repair time model to minimize the maximum value among the repair operation times of all candidate nodes; generating a data transmission and decoding repair plan based on the repair task allocation ratio, and scheduling the corresponding candidate nodes to perform data transmission and decoding operations to recover the failed data block.

[0007] The above-mentioned scheme incorporates decoding throughput as a key indicator with the same dimensions as network bandwidth into the repair decision, breaking the limitation of traditional schemes that only focus on the network dimension. This scheme utilizes the periodic patterns of the analysis load to predict computational capabilities, enabling the repair time model to accurately quantify both computational and network bottlenecks. This avoids scheduling tasks to computationally constrained nodes, ensuring the smoothness of the repair pipeline from a global perspective.

[0008] By establishing an optimization model aimed at minimizing the maximum repair operation time, adaptive load balancing of repair tasks among candidate nodes was achieved. This ensured that no single node became a bottleneck in the repair process, effectively eliminating tail latency caused by individual node computation or network congestion, bringing the overall repair performance close to the theoretical optimum, and improving the system's response speed under fault scenarios.

[0009] In some implementations, periodic and phased computation patterns are characterized by data analysis iteration cycles and time phase offsets within those cycles. Estimating the decoding throughput of candidate nodes for data decoding includes: acquiring historical decoding profile records for candidate nodes, which include the time phase offset relative to the start of the corresponding data analysis iteration cycle when the historical decoding operation occurred, historical computing resource status, and actual decoding throughput; training a throughput prediction model based on the historical decoding profile records; and inputting the current time phase offset relative to the start of the data analysis iteration cycle and the current computing resource status into the throughput prediction model when the repair task arrives to obtain the expected decoding throughput of the corresponding candidate node.

[0010] This scheme fully leverages the periodicity of data analysis task iterations by introducing time phase offset as a prediction feature. Compared to relying solely on instantaneous resource utilization, this method aligns discrete decoding samples to a unified iteration cycle, effectively capturing the structural characteristics of computational resource changes across task stages. This allows for accurate short-term prediction of decoding capabilities without the need for long-term offline sampling.

[0011] In some implementations, the computing resource status includes CPU utilization and memory bandwidth usage; when historical decoding profile records are missing or a change in the pattern of distributed data analysis load is detected, multiple decoding micro-probe tasks are actively executed within the initial iteration cycle of the data analysis iteration cycle, initial decoding profile records are collected based on the decoding micro-probe tasks, and the throughput prediction model is updated or retrained accordingly.

[0012] This solution addresses the issue of predictive model failure during cold starts or sudden load changes by proactively injecting decoding micro-probe tasks. It enables rapid collection of critical samples without impacting normal business operations, ensuring the predictive model remains synchronized with the actual computational characteristics of the current system, and enhancing the robustness and adaptability of the remediation strategy to dynamic environmental changes.

[0013] In some implementations, network resource status includes upload bandwidth and download bandwidth. The repair time model is used to calculate the decoding time, upload time, and download time of candidate nodes. The repair operation time of any candidate node is the maximum value among its decoding time, upload time, and download time. The repair task allocation ratio of each candidate node is determined based on the repair time model, including: establishing a linear programming model with the optimization objective of minimizing the maximum value among the repair operation times of all candidate nodes; and determining the data collection ratio of each candidate node as a collection node and the data transmission ratio as a sending node by solving the linear programming model.

[0014] This solution constructs a refined time model encompassing decoding, uploading, and downloading, and transforms the task allocation problem into a linear programming solution, achieving optimal matching of computational and network resources. This method dynamically adjusts the collection and transmission ratio based on the real-time resource endowment of each node, ensuring that the repair operation time for each node tends to be consistent, fundamentally avoiding resource waste and localized overload.

[0015] In some implementations, for candidate nodes Decoding time Upload time Download time ;in, The number of blocks required to recover the failed data blocks. For block size, Candidate nodes Data collection ratio Candidate nodes Data transmission ratio Candidate nodes Decoding throughput, Candidate nodes upload bandwidth, Candidate nodes The download bandwidth; the constraints of the optimization objective include: the sum of the data collection ratios of all candidate nodes is 1, and the sum of the data transmission ratios of all candidate nodes is... The sum of the data collection ratio and the data transmission ratio of any candidate node shall not exceed 1.

[0016] This scheme uses specific mathematical formulas to characterize the time cost of each node under different task roles and sets strict physical constraints to ensure the correctness of the repair logic. This quantitative modeling approach enables the optimization solver to find the exact solution that minimizes the overall system repair time while ensuring data integrity, providing a numerical basis for subsequent transmission path generation.

[0017] In some implementations, a repair plan for data transmission and decoding is generated based on the repair task allocation ratio, including: constructing a flow network containing source, sink, sending node logical entities, and collection node logical entities; setting the edge capacity in the flow network according to the data collection ratio and data sending ratio of candidate nodes; solving the traffic allocation from the sending node logical entity to the collection node logical entity based on the flow network with the set edge capacity, and converting the traffic allocation into sub-block transmission paths to generate a repair plan for data transmission and decoding based on the sub-block transmission paths.

[0018] This solution uses the maximum flow algorithm to transform the proportional allocation result into executable sub-block-level transmission paths, completing a closed loop from optimization decision-making to actual scheduling. This method not only ensures the feasibility of the task allocation ratio in the actual network topology but also automatically generates specific routing instructions, enabling the repair plan to be directly executed by the underlying system, thus improving the engineering feasibility and practicality of the solution.

[0019] In some implementations, when multiple failed data blocks exist in the same target coding stripe, after establishing a repair plan for the data transmission and decoding of the first failed data block, the candidate nodes selected to perform the decoding operation are grouped into a set of reuse nodes. For subsequent failed data blocks, if the estimated processing time of the candidate nodes in the set of reuse nodes after reusing multiple repair contexts is less than the estimated processing time of enabling new candidate nodes, the decoding task of the subsequent failed data blocks is preferentially assigned to the candidate nodes in the set of reuse nodes; otherwise, the repair task allocation ratio of each candidate node for the subsequent failed data blocks is re-determined based on the repair time model.

[0020] This solution reduces redundant data reading and memory loading overhead by selectively reusing multiple repair contexts. In multi-block failure scenarios, this strategy intelligently balances the benefits of context reuse with the cost of new node scheduling, accelerating batch repair while avoiding excessive load on a single node due to overuse, thus achieving dual optimization of multi-block repair efficiency and load balancing.

[0021] In some implementations, when a full node failure occurs and triggers multiple repair tasks involving different coding stripes, for a first candidate node that has been assigned repair tasks, it is assessed whether it will transition to a computationally intensive phase based on the historical idle phase duration and the current idle phase duration of the first candidate node; if the assessment is that the first candidate node will transition to a computationally intensive phase, then repair tasks to be migrated are determined from the repair tasks already assigned to the first candidate node; a second candidate node currently in a computationally idle phase is determined, and the repair tasks to be migrated are migrated from the first candidate node to the second candidate node.

[0022] This solution enables proactive migration and obstacle avoidance of repair tasks by predicting the evolution trend of node computing states. In high-load scenarios of full-node repair, this mechanism can identify potential slow nodes that are about to enter a busy period in advance and transfer tasks to idle nodes, effectively avoiding repair stagnation caused by sudden changes in node states and reducing the long-tail latency of full-node repair.

[0023] In some implementations, determining a second candidate node currently in a computational idle phase and migrating the repair task to be migrated from the first candidate node to the second candidate node includes: evaluating the repair capabilities of each candidate node currently in a computational idle phase in the candidate node set based on decoding throughput and download bandwidth in network resource status; sorting the candidate nodes currently in a computational idle phase in descending order of repair capability; and determining the candidate node with the higher ranking as the second candidate node to receive and execute the repair task to be migrated.

[0024] This solution assesses the recovery capability of the target node by comprehensively considering decoding throughput and download bandwidth, ensuring the effectiveness of task migration. It guarantees that tasks to be migrated are always assigned to the idle node with the strongest processing capacity in the current state, avoiding secondary bottlenecks caused by blind migration and further improving the overall throughput and stability of full node fault recovery.

[0025] The second aspect of this application relates to a compute-aware erasure coding repair system for analytical storage clusters, including a coordinator and multiple agent nodes. The agent nodes are deployed on worker nodes, co-located with storage nodes and data analysis executors. The agent nodes are used to monitor and report network resource status and compute resource status, and to perform data transmission and decoding tasks. The coordinator is used to maintain metadata of the encoded stripes, and in response to data block failures or node failures, executes the method described in the above scheme to generate a data transmission and decoding repair plan, and schedules the repair tasks to multiple agent nodes.

[0026] The aforementioned system achieves distributed deployment of computation-aware repair methods through a collaborative architecture of coordinator and agent nodes. The tight integration of agent nodes with the co-location environment ensures the real-time and accuracy of resource status acquisition, while the centralized scheduling of the coordinator ensures the effective execution of global optimization strategies. Together, they provide a highly efficient fault recovery infrastructure for the analytical storage cluster.

[0027] Furthermore, the modular design supports end-to-end automation from resource monitoring and model prediction to task scheduling. This system can adapt to various failure scenarios, including single-block, multi-block, and full-node failures, and can dynamically adjust its recovery strategies according to changes in analytical load, enhancing the self-healing capabilities and data service continuity of the storage cluster in complex production environments.

[0028] Thirdly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method described in the first aspect or any possible implementation thereof.

[0029] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0030] Fifthly, this application provides a computer program product that, when run on a processor, causes the processor to perform the method described in the first aspect or any possible implementation thereof.

[0031] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0032] Overall, the technical solutions conceived in this application have the following beneficial effects compared with the prior art: (1) This application incorporates decoding throughput into the repair scheduling model and uses the periodic characteristics of data analysis load for accurate prediction, thereby achieving joint perception of computing resources and network resources. It overcomes the one-sidedness of traditional solutions that only focus on network bandwidth, enabling repair decisions to adaptively avoid computing bottleneck nodes, and significantly improving the scheduling rationality and execution efficiency of repair tasks under the co-location architecture.

[0033] (2) By combining linear programming with the maximum flow algorithm, the abstract resource optimization objective is transformed into an executable sub-block-level repair plan. This method not only ensures the global optimality of task allocation in mathematics, but also ensures the feasibility of the generated transmission path in the actual network topology, realizing a seamless connection from theoretical optimization to engineering practice, and effectively reducing the overall time consumption of single-block and multi-block repair.

[0034] (3) Context reuse and proactive migration mechanisms were introduced for multi-block repair and full-node repair scenarios, respectively. Context reuse accelerates batch recovery by reducing redundant data transmission, while proactive migration avoids potential slow node risks by predicting node state changes. These two strategies complement the basic compute-aware scheduling, comprehensively improving the recovery performance and robustness of the storage cluster under different failure modes. Attached Figure Description

[0035] Figure 1 This is one of the flowcharts of the computation-aware erasure coding repair method for analytical storage clusters provided in the embodiments of this application.

[0036] Figure 2 This is a schematic diagram of the repair plan establishment provided in the embodiments of this application.

[0037] Figure 3 This is a schematic diagram of the architecture of a compute-aware erasure coding repair system for analytical storage clusters provided in an embodiment of this application. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0039] The embodiments of this application are described below with reference to the accompanying drawings. Figure 1 As shown, this embodiment provides a compute-aware erasure coding repair method for analytical storage clusters. This method is applied to a cluster environment where data analysis executors and storage nodes are co-located, aiming to optimize fault recovery efficiency by jointly using compute-aware computing and network resources. The method mainly includes the following steps S100-S600.

[0040] Step S100: A data block failure is detected in the target coding stripe. The coding parameters of the target coding stripe, the failed data block, and the set of candidate nodes storing available data blocks are determined. Specifically, when the coordinator receives a block loss report or node failure alarm, it first queries the stripe metadata to locate the location of the failure. Here, the coding stripe refers to an erasure coding unit composed of the same set of data blocks and parity blocks. For example, in the RS(k,m) coding scheme, a stripe contains k data blocks and m parity blocks. Simultaneously, the system needs to determine the coding parameters of the stripe (such as the k value, m value, block size S, etc.) and the specific identifier of the failed data block.

[0041] Based on this, the set of nodes storing available data blocks from the target coded stripe is further identified, and these nodes are defined as candidate nodes. Candidate nodes refer to physical servers, virtual machines, or containerized worker nodes that store valid data blocks or verification blocks required for repair and have the capability to participate in repair computation.

[0042] Step S200 involves acquiring the network and computing resource status of each candidate node in the candidate node set. Specifically, the coordinator sends resource query requests to the agent nodes on each candidate node to collect key indicators reflecting the node's current service capabilities in real time. The network resource status mainly includes the node's upload and download bandwidth, which can be obtained by reading the operating system's network counter or using dedicated network monitoring tools, characterizing the node's data transmission capabilities during the repair process. The computing resource status includes indicators such as CPU utilization and memory bandwidth usage, characterizing the node's remaining processing capacity when performing erasure coding decoding operations. In a co-location architecture, because data analysis tasks dynamically preempt computing resources, static hardware specifications alone cannot reflect the true repair capabilities; therefore, it is necessary to acquire this dynamic status information in real time. This step, by simultaneously collecting resource status in both network and computing dimensions, breaks through the limitation of traditional repair schemes that only focus on network bandwidth, laying a data foundation for achieving true computing-aware scheduling.

[0043] Step S300: Based on the periodic and phased computation patterns of the distributed data analysis workload running on the data analysis executor, and combined with the computing resource status, the decoding throughput of candidate nodes for data decoding is estimated. Specifically, in the analysis storage cluster, the distributed data analysis workload running in the foreground typically exhibits obvious iterative characteristics. Its computing resource usage is not random but shows regular phased changes as the iteration cycle progresses. This embodiment utilizes this characteristic to map the current computing resource status to the workload's running cycle, thereby predicting the actual throughput of candidate nodes available for decoding operations at the current moment. For example, if the current phase is a redistribution phase of the data analysis task, CPU and memory bandwidth may be heavily occupied, resulting in a correspondingly lower estimated decoding throughput; conversely, if the current phase is a computing idle period, the decoding throughput will be higher. This estimation method avoids prediction noise caused by relying solely on instantaneous resource snapshots and can more accurately reflect the true decoding capability of nodes within a short future time window, thus preventing the misscheduling of repair tasks to nodes about to enter a computationally intensive period.

[0044] Step S400: Based on the network resource status and decoding throughput of each candidate node, a repair time model is constructed to characterize network transmission time and decoding computation time. Specifically, the repair time model is a mathematical performance evaluation function that takes the upload bandwidth, download bandwidth, and decoding throughput estimated in step S300 of the candidate node as input parameters to calculate the network transmission time and decoding computation time required for the node to complete the specified repair task. The core value of this model lies in unifying heterogeneous resource indicators (bandwidth and throughput) into time indicators of the same dimension, enabling the system to intuitively compare the bottleneck differences of different nodes in the network and computation dimensions. In this way, the repair time model can accurately identify which nodes are network-constrained and which are computationally constrained, providing a quantitative basis for subsequent task balancing.

[0045] Step S500 determines the repair task allocation ratio for each candidate node based on the repair time model to minimize the maximum repair operation time among all candidate nodes. Specifically, this step uses the optimization idea of ​​minimizing the maximum time to determine the proportion of repair workload that each candidate node should undertake. The completion time of the entire repair pipeline depends on the slowest node; therefore, the optimization goal is not to pursue the shortest average time, but to equalize the completion times of all participating nodes as much as possible, eliminating the bottleneck effect. Based on the repair time model constructed in step S400, the system dynamically adjusts the collection and transmission ratios of each node: for nodes with strong decoding capabilities but low bandwidth, their data transmission tasks are appropriately reduced; for nodes with ample bandwidth but heavy computation, their decoding tasks are reduced. This adaptive task allocation mechanism ensures that no single node becomes the absolute bottleneck of the repair process, thereby making the overall repair performance approach the theoretical optimal value and improving the system's fault recovery speed under complex load environments.

[0046] Step S600 involves generating a data transmission and decoding repair plan based on the repair task allocation ratio, and scheduling corresponding candidate nodes to execute data transmission and decoding operations to recover the failed data block. Specifically, the allocation ratio determined in step S500 is a numerical solution, which is transformed into executable physical instructions in this step. The coordinator generates a detailed repair plan based on the allocation ratio, which specifies how much data each candidate node needs to receive from which source nodes, what decoding operations it needs to perform, and which target node it needs to send the results to. Subsequently, the coordinator distributes the repair plan to the corresponding proxy nodes, driving them to execute data transmission and decoding operations in parallel. Finally, the repair sub-blocks generated by each node are aggregated and reconstructed into a complete failed data block. This step completes the closed loop from optimization decision-making to actual execution, ensuring that the computation-aware scheduling strategy can be effectively implemented and shortening the fault recovery time.

[0047] Through steps S100 to S600, this embodiment establishes a complete computation-aware erasure coding repair process. This process incorporates decoding throughput into the repair decision model and utilizes the periodic characteristics of the analysis load for capability estimation, achieving joint awareness and collaborative optimization of computational and network resources. It not only solves the repair latency problem caused by neglecting computational bottlenecks in traditional co-location architectures but also ensures global load balancing through a Min-Max optimization mechanism.

[0048] In some embodiments, periodic and phased computation patterns are characterized by the data analysis iteration cycle and the time phase offset within the iteration cycle. Specifically, distributed data analysis workloads typically consist of multiple repeated iterations, with each iteration exhibiting highly similar phased characteristics in terms of computational resource consumption. The time phase offset refers to the time difference between the moment a decoding operation occurs and the start point of its corresponding data analysis iteration cycle. Unlike absolute timestamps, the time phase offset aligns decoded samples scattered across different iterations to the same standardized periodic coordinate system. For example, if an iteration cycle is 10 seconds, the time phase offset is 3 seconds regardless of whether the decoding operation occurs at the 3rd second of the 1st iteration or the 3rd second of the 100th iteration. This alignment mechanism effectively eliminates the cumulative time drift noise caused by the increasing number of iterations, enabling the system to identify the structural pattern that "the 3-second phase usually corresponds to the Shuffle stage, where resource contention is intense," thus laying the foundation for subsequent accurate predictions.

[0049] Based on this, the decoding throughput of the candidate nodes for data decoding is estimated. First, historical decoding profile records of the candidate nodes are obtained. Historical decoding profile records are structured data sets reflecting the decoding performance of a node at a specific point in the past. As an example, each historical decoding profile record can be represented as a quadruple. .in, Indicates the first The time phase offset relative to the start of the corresponding data analysis iteration cycle when the next historical decoding operation occurs; This indicates the CPU utilization at the start of the operation; This indicates the amount of memory bandwidth used at the start of the operation; This represents the actual decoding throughput for this operation. Actual decoding throughput It can be done through formula The calculation shows that, among which For block size, The number of blocks required to recover the failed data blocks. This represents the measured time taken for this decoding operation. These records constitute the training dataset for the prediction model, accurately reflecting the decoding capability boundaries of the node under different load stages.

[0050] Subsequently, a throughput prediction model is trained based on historical decoding profiles. This model is a function model that establishes a non-linear mapping between resource status and decoding performance. It can employ machine learning algorithms such as multiple linear regression, support vector regression (SVR), or multilayer perceptron (MLP). The model's input feature vector contains three dimensions: time phase offset, CPU utilization, and memory bandwidth usage; the output label is the corresponding decoding throughput. Through training, the model learns the implicit rule that, at the same time phase, higher CPU utilization corresponds to lower decoding throughput; and at the same CPU utilization, different time phases may correspond to different levels of memory bandwidth contention, leading to differences in decoding performance. This multi-dimensional feature fusion makes the prediction results more accurate and robust than relying solely on a single instantaneous indicator.

[0051] When a repair task arrives, the current time phase offset relative to the start of the data analysis iteration cycle and the current computing resource status are input into the throughput prediction model to obtain the expected decoding throughput of the corresponding candidate node. Specifically, when the coordinator needs to evaluate the repair capability of a candidate node, it first calculates the phase offset in the iteration cycle of the data analysis task currently running on that node, and reads the current CPU utilization and memory bandwidth usage in real time. Then, this set of real-time data is input into the trained throughput prediction model, and the model outputs the expected throughput of the node performing decoding operations in the current state. This process realizes a rapid transformation from historical experience to real-time decision-making, enabling the repair scheduler to perceive the dynamic changes in node computing capabilities within milliseconds, avoiding the allocation of tasks to nodes that are currently experiencing peak computing periods.

[0052] Furthermore, computing resource status includes CPU utilization and memory bandwidth usage. CPU utilization directly determines the execution speed of Galois domain operations, while memory bandwidth usage affects the efficiency of data transfer during decoding. During data analysis tasks, both often exhibit asynchronous fluctuations, therefore they must be monitored simultaneously to comprehensively characterize the node's performance status.

[0053] To address the issues of model cold start or sudden environmental changes, this embodiment employs an adaptive calibration mechanism. When historical decoded profile records are lacking or a change in the distributed data analysis load pattern is detected, multiple decoding micro-probe tasks are proactively executed within the initial iteration cycle of the data analysis iteration cycle. Initial decoded profile records are collected based on these micro-probe tasks, and the throughput prediction model is updated or retrained accordingly. Here, a decoding micro-probe task refers to a specially designed, lightweight decoding operation unit. Unlike normal repair decoding tasks, it does not carry out actual data recovery operations. Instead, it uses a pre-set small-scale test data block to execute a standard decoding algorithm. Its sole purpose is to measure the decoding time under the current environment to generate profile samples. Due to the use of small-scale data, the execution time of a single micro-probe task is extremely short (typically in the millisecond range), and its resource consumption and interference with normal data analysis operations are negligible.

[0054] The triggering conditions for decoding micro-probing tasks mainly include two scenarios: first, the system has just started or a node has just joined the cluster, and there are no historical decoding profile records locally, i.e., it is in a cold start state; second, the system detects a change in the type of data analysis task (such as switching from K-means to PageRank) or an adjustment in resource configuration, causing the original historical profile to no longer be applicable, i.e., it is in a pattern mutation state. Regarding execution timing, micro-probing tasks are limited to the initial iteration cycle of the data analysis iteration cycle. This is because the initial iteration usually contains the most complete stage characteristics of the task, and at this time the system load has not yet entered a steady-state oscillation period, making the collected samples more representative. In terms of the number of executions, the system will evenly inject multiple micro-probing tasks at different phase points within this initial cycle to ensure coverage of different stages such as computationally intensive periods and idle periods. Through this proactive probing mechanism, a baseline performance profile under the new environment can be built in a very short time, and the model parameters can be continuously fine-tuned based on subsequent real-time data collection, thereby ensuring that the throughput prediction model always maintains a high degree of consistency with the actual physical characteristics of the current system, enhancing the robustness and adaptability of the repair strategy under non-stationary load environments.

[0055] In some embodiments, network resource status includes upload bandwidth and download bandwidth, and the repair time model is used to calculate the decoding time, upload time and download time of candidate nodes. The repair operation time of any candidate node is the maximum value among the decoding time, upload time and download time of that candidate node.

[0056] Specifically, to achieve fine-grained control over the repair process, this embodiment transforms resource status into precise time metrics. For any candidate node... The repair operation is broken down into three sequential sub-processes: downloading the required data blocks from other nodes, performing decoding calculations locally, and uploading the recovered data or intermediate results to the requesting node or other collection nodes. Since these three sub-processes typically share a network interface card or memory bus at the hardware level and are logically dependent on each other, the time it takes for a node to complete the entire repair task depends on the slowest step. This maximum-value modeling approach accurately reflects the bottleneck effect in pipeline operations, ensuring that subsequent scheduling decisions optimize for the real bottlenecks rather than simply focusing on average performance.

[0057] Based on this, this embodiment further provides a specific mathematical expression for the repair time model. For candidate nodes... Decoding time Upload time and download time Calculate using the following formulas respectively:

[0058]

[0059]

[0060] in, The number of blocks required to recover the corrupted data blocks, i.e., the number of original data blocks in the erasure coding parameters; The size of a single data block; Candidate nodes The data collection ratio indicates the proportion of failed data blocks that the node is responsible for recovering out of the total amount of failed data. Candidate nodes The data transmission ratio indicates the proportion of data provided by this node as a data source to other collecting nodes relative to the size of a single block. Candidate nodes The decoding throughput is given by the prediction model in the aforementioned embodiments; and Candidate nodes Upload bandwidth and download bandwidth.

[0061] Upload time In the formula, the numerator term includes This is because when a node acts as both a sending node and a data provider to others (in proportion of...), it is a true data provider. As a collection node, it recovered some invalid data that needed to be reported (the proportion was...). When both data streams are in use, they both require upload bandwidth. This repair time model avoids the bandwidth estimation bias caused by separating send and receive traffic in traditional models, allowing it to accurately reflect the coupled competition between computing and network resources in a co-located architecture.

[0062] The process of determining the repair task allocation ratio of each candidate node based on the repair time model specifically includes: establishing a linear programming model with the optimization objective of minimizing the maximum value among the repair operation times of all candidate nodes; and determining the data collection ratio of each candidate node as a collection node and the data transmission ratio as a sending node by solving the linear programming model.

[0063] The physical meaning of this optimization objective is to pursue the shortest completion time of the global repair pipeline. In a parallel repair scenario, the entire repair process is only considered complete when all participating nodes have completed their respective tasks. Therefore, the system uses a linear programming solver to find an optimal set of... and This combination of nodes minimizes the repair time of the node with the longest processing time. This Min-Max optimization strategy automatically balances the load, allocating more tasks to nodes with fast computation and high bandwidth, while reducing the workload on slow nodes, thereby eliminating the drag on overall performance from local bottlenecks.

[0064] To ensure the correctness and feasibility of the repair logic, the linear programming model includes the following three constraints: the sum of the data collection ratios of all candidate nodes is 1, the sum of the data transmission ratios of all candidate nodes is... The sum of the data collection ratio and data transmission ratio of any candidate node does not exceed 1. The sum of the data collection ratios of all candidate nodes is 1, ensuring that failed data blocks are fully recovered without omission or redundancy; the sum of the data transmission ratios of all candidate nodes is... This stems from the principle of erasure coding decoding, which states that recovering any invalid block requires at least reading... The remaining nodes must provide a total of [number] available blocks, minus the one block that the collecting node itself may hold (if it stores one block of this stripe). The physical storage constraint is that the sum of the data collection ratio and the data transmission ratio of any candidate node does not exceed 1, because each node stores at most one block of data in the same stripe, and the amount of data it provides externally plus the amount of data it recovers cannot exceed its local storage capacity or the upper limit of its single processing capability. These three constraints together constitute a closed feasible region, ensuring that the solution obtained by linear programming is not only mathematically optimal, but also engineering-executable.

[0065] After obtaining the optimal task allocation ratio, it needs to be translated into specific transmission instructions. A data transmission and decoding repair plan is generated based on the repair task allocation ratio, including: constructing a flow network containing source, sink, sending node logical entities, and collection node logical entities; setting the edge capacity in the flow network according to the data collection and sending ratios of candidate nodes; solving the traffic allocation from the sending node logical entity to the collection node logical entity based on the flow network with set edge capacity, and converting the traffic allocation into sub-block transmission paths to generate the data transmission and decoding repair plan based on the sub-block transmission paths.

[0066] Specifically, linear programming gives... and The document only describes the workload share that each node should undertake, without specifying how data actually flows between nodes. Therefore, this embodiment introduces a maximum flow model to generate an executable scheduling scheme.

[0067] like Figure 2 As shown, the flow network is constructed as follows: First, a virtual source node and a virtual sink node are created. Then, two logical entities are created for each candidate node participating in the repair: a sending node logical entity and a collecting node logical entity. This is because in actual repair, the same physical node may simultaneously play the roles of data provider (sender) and data restorer (collector). If represented by only a single node, it will be impossible to distinguish between inflow and outflow data flows, and bandwidth constraints cannot be applied correctly. By splitting the physical node into two logical entities, the sending link and the receiving link can be clearly separated in the flow network.

[0068] Next, the edge capacity is set according to the ratio obtained from the linear programming: the edge capacity from the source node to each sending node logical entity is set to... This represents the maximum amount of data that the node can provide; the edge capacity from each collection node logical entity to the sink is set to... This represents the amount of data that the node needs to recover; the edge capacity between the sending node logical entity and the collecting node logical entity is set according to the actual network topology and bandwidth limitations. If the two nodes can communicate directly, the capacity is set to the theoretical maximum value; otherwise, it is set to 0.

[0069] Finally, a maximum flow algorithm (such as the Dinic algorithm or the Push-Relabel algorithm) is used to solve the flow allocation in the network. The actual flow value on each edge output by the algorithm corresponds to the specific sub-block transmission path and data volume. For example, if the flow on the edge from the sending node N1 to the collecting node N3 is 10MB, it means that N1 needs to transmit 10MB of sub-block data to N3. After receiving all the predetermined sub-blocks, the collecting node decodes them in conjunction with its local data and sends the recovery result to the requesting node. In this way, this embodiment transforms the mathematical optimization result into a sequence of atomic operations that the underlying system can directly execute, completing a complete closed loop from perception and decision-making to execution.

[0070] In some embodiments, when multiple failed data blocks exist in the same target coding stripe, after establishing a repair plan for the data transmission and decoding of the first failed data block, the candidate nodes selected to perform the decoding operation are grouped into a reuse node set. Specifically, in a distributed storage system, although the probability of two or more data blocks being lost simultaneously within a single stripe is low, once it occurs, the repair overhead is often several times that of single-block repair. The multi-block repair context refers to the available data block copies that have been read into the node's local memory during the repair of the first failed data block, as well as intermediate results generated during the decoding computation, such as the intermediate coefficient matrix of Galois field operations or partial decoding fragments. This data can usually be reused when repairing other failed blocks in the same coding stripe because different failed blocks within the same stripe often share the same available data source and some computational logic. The reuse node set refers to the physical nodes that have already assumed the role of collection nodes in the previous round of repair and still retain the aforementioned context in their local cache.

[0071] For subsequent failed data blocks, if the estimated processing time of candidate nodes in the reuse node set after reusing multiple blocks of repair context is less than the estimated processing time of enabling a new candidate node, then the decoding task of the subsequent failed data blocks will be preferentially assigned to a candidate node in the reuse node set. The core of this decision logic lies in dynamic comparison based on time cost, rather than simple static rules. The coordinator estimates the estimated time required for each node in the reuse node set to complete the new task while reusing the existing context. This estimate typically consists of two parts: first, the remaining computation time, which is significantly shorter than the full repair time due to skipping data download and some pre-computation; and second, the potential memory access overhead, as context reuse implies frequent access to specific memory regions. Simultaneously, the system also estimates the full time required to perform the task if a new candidate node not in the reuse set is selected, including downloading from the network. The system considers the transmission time of each data block and the complete decoding computation time. Reuse is only decided when the benefits (saving transmission and computation time) clearly outweigh its potential costs. This ensures that the repair strategy always evolves towards minimizing global time consumption, avoiding performance pitfalls caused by blindly pursuing reuse.

[0072] Otherwise, the allocation ratio of repair tasks for subsequent failed data blocks for each candidate node is re-determined based on the repair time model. When reuse is no longer the optimal solution, the system will automatically fall back to the general optimization process described in the previous embodiments, that is, re-collect resource status, predict throughput, and solve a new allocation scheme through linear programming. The reason for designing this fallback mechanism is that reuse does not always bring positive benefits in a co-location architecture. For example, although a node in the reuse node set has accumulated context in the previous round of repair, it may be in the computationally intensive stage of the data analysis task at the current moment, and its memory bandwidth is already heavily occupied. At this time, if a reuse operation is forcibly performed on this node, the dense memory access requests will cause fierce bandwidth competition with the background analysis task, resulting in a sharp drop in decoding throughput, and may even make the actual processing time longer than the time to start from scratch on a new node due to cache jitter. In addition, excessively concentrated reuse on a few nodes may also cause these nodes to overheat and cause new bottlenecks. Therefore, this embodiment effectively distinguishes the boundary between effective reuse and ineffective reuse by comparing time costs, ensuring the robustness of the system in complex dynamic environments.

[0073] In some embodiments, when a full node failure occurs and triggers multiple repair tasks involving different coding stripes, for a first candidate node already assigned repair tasks, its historical idle phase duration and current idle phase duration are used to assess whether it will transition to a compute-intensive phase. Specifically, a full node failure is one of the most extreme failure scenarios in a distributed storage cluster. When a physical node completely fails, multiple coding stripes belonging to all data blocks stored on that node will simultaneously trigger repair processes, resulting in a surge of concurrent repair tasks within the cluster. Under such high load pressure, if repair tasks are scheduled to nodes about to enter a compute-intensive period, it will lead to severe long-tail latency. This embodiment aims to mitigate this risk.

[0074] The historical idle period duration here refers to the average duration or distribution characteristics of the node's idle state over several data analysis iterations in the past, reflecting the periodic pattern of node load changes. The current idle period duration refers to the cumulative duration the node has remained idle within the current iteration cycle. By combining these two dimensions, it is possible to more accurately determine whether the node's current idle state is truly long-term idle or merely a brief gap between two computationally intensive phases. For example, if historical data shows that a node's computationally intensive period is from the 5th to the 8th second of each iteration, and the current time is the 4.9th second with the node idle for 0.9 seconds (close to the upper limit of the historical average gap), the system can proactively predict that the node will enter a busy state in 0.1 seconds. This predictive logic can be implemented using weighted average methods, linear trend extrapolation, or time-series-based classification models, thus overcoming the limitations of traditional monitoring that only reflects the current state and achieving accurate perception of the node's performance status within a short future time window.

[0075] If the assessment indicates that the first candidate node will transition to a compute-intensive phase, then the remediation tasks to be migrated are determined from those already assigned to that first candidate node. This means proactively offloading the load before the node's actual performance degrades, rather than passively responding after the node becomes a bottleneck. The specific remediation tasks to be migrated are not randomly selected, but rather follow specific screening principles to maximize migration benefits.

[0076] Specifically, the coordinator prioritizes tasks with a large remaining data transfer volume or those that have a critical path impact on the overall repair progress as migration targets. For example, if a repair task has completed 90% of its data transfer and only a small amount of decoding work remains, the context switching overhead of migration may outweigh the overhead of waiting for it to complete, making migration unsuitable. Conversely, if a task has just started and is expected to take a long time, migrating it to an idle node can significantly reduce the overall waiting time. Furthermore, tasks can be weighted and sorted based on their urgency (such as the access frequency of associated data) to ensure that high-priority tasks receive a more stable execution environment. Through this task selection mechanism, the system can minimize the additional overhead introduced by task migration itself while ensuring repair efficiency.

[0077] Furthermore, the process of identifying a second candidate node currently in a computationally idle phase and migrating the task to be migrated from the first candidate node to the second candidate node also includes the optimal ranking of target nodes. Specifically, in a batch repair scenario caused by a full node failure, multiple nodes may be in a computationally idle state simultaneously, but their repair capabilities are not the same. To avoid migrating tasks to a node that is idle but has extremely low network bandwidth or poor decoding performance, this embodiment establishes a multi-dimensional repair capability evaluation mechanism. This mechanism comprehensively evaluates the repair capabilities of each candidate node currently in a computationally idle phase in the candidate node set based on decoding throughput and download bandwidth in network resource status. Considering that erasure coding repair is a typical computationally and I / O-intensive mixed load, a single indicator cannot fully characterize the actual processing capabilities of a node. For example, a node may have an extremely powerful CPU but limited network card bandwidth, while another node may have ample bandwidth but insufficient decoding algorithm optimization; only nodes that possess both are ideal migration targets. In specific implementation, a repair capability scoring function can be constructed, such as... ,in For the predicted decoding throughput, For download bandwidth, and These are weighting coefficients that are dynamically adjusted based on the overall bottleneck of the current cluster. When the cluster is generally computationally constrained, the weighting coefficient is increased. When network congestion is severe, increase .

[0078] The system sorts candidate nodes currently in idle computing phases according to their repair capabilities from highest to lowest. The node with the highest ranking is designated as the second-ranked candidate node to receive and execute the repair tasks to be migrated. This sorting ensures that tasks to be migrated are always assigned to the idle node with the strongest processing capacity in the current environment, thus avoiding secondary bottlenecks caused by blind migration. When the resources of the highest-ranked node are insufficient to handle all tasks to be migrated, the system can select subsequent nodes according to the sorting results until all tasks to be migrated are properly assigned.

[0079] like Figure 3As shown, this embodiment provides a compute-aware erasure coding repair system for analytical storage clusters. This system provides the physical carrier and operating environment for the aforementioned method embodiments. The system includes a coordinator and multiple agent nodes. Specifically, the coordinator is typically deployed on the cluster's main management node or in a separate high-availability service container, responsible for global decision-making and task orchestration; while the multiple agent nodes are distributed across various worker nodes in the cluster, serving as the execution endpoints of repair instructions and the tentacles of resource awareness. This architecture, combining centralized decision-making with distributed awareness, ensures the consistency of the global optimization strategy while fully utilizing the local computing power of edge nodes, avoiding network congestion caused by converging all repair traffic to the central node.

[0080] Agent nodes are deployed on worker nodes, co-located with storage nodes and data analysis executors. Specifically, co-location means that the agent node, storage node (such as an HDFS DataNode), and data analysis executor (such as a SparkExecutor) run on the same physical server or the same operating system instance. This deployment method is the physical foundation for achieving compute awareness. In traditional split architectures, the remediation module often cannot directly know the resource usage of other applications on the same node, and can only infer indirectly through lagging network monitoring. However, in the co-location architecture of this embodiment, the agent node can directly read the actual CPU and memory bandwidth usage of the data analysis executor with microsecond-level latency through local inter-process communication or shared memory mechanisms. This zero-network-overhead local awareness capability enables the system to capture the millisecond-level resource fluctuation characteristics of the analysis load within the iteration cycle, thereby providing a high-quality data source for the accurate throughput prediction based on time phase offset in the aforementioned embodiment.

[0081] The agent node monitors and reports network and computing resource status, and performs data transmission and decoding tasks. Specifically, the agent node integrates a resource monitoring module, a repair module, and an encoding module. The resource monitoring module collects CPU utilization and memory bandwidth usage by calling the operating system's local interface, and obtains upload and download bandwidth through network socket statistics. Compared to traditional remote SNMP polling, this local sampling mechanism has extremely low overhead and avoids measurement noise caused by cross-network transmission. The collected resource status data is encapsulated into lightweight heartbeat packets and reported to the coordinator periodically or proactively when the status changes significantly. The repair module is responsible for performing specific sub-block reception, Galois domain decoding operations, and sending the result data according to the repair plan issued by the coordinator; the encoding module provides erasure coding support during data writing or reconstruction.

[0082] The coordinator maintains the metadata of the encoded stripe. In response to data block or node failures, it executes the computation-aware erasure coding repair method from any of the aforementioned method embodiments to generate a repair plan for data transmission and decoding, and schedules the repair tasks to multiple agent nodes. Specifically, the coordinator is logically divided into a COAR module and a scheduler. The COAR module (Compute-Oriented Adaptive Repair Module) is the core engine of computation-aware optimization. It is responsible for receiving resource status reports from each agent node, maintaining historical decoding profile records, training and updating the throughput prediction model, and running a linear programming solver to determine the optimal task allocation ratio. This module carries complex computational logic; separating it avoids optimization computation from interfering with real-time task distribution. The scheduler focuses on execution-level control. Based on the allocation ratio output by the COAR module, it constructs a maximum flow network model, generates specific sub-block transmission paths, and issues repair instructions containing source address, destination address, data offset, and check parameters to the corresponding agent nodes. Simultaneously, the scheduler is also responsible for tracking the execution progress of the repair tasks and handling timeout retries and exception recovery. This decoupled design enables the coordinator to maintain a high response speed in large-scale cluster environments. Even when dealing with a massive number of concurrent repair requests caused by full node failures, it ensures that the computation-aware optimization logic is not blocked, thereby guaranteeing the overall repair timeliness and stability of the system.

[0083] Based on the above embodiments, this embodiment provides a specific application scenario for computation-aware erasure coding repair of analytical storage clusters. In this scenario, the storage cluster adopts the RS(6,3) erasure coding scheme, that is, each stripe contains 6 data blocks and 3 parity blocks, and recovering any failed data block requires reading k=6 available blocks. The size S of a single data block is set to 64MB. The cluster foreground runs a typical distributed data analysis workload, specifically a K-means clustering algorithm job under the Spark framework. This job has significant iterative characteristics, with each iteration cycle containing different stages such as Map, Shuffle, and Reduce, resulting in regular periodic fluctuations in CPU utilization and memory bandwidth usage on each worker node.

[0084] During the execution of data analysis tasks, the proxy node periodically records CPU utilization and memory bandwidth usage, and actively executes several decoding micro-probing tasks in the initial stage to obtain an initial decoding profile.

[0085] When a block on node N6 fails, the coordinator determines that available blocks in the stripe containing the failed block are located on nodes N1, N2, N3, N4, N5, N7, N8, and N9. The coordinator collects estimated upload bandwidth, download bandwidth, and decoding throughput from these nodes. If a node has high network bandwidth but low decoding throughput, the linear programming model will reduce its collection ratio to prevent it from becoming a decoding bottleneck; conversely, if a node has high upload bandwidth, download bandwidth, and decoding throughput, it can be allocated a larger collection or transmission ratio.

[0086] The solution is obtained and Then, the coordinator constructs a flow network and uses the maximum flow algorithm to determine the specific transmission path. For example, if N1, N3, and N5 are selected as collection nodes, and they recover sub-blocks of the failed block at proportions of 0.4, 0.4, and 0.2 respectively, then the sending node transmits sub-blocks of the corresponding size to these collection nodes. After the collection nodes complete decoding, they send the recovered sub-blocks to the requesting nodes, which then concatenate them to obtain the complete failed block.

[0087] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A computation-aware erasure coding repair method for analytical storage clusters, characterized in that, The analytical storage cluster is a cluster where the data analysis executor and storage nodes are co-located, and the method includes: If a data block failure is detected in the target coding stripe, the coding parameters of the target coding stripe, the failed data block, and the candidate node set for storing available data blocks are determined, and the network resource status and computing resource status of each candidate node in the candidate node set are obtained. Based on the periodic and phased computing patterns of the distributed data analysis load running by the data analysis executor, and in conjunction with the computing resource status, the decoding throughput of the candidate nodes for data decoding is estimated. Based on the network resource status and decoding throughput of each candidate node, a repair time model is constructed to characterize network transmission time and decoding computation time. The repair task allocation ratio for each candidate node is determined based on the repair time model in order to minimize the maximum value of the repair operation time of all candidate nodes. A repair plan for data transmission and decoding is generated based on the repair task allocation ratio, and corresponding candidate nodes are scheduled to perform data transmission and decoding operations to recover the failed data block.

2. The method according to claim 1, characterized in that, The periodic and phased calculation pattern is characterized by the data analysis iteration cycle and the time phase offset within the iteration cycle; estimating the decoding throughput of the candidate nodes for data decoding includes: Obtain the historical decoding profile record of the candidate node. The historical decoding profile record includes the time phase offset of the historical decoding operation relative to the start of the corresponding data analysis iteration cycle, the historical computing resource status, and the actual decoding throughput. A throughput prediction model is trained based on the aforementioned historical decoded profile records; When the repair task arrives, the current time phase offset relative to the start of the data analysis iteration cycle and the current computing resource status are input into the throughput prediction model to obtain the expected decoding throughput of the corresponding candidate node.

3. The method according to claim 2, characterized in that, The computing resource status includes CPU utilization and memory bandwidth usage; the method further includes: When the historical decoded profile record is missing or a change in the pattern of the distributed data analysis load is detected, multiple decoding micro-probe tasks are actively executed within the initial iteration cycle of the data analysis iteration cycle. Based on the decoding micro-probe tasks, initial decoded profile records are collected, and the throughput prediction model is updated or retrained accordingly.

4. The method according to claim 1, characterized in that, The network resource status includes upload bandwidth and download bandwidth. The repair time model is used to calculate the decoding time, upload time and download time of candidate nodes. The repair operation time of any candidate node is the maximum value among the decoding time, upload time and download time of that candidate node. The step of determining the repair task allocation ratio for each candidate node based on the repair time model includes: A linear programming model is established with the optimization objective of minimizing the maximum value of the repair operation time of all candidate nodes; The data collection ratio of each candidate node as a collection node and the data transmission ratio as a sending node are determined by solving the linear programming model.

5. The method according to claim 4, characterized in that, For candidate nodes The decoding time The upload time The download time ; in, The number of blocks required to recover the failed data block. For block size, Candidate nodes Data collection ratio Candidate nodes Data transmission ratio Candidate nodes Decoding throughput, Candidate nodes upload bandwidth, Candidate nodes Download bandwidth; The constraints of the optimization objective include: the sum of the data collection ratios of all candidate nodes is 1, and the sum of the data transmission ratios of all candidate nodes is [missing information]. The sum of the data collection ratio and the data transmission ratio of any candidate node shall not exceed 1.

6. The method according to claim 4, characterized in that, The step of generating a data transmission and decoding repair plan based on the repair task allocation ratio includes: Construct a flow network that includes source nodes, sink nodes, sending node logical entities, and collecting node logical entities; The edge capacity in the stream network is set according to the data collection ratio and data transmission ratio of the candidate nodes; Based on the flow network with the set edge capacity, the flow allocation from the sending node logical entity to the collecting node logical entity is solved, and the flow allocation is converted into sub-block transmission paths to generate the data transmission and decoding repair plan based on the sub-block transmission paths.

7. The method according to claim 1, characterized in that, When multiple invalid data blocks exist in the same target coding stripe, the method further includes: After establishing a repair plan for the data transmission and decoding of the first failed data block, the candidate nodes that have been selected to perform the decoding operation are combined into a set of reuse nodes; For subsequent failed data blocks, if the estimated processing time of the candidate nodes in the reuse node set after reusing multiple block repair contexts is less than the estimated processing time of enabling new candidate nodes, then the decoding task of the subsequent failed data blocks will be preferentially assigned to the candidate nodes in the reuse node set. Otherwise, the repair task allocation ratio for each candidate node for the subsequent failed data block is re-determined based on the repair time model.

8. The method according to any one of claims 1 to 7, characterized in that, When a full node failure occurs, triggering multiple repair tasks involving different coding stripes, the method further includes: For the first candidate node that has been assigned the repair task, assess whether it will transition to a computationally intensive phase based on the historical idle phase duration and the current idle phase duration of the first candidate node. If the assessment indicates that the first candidate node will transition to a computationally intensive phase, then determine the repair tasks to be migrated from the repair tasks already assigned to the first candidate node. A second candidate node that is currently in a computational idle phase is identified, and the task to be migrated and repaired is migrated from the first candidate node to the second candidate node.

9. The method according to claim 8, characterized in that, The step of determining a second candidate node currently in a computationally idle phase and migrating the task to be migrated and repaired from the first candidate node to the second candidate node includes: Based on the decoding throughput and the download bandwidth in the network resource status, evaluate the repair capability of each candidate node in the candidate node set that is currently in the computing idle phase. The candidate nodes currently in the computational idle phase are sorted in descending order of their repair capabilities. The candidate node with the higher ranking is selected as the second candidate node to receive and execute the task to be migrated and repaired.

10. A computation-aware erasure coding repair system for analytical storage clusters, characterized in that, Includes a coordinator and multiple agent nodes; The proxy node is deployed on the worker node, co-located with the storage node and data analysis executor. The proxy node is used to monitor and report the network resource status and computing resource status, and to perform data transmission and decoding tasks. The coordinator is used to maintain metadata of the encoded stripe, and in response to data block failure or node failure, executes the method of any one of claims 1 to 9 to generate a data transmission and decoding repair plan, and schedules the repair tasks to the plurality of proxy nodes.

Images