Server cluster data exception repair method based on distributed task priority

Abstract

This invention belongs to the field of resource management technology. It discloses a method for repairing data anomalies in server clusters based on distributed task priorities. The method involves real-time monitoring of the running status of all distributed tasks in the server cluster, collecting task priority indicators and data anomaly events. A distributed task priority graph model is constructed, with tasks as nodes and dependencies as edges, based on task priority indicators and data anomaly events. The dynamic repair priority score for each data anomaly event is calculated using this graph model, and a repair task queue is generated based on the dynamic repair priority score. Adaptive resource allocation is performed on the repair task queue, taking into account the real-time load distribution of the server cluster, to generate an optimal resource allocation scheme. Based on the resource allocation scheme, repair operations are initiated and executed on designated computing nodes, thus realizing data anomaly repair for server clusters based on distributed task priorities.

Description

Technical Field

[0001] This invention relates to the field of resource management technology, and more specifically, to a method for repairing data anomalies in server clusters based on distributed task priorities. Background Technology

[0002] With the rapid development of cloud computing and big data technologies, server clusters, as the core infrastructure of distributed computing, are widely used in enterprise-level data centers, financial transaction systems, and online service platforms. These clusters handle massive data requests through task distribution and load balancing mechanisms, achieving high availability and scalability. Traditional server cluster data anomaly repair methods mainly rely on redundant backups, checksum algorithms, and automated retry mechanisms. These methods can detect and repair data corruption, loss, or inconsistency, helping to maintain system stability and data integrity.

[0003] However, existing server cluster data anomaly repair technologies cannot dynamically allocate repair resources based on task priority when handling distributed tasks. This leads to delays in the repair of high-priority tasks, severely impacting overall system performance. Specifically, traditional methods and existing technologies generally employ static or uniform resource scheduling strategies, ignoring the priority differences between tasks (such as critical business tasks and background maintenance tasks). During peak cluster load periods, abnormal data from high-priority tasks may be preempted by low-priority tasks, resulting in congestion in the repair queue and amplified latency. Furthermore, existing methods lack a priority-aware dynamic adjustment mechanism, failing to assess task impact in real time and reorder the repair sequence. Therefore, the lack of priority management exacerbates resource waste, with low-priority tasks excessively occupying computing nodes while the repair window for high-priority tasks is compressed. This leads to increased overall system response time and significantly reduced fault recovery efficiency, hindering the reliability and efficiency of server clusters in high-concurrency commercial scenarios.

[0004] In view of this, the present invention proposes a server cluster data anomaly repair method based on distributed task priority to solve the above problems. Summary of the Invention

[0005] To overcome the aforementioned shortcomings of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a server cluster data anomaly repair method based on distributed task priority, comprising: Step S1: Monitor the running status of all distributed tasks in the server cluster in real time, and collect task priority indicators and data anomaly events; Step S2: Using tasks as nodes and dependencies as edges, construct a distributed task priority graph model based on task priority metrics and data anomaly events; Step S3: Calculate the dynamic repair priority score for each data anomaly event using a distributed task priority graph model, and generate a repair task queue based on the dynamic repair priority score; Step S4: Based on the real-time load distribution of the server cluster, adaptively allocate resources to the repair task queue to generate the optimal resource allocation scheme. Step S5: Based on the resource allocation plan, start and execute the repair operation on the specified computing node.

[0006] Preferably, the task priority indicators include the task's business urgency and dependency weights; data anomaly events include the location of data corruption and the severity of the anomaly.

[0007] Preferably, the method for constructing a distributed task priority graph model includes: The task priority metrics are quantified and encoded to generate a priority vector for each task, where the priority vector integrates business urgency and dependency weights. Data anomaly events are mapped to corresponding nodes, and the impact factor of the event on the node is calculated based on the severity of the anomaly and the location of the data corruption. Use the dependency weights as the edge weights; A distributed task priority graph model with a low-dimensional representation is generated through graph embedding processing.

[0008] Preferably, the method for generating a low-dimensional representation of the distributed task priority graph model through graph embedding processing includes: The Node2Vec algorithm is applied to process the nodes, and a parameterized random walk strategy is used to capture the graph structure information composed of dependency weights, generating a low-dimensional vector for each node. The dimensions of the low-dimensional vector are preset, and cluster analysis is performed on the low-dimensional vector to identify a set of task nodes with similar priority propagation characteristics, and the correlation between the set and the business urgency of the task is verified. The validated set of low-dimensional vectors is used as a low-dimensional representation of the distributed task priority graph model.

[0009] Preferably, the method for calculating the dynamic repair priority score for each data anomaly event using a distributed task priority graph model includes: In the dependency structure defined by the graph model, a random walk is applied to simulate the propagation path of task priority and calculate the influence diffusion value of each node. The initial priority score is calculated by combining the impact diffusion value with the severity of the corresponding data anomaly event. The time decay coefficient is calculated based on the occurrence time of the data anomaly event, and multiplied by the initial priority score to generate a dynamic repair priority score; The dynamic repair priority scores of all data anomaly events are normalized and sorted to form a priority-driven repair task sequence.

[0010] Preferably, the method for calculating a time decay coefficient based on the occurrence time of the data anomaly event and multiplying it by the initial priority score to generate a dynamic repair priority score includes: The time decay coefficient is defined as an exponential decay function to obtain the total time delay from the occurrence of a data anomaly event to its monitoring. The output value of this function decreases monotonically as the total delay of the data anomaly event increases. Set the decay rate parameter of the exponential decay function according to the required time sensitivity requirements; Set a maximum latency threshold. If the total latency of abnormal data events exceeds this threshold, the decay rate can be increased by adjusting the decay rate parameter.

[0011] Preferably, the method for forming a priority-driven repair task sequence includes: Based on dynamic repair priority scores, all data anomaly events are divided into multiple priority queues, with high-scoring data anomaly events placed in the priority processing queue. Embed dependency constraint rules in the queue; The length of each queue is dynamically monitored. If the length exceeds the preset threshold, queue compression is performed by merging data anomaly events with adjacent data corruption locations or similar data anomaly types. Output the compressed repair task queue.

[0012] Preferably, the method for adaptively allocating resources to the repair task queue based on the real-time load distribution of the server cluster to generate an optimal resource allocation scheme includes: Collect real-time load metrics for each computing node in the server cluster; A resource allocation optimization mechanism is constructed, and the objective function is set as maximizing the sum of the dynamic repair priority scores of all tasks in the repair task queue, with the real-time load status of each computing node as the constraint. A genetic algorithm is applied to solve the resource allocation optimization mechanism, which prioritizes the allocation of high-priority repair tasks to low-load nodes, thereby generating a resource allocation scheme.

[0013] Preferably, the method for applying a genetic algorithm to solve the resource allocation optimization mechanism, and preferentially allocating high-priority repair tasks to low-load nodes to generate a resource allocation scheme includes: Initialize the population with a set of randomly generated resource allocation schemes; Define a fitness function, which integrates the total dynamic repair priority score and the load balancing of cluster nodes in the fitness function fusion scheme; The selection, crossover, and mutation operations are performed iteratively on the population, an elite retention strategy is introduced, and the highest fitness scheme is retained in each generation; When the population evolution reaches the convergence condition, the current optimal solution is output as the final resource allocation scheme.

[0014] Preferably, the method for initiating and executing the repair operation on the designated computing node includes: According to the resource allocation plan, a parallel repair thread is started on the corresponding computing node to perform repair operations based on error correction coding on the data corruption location; Real-time collection of feedback data during the repair process, including the actual execution time of the repair task and the repair success rate; Analyze the abnormal patterns in the feedback data. When the repair success rate is consistently lower than the preset threshold, mark the corresponding abnormal event as a secondary abnormal event and feed it back to the dynamic repair priority score calculation process.

[0015] The technical effects and advantages of this invention's server cluster data anomaly repair method based on distributed task priority are as follows: By introducing a dynamic priority-driven mechanism, the core problems of traditional methods, such as increased delays in repairing high-priority tasks, resource contention, and SLA default risks caused by static resource scheduling, are solved.

[0016] By constructing a distributed task priority graph model, the urgency of tasks, dependencies, and anomaly impact factors are quantitatively integrated, solving the problem that traditional methods cannot perceive differences in task importance and achieving accurate assessment of the impact of anomalies.

[0017] By using a dynamic repair priority scoring mechanism that combines impact diffusion value, anomaly severity, and time decay factor, the repair priority is dynamically adjusted in real time, ensuring that recent anomalies with high impact and high severity are given priority processing and reducing the delay in repairing high-priority tasks.

[0018] By using adaptive resource allocation and parallel repair, high-priority tasks are intelligently scheduled to low-load nodes based on genetic algorithms, and closed-loop learning is formed by combining feedback data to improve repair throughput and resource utilization.

[0019] It effectively prevents resource contention and latency amplification during the repair process, ensuring the data integrity and system stability of critical business operations, and providing reliable anomaly repair guarantees for server clusters in high-concurrency scenarios. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the server cluster data anomaly repair method based on distributed task priority according to the present invention; Figure 2This is a schematic diagram of the method for solving the resource allocation optimization mechanism using a genetic algorithm in this invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Please see Figure 1 and Figure 2 In this embodiment of the invention, a method for repairing data anomalies in a server cluster based on distributed task priorities includes: Step S1: Monitor the running status of all distributed tasks in the server cluster in real time, and collect task priority indicators and data anomaly events; Step S2: Using tasks as nodes and dependencies as edges, construct a distributed task priority graph model based on task priority metrics and data anomaly events; this graph model can intuitively display the priority, dependencies, and impact of data anomaly events of all tasks.

[0023] Step S3: Calculate the dynamic repair priority score for each data anomaly event using a distributed task priority graph model, and generate a repair task queue based on the dynamic repair priority score; Step S4: Based on the real-time load distribution of the server cluster (such as the CPU utilization and memory usage of each node), adaptive resource allocation is performed on the repair task queue. The goal is to allocate high-priority repair tasks to nodes with lower current load as much as possible, thereby generating the optimal resource allocation scheme. Step S5: According to the resource allocation plan, initiate and execute the repair operation on the designated computing nodes. A computing node refers to a physical or virtual server entity that constitutes a server cluster; it is the carrier of hardware resources. It is an independent server that actually performs computations, runs tasks, and provides computing resources such as CPU and memory. It is not the same as a node in a graph model.

[0024] The task priority indicators include the business urgency of the task (e.g., real-time transaction tasks are high urgency, batch processing tasks are low urgency) and dependency weight (i.e., the dependency relationship and influence strength between tasks); data anomaly events include the location of data corruption where the anomaly occurs and the severity of the anomaly (e.g., complete data loss is high severity, partial verification failure is medium severity).

[0025] It's important to note that the task's urgency is directly retrieved. When any task is created and submitted to the cluster, its priority label is explicitly specified through development specifications or scheduling configuration. Therefore, the urgency can be collected simply by automatically reading this predefined metadata when the task is entered into the database. Dependency weights are derived from topology analysis of the entire workflow. Distributed tasks do not run in isolation; their dependencies are already defined and recorded in a unified workflow configuration file or through service call chain data. Therefore, dependency weights can be directly retrieved by parsing these existing dependency network graphs between tasks.

[0026] When data storage verification fails, or when an application encounters an error while reading or processing data, the resulting error reports and logs will inevitably contain the precise coordinates of the failure point (such as file path and data block ID). Therefore, these can be collected directly through listening and retrieving. Anomaly severity is a real-time assessment based on impact. It observes the direct consequences of the anomaly through a real-time monitoring dashboard, i.e., its impact on cluster task status and service health. Then, a simple rule engine (e.g., causing core service interruption = highest severity) directly completes the quantitative assessment.

[0027] As mentioned in the background technology, traditional server cluster data anomaly repair methods cannot dynamically allocate repair resources based on task priority. This is because traditional methods cannot quantify and assess the urgency and dependencies of tasks, lack priority awareness, and suffer from resource contention issues. Specifically, high-priority tasks may have their repair resources preempted by low-priority tasks during peak load periods. Furthermore, they ignore the dependencies between tasks and do not consider the cascading effects of anomalies through task dependency chains. Therefore, by constructing a distributed task priority graph model, the originally scattered task attributes, dependencies, and data anomaly events are uniformly encoded into a computable graph structure, providing a mathematical model foundation for subsequent dynamic priority calculation. Graph embedding processing solves the computational complexity problem of high-dimensional graph data, ensuring the real-time performance of the method in large clusters.

[0028] The method for constructing a distributed task priority graph model includes: The task priority index is quantified and encoded to generate a priority vector for each task. The priority vector integrates business urgency and dependency weights. Specifically, business urgency is mapped from text labels (such as high, medium, low) to numerical values ​​(for example, defining a scale of 1-10, with high mapped to 9, medium to 5, and low to 1). Similarly, for dependency weights, for example, a task depended on by 10 downstream tasks may have a weight of 10; another task depends on only 1 task and has a weight of 1. Then, the ratio of the current weight to the maximum weight in the cluster is calculated and normalized to the interval [0,1]. This yields the normalized business urgency E and dependency weight D. Then, based on the emphasis of E and D, for example, if business urgency is more important, then w1+w2=1 and w1>w2, the priority vector P=[w1×E,w2×D]. Thus, each task T_i obtains a unique priority vector P_i.

[0029] Data anomaly events are mapped to corresponding nodes, and the impact factor of the event on the node is calculated based on the severity of the anomaly and the location of the data corruption. Specifically, when a data anomaly event is detected, the event carries the location of the data corruption (e.g., file path F: / data / file.parquet). First, a pre-set data lineage table or task metadata database is queried to find all tasks that read or depend on this data location. The data anomaly event is then used as an attribute and associated with the corresponding node of each task. The impact factor I = S × f(L) is calculated, where S represents the severity of the anomaly, which is determined by a pre-set rule engine. For example, 1.0 is for complete data loss, 0.7 is for data verification errors, etc. f(L) represents the assessment of the distance of the corruption point from the current task. If the anomaly occurs on the data file directly processed by the task, then f(L) = 1. If the anomaly occurs in the upstream data source of the task, then f(L) = 1 ÷ (d + 1), where d is the shortest hop count from the task to the corrupted data source on the data lineage graph. The greater the distance, the greater the attenuation of the impact, ensuring that anomalies with a large impact are processed first.

[0030] Dependency weights are used as edge weights; this visualizes tasks and their relationships as a graph data structure. In this graph, each task T_i is a node, with its priority vector P_i and the currently associated impact factor I_i (I_i=0 if there is no anomaly). If the output of task T_a is the input of task T_b, i.e., T_b depends on T_a, then a directed edge is created from T_a to T_b. The edge weight is the dependency weight D. Thus, a distributed task priority graph model containing static attributes (task priority, dependency strength) and dynamic states (anomaly impacts) is constructed.

[0031] A low-dimensional distributed task priority graph model is generated through graph embedding to capture the dynamics of priority propagation between tasks. Specifically, based on the constructed graph model, the Node2Vec algorithm is controlled to walk along high-weight edges. When generating paths during random walks, the probability of selecting the next node is proportional to the edge weight. That is, starting from the current task node, the probability of reaching the next task node with a higher dependency weight is greater. Then, a large number (e.g., hundreds or thousands) of random walk paths are generated for each node in the graph. These paths record the patterns of frequent common visits between tasks. High-weight edges represent strong dependencies, perfectly capturing the graph structure information composed of dependency weights.

[0032] Using the large number of random walk paths obtained in the previous step as input, the Skip-gram model (a type of Word2Vec) is used to predict the neighboring nodes around a task node in the walk path. After training, the weight matrix of the model is the low-dimensional vector representation of each node.

[0033] K-Means clustering is used to group all the generated low-dimensional vectors of nodes. After clustering, tasks belonging to the same cluster are similar in structure and influence in the graph structure because their low-dimensional vectors are close to each other in the mathematical space. This is also known as similarity priority propagation feature. For example, a cluster may contain all the key tasks that are in the core hub position and are depended on by a large number of downstream tasks.

[0034] Calculate the average business urgency of all tasks within each cluster and observe whether there are significant differences in average business urgency between different clusters. For example, the average business urgency of cluster 1 (core hub tasks) is 9.2, the average business urgency of cluster 2 (normal processing tasks) is 5.1, and the average business urgency of cluster 3 (edge ​​backend tasks) is 1.5. If the cluster priority ranking is consistent with the average business urgency ranking, the validation passes, proving that this graph embedding model is successful and reliable. If the validation fails, backtracking and adjustment are required. Through the above processing, not only static business labels can be reflected, but also dynamic structural influences can be captured.

[0035] The method for generating a low-dimensional representation of a distributed task priority graph model through graph embedding processing includes: The Node2Vec algorithm is applied to process the nodes, and a parameterized random walk strategy is used to capture the graph structure information composed of dependency weights, generating a low-dimensional vector for each node. The dimension of the low-dimensional vector is preset (generally within one-tenth of the original graph node size), and cluster analysis is performed on the low-dimensional vector to identify a set of task nodes with similar priority propagation characteristics, and the correlation between the set and the business urgency of the task is verified. The validated set of low-dimensional vectors is used as a low-dimensional representation of the distributed task priority graph model for subsequent priority calculation and model updates.

[0036] To address the shortcomings of static resource scheduling in the background technology, which leads to repair delays and a lack of timeliness awareness, this solution quantifies the global influence of a task in the dependency network by calculating the impact diffusion value through random walks, enabling the automatic identification of core tasks with high dependency weights. By fusing the impact diffusion value and anomaly severity, a multiplicative evaluation model is established to ensure that anomalies with both high global impact and high local damage receive the highest initial priority, thus preventing low-priority anomalies from preempting resources at the source. An exponential decay mechanism with a time decay coefficient is introduced, and an automatic degradation rule of "the older, the more delayed" is established through timeliness sensitivity parameters and maximum delay thresholds, completely eliminating the blocking effect of old anomalies on new critical anomalies.

[0037] The method for calculating the dynamic repair priority score for each data anomaly event using a distributed task priority graph model includes: In the dependency structure defined by the graph model, random walks are applied to simulate the propagation path of task priorities and calculate the influence diffusion value of each node. The purpose is to quantify the global influence of a node in the dependency network (that is, the network formed by the dependency structure in the graph model). The higher the influence diffusion value of a node, the more tasks its anomaly affects through the dependency relationship.

[0038] Specifically, the random walk algorithm simulates a "messenger" starting from a specific node in the graph. At each step, it randomly selects a connecting edge (i.e., a dependency) to move to the next node. The probability of selecting an edge is proportional to its weight (the higher the weight, the greater the probability of being selected). At each step, this "messenger" has a fixed probability (e.g., 0.15) to jump back to the starting node to restart. Numerous such random walks are initiated for each node in the graph model that contains anomalies (i.e., the source node). The walk paths simulate the process of the influence (or priority) of the anomaly spreading outwards along the dependency network from the source node.

[0039] After multiple iterations, the algorithm converges. At this point, for each source node, a stable probability distribution vector can be obtained. Each value in this vector represents the probability that a "messenger" originating from the source node will appear at any other node in the graph in steady state. This probability value is the diffusion value of the source node's influence on that node. The final influence of a source node can be expressed as the sum of its probabilities of reaching all other nodes, or as the weighted sum of its probabilities of reaching all important nodes in the graph (such as nodes with high business urgency). The specific method is chosen based on the actual application. This process transforms the abstract "influence" into a concrete, computable value, facilitating subsequent use.

[0040] The impact diffusion value is combined with the severity of the corresponding data anomaly (i.e., multiplied, because multiplication can effectively amplify data anomalies that have both high impact and high severity) to calculate the initial priority score S_in; The time decay coefficient is calculated based on the occurrence time of the data anomaly event, and multiplied by the initial priority score to generate the dynamic repair priority score S_dy=S_in×C_time(t); The dynamic repair priority scores of all data anomaly events are normalized, mapping all scores to the range of [0,1], eliminating the bias caused by the absolute value. The normalized scores are sorted from high to low to form a priority-driven repair task sequence. The task at the top of the sequence is the highest priority anomaly with a wide impact, great destructive power, and the latest occurrence time, and is therefore given priority in resource allocation for repair.

[0041] The method for calculating a time decay coefficient based on the occurrence time of data anomalies and multiplying it by an initial priority score to generate a dynamic repair priority score includes: The time decay coefficient is defined as an exponential decay function C_time(t)=e^(-λ×t), which obtains the total time t from the occurrence of a data anomaly event to its monitoring. The output value of this function decreases monotonically as the total time of the data anomaly event increases. Set the decay rate parameter λ of the exponential decay function according to the required time sensitivity requirements (set by the administrator according to the required time sensitivity requirements; the larger the value of λ, the faster the score decays over time). Set a maximum latency threshold T_max (e.g., 24 hours). If the total latency of data anomalies exceeds this threshold, the decay rate parameter is adjusted to increase the decay rate. That is, a larger decay rate parameter λ' (e.g., λ'=2×λ) is automatically used to recalculate its C_time(t). This process will drastically reduce the decay coefficient of old anomalies, thereby significantly reducing their S_dy, ensuring that repair resources can be allocated to newly occurring and more urgent anomalies.

[0042] The effectiveness of the adjustment mechanism was verified through simulation testing to ensure that the generated dynamic repair priority score is sensitive to the timeliness of anomalies.

[0043] The method for forming a priority-driven repair task sequence includes: Based on dynamic repair priority scores, all data anomaly events are divided into multiple priority queues, with high-scoring data anomaly events placed in the priority queue. Specifically, fixed score thresholds are set to create queues with different priorities. For example: the emergency queue has a dynamic repair priority score ≥ 0.8, the high-priority queue has a dynamic repair priority score ≤ 0.5 < 0.8, and the normal queue has a dynamic repair priority score < 0.5. Each data anomaly event is then distributed to the corresponding priority queue based on its score.

[0044] Dependency constraint rules are embedded in the queues. These rules stipulate that if a task (called a "dependency") is a prerequisite for the normal operation of another higher-priority task (called a "dependent"), then even if the initial dynamic score of the "dependency" is low, its repair order must precede that of the "dependent," ensuring that tasks with higher dependency weights take precedence over their dependencies. The length of each queue is dynamically monitored. If the length exceeds a preset threshold (e.g., 20 for the emergency queue and 50 for the high-priority queue), queue compression is triggered. This compression is achieved by merging data anomaly events with adjacent data corruption locations or similar data anomaly types. Specifically, for merging adjacent data corruption locations, the system checks whether there are consecutive or shared small storage units (e.g., multiple replicas of the same HDFS data block, adjacent sectors on the same disk, or adjacent data pages of the same database table) on the physical storage or logical address. If so, these adjacent data anomaly events are merged into a new repair task. For example, "Repair Block A" and "Repair Block B" are merged into "Repair Consecutive Block AB Region."

[0045] For merged data anomalies of similar types, check if the root causes are consistent (e.g., all caused by "disk sector read / write errors" or "inconsistent data checksums due to network jitter"). If so, merge these anomalies of the same type and trigger a unified repair strategy (e.g., "restart the disk node and perform a full checksum" or "batch regenerate the data checksums for this batch"). The dynamic repair priority score of the new repair task generated after merging will be set based on the highest score among the merged anomalies to ensure that merging does not unduly reduce their urgency.

[0046] Output a compressed repair task queue to ensure that the queue reflects real-time priority changes.

[0047] The method for adaptively allocating resources to the repair task queue based on the real-time load distribution of the server cluster to generate the optimal resource allocation scheme includes: Collect real-time load metrics for each computing node in the server cluster; A resource allocation optimization mechanism is constructed (the design idea is to repair the most critical anomalies as quickly as possible without overwhelming any individual node). The objective function is set as maximizing the sum of the dynamic repair priority scores of all tasks in the repair task queue, with the real-time load status of each computing node as a constraint. Specifically, the objective function is... ,in, The dynamic repair priority score represents the repair task v. It is a decision variable (0 or 1). =1 indicates that the repair task v will be assigned to compute node j for execution; =0 means no allocation. This indicates the total number of tasks to be repaired. This function represents the total number of computing nodes in the cluster. Its purpose is to address the issue of "high-priority task repair delay" in the background technology.

[0048] The constraints based on the real-time load status of each compute node include: each repair task can be assigned to at most one compute node for execution. This prevents the same task from being executed repeatedly on multiple nodes, resulting in resource waste and data inconsistency; and the sum of the resource requirements of all tasks assigned to any given node cannot exceed the current actual available resource capacity of that node, preventing node overload, performance fluctuations, or even crashes due to improper allocation, thereby ensuring the stability of the entire cluster during the repair period.

[0049] A genetic algorithm is applied to solve the resource allocation optimization mechanism, which prioritizes the allocation of high-priority repair tasks to low-load nodes, thereby generating a resource allocation scheme. The system verifies the effectiveness of the allocation scheme in real time. When load changes cause the current resource allocation scheme to fail, it triggers a recalculation and allocation of the scheme to ensure the adaptability of resource allocation.

[0050] The method for applying a genetic algorithm to solve the resource allocation optimization mechanism, which prioritizes the allocation of high-priority repair tasks to low-load nodes, and generates a resource allocation scheme includes: The population is initialized with a set of randomly generated resource allocation schemes. Specifically, each individual (i.e., a resource allocation scheme) is represented by an array of length M, where M is the number of tasks to be repaired. The value j of the i-th element in the array (ranging from 1 to N, where N is the number of nodes) indicates that task i is assigned to node j for execution. A computing node is randomly selected for each task, thereby encoding the scheme and creating a population containing dozens to hundreds of distinct initial schemes.

[0051] Define a fitness function that combines the total dynamic repair priority score and the cluster node load balancing in the scheme; that is, the fitness function is a weighted sum of the sum of the dynamic repair priority scores and the cluster node load balancing. The cluster node load balancing is obtained by using the reciprocal of the standard deviation of the cluster node load rate, where load rate = allocated resources ÷ total resources.

[0052] The population is iteratively selected (using roulette wheel selection), crossover, and mutation operations (both of which are routine operations in genetic algorithms). An elite preservation strategy is introduced (in each generation of evolution, the top k individuals with the highest fitness (e.g., the top 10%) are directly retained to the next generation to ensure that the known optimal solution is not lost), and the solution with the highest fitness is retained in each generation. When the population evolution reaches the convergence condition (e.g., reaching the maximum number of iterations), the individual with the highest fitness in the current population is selected as the final resource allocation scheme, that is, the current optimal scheme is output as the final resource allocation scheme.

[0053] The method for initiating and executing the repair operation on the specified computing node includes: According to the resource allocation scheme, parallel repair threads are started on the corresponding computing nodes to perform repair operations based on error correction coding at the data corruption locations. Specifically, on the target computing node, a repair thread pool (with a pre-defined repair method) is created and managed by the repair management service. For each repair task assigned to that node, an independent execution thread is allocated from its thread pool. Each repair thread calls a pre-defined repair processor for the "data corruption location" it is responsible for. The processor reads the still healthy data copy or uses the redundant information of error correction coding (such as Reed-Solomon codes, LRC, etc.) to recalculate or recover the corrupted data block and write it back to the correct storage location. Real-time collection of feedback data during the repair process, including the actual execution time of the repair task and the repair success rate; the failure criteria for repair can include abnormal termination or timeout of the repair thread, write failure or verification error returned by the storage system, and the inability of the error correction code to recover the damaged data (insufficient redundant data); a status monitoring point is set within each repair thread to record the start time, end time and final status code of the thread execution, and to capture any exceptions or error information thrown by the thread. After the repair operation is completed, storage integrity verification is triggered to obtain the verification and validation results of the data blocks, and to record the write confirmation status returned by the storage to determine whether it is successful. The repair success rate = (number of successfully repaired tasks ÷ total number of attempted repair tasks) × 100%.

[0054] Analyze abnormal patterns in the feedback data (e.g., trend changes in repair success rate). When the repair success rate is consistently below a preset threshold, the corresponding abnormal event is marked as a secondary abnormal event and fed back to the dynamic repair priority score calculation process. For example, a higher severity (e.g., 0.9) is assigned to the secondary abnormal event, or when the repair capability of a specific node is detected to be declining, subsequent repair tasks are automatically redirected to healthy nodes.

[0055] The feedback data from all repair tasks is aggregated to generate a systematic repair performance log, which is used for subsequent iterative updates of the distributed task priority graph model.

[0056] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0057] It should be noted that all formulas in this manual are calculated by removing dimensions and taking their numerical values. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0058] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for repairing data anomalies in a server cluster based on distributed task priority, characterized in that, include: Step S1: Monitor the running status of all distributed tasks in the server cluster in real time, and collect task priority indicators and data anomaly events; Step S2: Using tasks as nodes and dependencies as edges, construct a distributed task priority graph model based on task priority metrics and data anomaly events; Step S3: Calculate the dynamic repair priority score for each data anomaly event using a distributed task priority graph model, and generate a repair task queue based on the dynamic repair priority score; Step S4: Based on the real-time load distribution of the server cluster, adaptively allocate resources to the repair task queue to generate the optimal resource allocation scheme. Step S5: Based on the resource allocation plan, start and execute the repair operation on the specified computing node.

2. The server cluster data anomaly repair method based on distributed task priority as described in claim 1, characterized in that, The task priority indicators include the task's business urgency and dependency weight; data anomaly events include the location of data corruption and the severity of the anomaly.

3. The server cluster data anomaly repair method based on distributed task priority according to claim 2, characterized in that, The method for constructing a distributed task priority graph model includes: The task priority metrics are quantified and encoded to generate a priority vector for each task, where the priority vector integrates business urgency and dependency weights. Data anomaly events are mapped to corresponding nodes, and the impact factor of the event on the node is calculated based on the severity of the anomaly and the location of the data corruption. Use the dependency weights as the edge weights; A distributed task priority graph model with a low-dimensional representation is generated through graph embedding processing.

4. The server cluster data anomaly repair method based on distributed task priority according to claim 3, characterized in that, The method for generating a low-dimensional representation of a distributed task priority graph model through graph embedding processing includes: The Node2Vec algorithm is applied to process the nodes, and a parameterized random walk strategy is used to capture the graph structure information composed of dependency weights, generating a low-dimensional vector for each node. The dimensions of the low-dimensional vector are preset, and cluster analysis is performed on the low-dimensional vector to identify a set of task nodes with similar priority propagation characteristics, and the correlation between the set and the business urgency of the task is verified. The validated set of low-dimensional vectors is used as a low-dimensional representation of the distributed task priority graph model.

5. The server cluster data anomaly repair method based on distributed task priority according to claim 4, characterized in that, The method for calculating the dynamic repair priority score for each data anomaly event using a distributed task priority graph model includes: In the dependency structure defined by the graph model, a random walk is applied to simulate the propagation path of task priority and calculate the influence diffusion value of each node. The initial priority score is calculated by combining the impact diffusion value with the severity of the corresponding data anomaly event. The time decay coefficient is calculated based on the occurrence time of the data anomaly event, and multiplied by the initial priority score to generate a dynamic repair priority score; The dynamic repair priority scores of all data anomaly events are normalized and sorted to form a priority-driven repair task sequence.

6. The server cluster data anomaly repair method based on distributed task priority according to claim 5, characterized in that, The method for calculating a time decay coefficient based on the occurrence time of data anomalies and multiplying it by an initial priority score to generate a dynamic repair priority score includes: The time decay coefficient is defined as an exponential decay function to obtain the total time delay from the occurrence of a data anomaly event to its monitoring. The output value of this function decreases monotonically as the total delay of the data anomaly event increases. Set the decay rate parameter of the exponential decay function according to the required time sensitivity requirements; Set a maximum latency threshold. If the total latency of abnormal data events exceeds this threshold, the decay rate can be increased by adjusting the decay rate parameter.

7. The server cluster data anomaly repair method based on distributed task priority according to claim 6, characterized in that, The method for forming a priority-driven repair task sequence includes: Based on dynamic repair priority scores, all data anomaly events are divided into multiple priority queues, with high-scoring data anomaly events placed in the priority processing queue. Embed dependency constraint rules in the queue; The length of each queue is dynamically monitored. If the length exceeds the preset threshold, queue compression is performed by merging data anomaly events with adjacent data corruption locations or similar data anomaly types. Output the compressed repair task queue.

8. The server cluster data anomaly repair method based on distributed task priority according to claim 7, characterized in that, The method for adaptively allocating resources to the repair task queue based on the real-time load distribution of the server cluster to generate the optimal resource allocation scheme includes: Collect real-time load metrics for each computing node in the server cluster; A resource allocation optimization mechanism is constructed, and the objective function is set as maximizing the sum of the dynamic repair priority scores of all tasks in the repair task queue, with the real-time load status of each computing node as the constraint. A genetic algorithm is applied to solve the resource allocation optimization mechanism, which prioritizes the allocation of high-priority repair tasks to low-load nodes, thereby generating a resource allocation scheme.

9. The server cluster data anomaly repair method based on distributed task priority according to claim 8, characterized in that, The method for applying a genetic algorithm to solve the resource allocation optimization mechanism, which prioritizes the allocation of high-priority repair tasks to low-load nodes, and generates a resource allocation scheme includes: Initialize the population with a set of randomly generated resource allocation schemes; Define a fitness function, which integrates the total dynamic repair priority score and the load balancing of cluster nodes in the fitness function fusion scheme; The selection, crossover, and mutation operations are performed iteratively on the population, an elite retention strategy is introduced, and the highest fitness scheme is retained in each generation; When the population evolution reaches the convergence condition, the current optimal solution is output as the final resource allocation scheme.

10. The server cluster data anomaly repair method based on distributed task priority according to claim 9, characterized in that, The method for initiating and executing the repair operation on the specified computing node includes: According to the resource allocation plan, a parallel repair thread is started on the corresponding computing node to perform repair operations based on error correction coding on the data corruption location; Real-time collection of feedback data during the repair process, including the actual execution time of the repair task and the repair success rate; Analyze the abnormal patterns in the feedback data. When the repair success rate is consistently lower than the preset threshold, mark the corresponding abnormal event as a secondary abnormal event and feed it back to the dynamic repair priority score calculation process.

Images