A method, system, and storage medium for distributed task failover

Abstract

This invention relates to the field of fault takeover, and more particularly to a method, system, and storage medium for distributed task fault takeover. The invention calculates survival characteristics parameters by setting survival evaluation indicators based on the dirty page ratio and the number of resource handles. For tasks meeting the conditions for continued operation, a continueable identifier is generated. For continueable tasks, an innovative dynamic matching strategy for hot data storage nodes is constructed. Combined with elastic control of the synchronization time window based on the survival characteristics parameters, the real-time nature and integrity of the takeover requirement data are ensured. Furthermore, a load evaluation value is obtained through multi-dimensional analysis of the incremental data ratio and IO pressure index. The load evaluation value is combined with the time sequence to determine the takeover time point, and the continueable threshold is adaptively adjusted based on the hot data storage node occupancy ratio. This invention improves fault handling efficiency, ensures fault takeover reliability, and increases storage utilization.

Description

Technical Field

[0001] This invention relates to the field of fault takeover, and more particularly to a method, system, and storage medium for distributed task fault takeover. Background Technology

[0002] With the widespread application of distributed computing architectures, the number of nodes in large-scale task clusters is growing exponentially, posing a severe challenge to the traditional operation and maintenance model. In distributed task scheduling scenarios, a large number of task nodes and massive amounts of task data work together. The failure of a single node may trigger a chain of anomalies through the task chain, causing the entire business chain to collapse. Therefore, fault handling methods for distributed computing systems are crucial.

[0003] For example, Chinese Patent Publication No. CN111193759A discloses a distributed computing system, method, and device, relating to the field of communication technology. This distributed computing system includes: a cloud management platform, multiple service nodes, and backup service nodes. The cloud management platform is communicatively connected to each service node and the backup service nodes. Each service node deploys corresponding data packets, and the backup service nodes deploy full data, including all data packets. When a service node fails, the cloud management platform routes the tasks to be processed by the failed service node and the business data required to process those tasks to the backup service node, enabling the backup service node to take over the tasks to be processed by the failed service node based on the full data and business data. This reduces takeover latency by saving time spent loading data packets from the cache.

[0004] However, the following problems still exist in the existing technology:

[0005] Existing technologies lack a dynamic sustainability assessment system for task execution, particularly lacking real-time joint analysis of dirty page ratio and resource handle count. This makes it impossible to distinguish whether a task's state is suitable for resumption during fault recovery, forcing the adoption of a uniform restart or archiving strategy. This results in the forced interruption of business continuity for resumable tasks, while unsuitable tasks waste storage space. Consequently, this leads to long task interruption times, low efficiency in reprocessing faulty tasks, and problems of wasted computing power and inefficient use of storage resources. Summary of the Invention

[0006] To address this, the present invention provides a method, system, and storage medium for distributed task fault takeover, which overcomes the problems in the prior art where, during fault recovery, it is impossible to distinguish whether a task's status meets the conditions for continued operation, forcing the adoption of a unified restart strategy or an archiving strategy, resulting in the forced interruption of business continuity for tasks that can continue to run, leading to long task interruption times and low efficiency in reprocessing faulty tasks.

[0007] To achieve the above objectives, in one aspect, the present invention provides a method for distributed task fault takeover, comprising:

[0008] In response to the execution of a task by a task node, a persistence analysis is performed on the task node, including analyzing the cache to determine the proportion of dirty pages and determining the number of resource handles corresponding to the task.

[0009] Based on the dirty page ratio and the number of resource handles, calculate the persistence characteristics of the task, and combine them with the persistence threshold to determine whether to set a persistence flag for the task.

[0010] For the task's reproducibility flag, a hot data storage node is matched for the task node to perform caching operations on the task node, including determining a synchronization time window based on the persistence parameter and storing the takeover requirement data of the task node within the synchronization time window through the hot data storage node.

[0011] In response to a task node anomaly, takeover is initiated based on the task being executed by that task node having a continue-run flag set, including:

[0012] Determine the takeover load characteristics corresponding to several check time points within the synchronization time window, sort each check time point based on the takeover load characteristics, select the takeover time point, obtain the takeover requirement data corresponding to the takeover time point, and send it to the available nodes for task takeover.

[0013] The takeover load characteristics include the incremental data ratio of the takeover demand data for each node and the IO pressure index. The continued operation threshold needs to be adjusted based on the occupancy ratio of each hot data storage node.

[0014] Furthermore, in response to a task node anomaly, the system takes over and restarts the task node because the task being executed by the task node has not been set with a continue-run flag.

[0015] Furthermore, the process of calculating the persistence characteristic parameter based on the dirty page ratio and the number of resource handles includes,

[0016] The ratio of the dirty page ratio in the processor cache to the dirty page threshold is determined as the dirty page parameter;

[0017] The ratio of the number of resource handles to the handle threshold is determined as the handle parameter;

[0018] The survival characterization parameter is obtained by weighted summation of the dirty page parameter and the handle parameter.

[0019] Furthermore, based on the continuation threshold, it is determined whether a continuation flag is set for the task, including:

[0020] If the persistence parameter of the task is less than the preset persistence threshold, then a persistence flag is set for the task.

[0021] Furthermore, a synchronization time window is determined based on the persistence characteristic parameter, and the takeover requirement data of the task node within the synchronization time window is stored through the hot data storage node, wherein,

[0022] The length of the determined synchronization time window is positively correlated with the survival characterization parameter;

[0023] Within the synchronization time window, several equally spaced inspection time points are determined;

[0024] The end time of the synchronization time window is the current time.

[0025] Furthermore, the takeover load characteristics corresponding to several inspection time points within the synchronization time window are determined, including:

[0026] Obtain takeover requirement data corresponding to several inspection time points within the synchronization time window;

[0027] For each inspection time point, the change in the corresponding takeover demand data of the task in the hot data storage node between adjacent inspection time points is obtained, and the proportion of the change to the total takeover demand data in the reference period is calculated as the incremental data ratio.

[0028] Collect the read and write operation frequency and latency of task nodes to generate an IO pressure index;

[0029] The incremental data ratio and IO pressure index were identified as characteristics of the takeover load.

[0030] Further, the inspection time points are sorted based on the aforementioned takeover load characteristics to select the takeover time point, including:

[0031] Input the characteristics of the takenover load into the preset load assessment model to generate load evaluation values;

[0032] The sorted sequence is obtained by arranging the load evaluation values ​​in descending order;

[0033] A predetermined proportion of sequence segments are extracted from the beginning of the sorted sequence. The sequence segments are then rearranged in ascending order based on the time length of the inspection time point corresponding to each load evaluation value in the sequence segment relative to the current time, to obtain the final sorted sequence.

[0034] The inspection time point corresponding to the first end of the final sorted sequence is selected as the takeover time point.

[0035] Furthermore, the continued running threshold needs to be adjusted based on the occupancy ratio of each of the hot data storage nodes, wherein,

[0036] The continued running threshold is positively correlated with the occupancy ratio of hot data storage nodes;

[0037] The occupancy ratio of each hot data storage node is the proportion of the number of hot data storage nodes whose storage occupancy rate exceeds a preset occupancy threshold to the total number of hot data storage nodes.

[0038] Secondly, the present invention also provides a storage medium storing a computer program, which, when executed by a computer system, can be used to perform a distributed task fault takeover method.

[0039] Thirdly, the present invention also provides a computer system, comprising:

[0040] A computing unit consisting of one or more processors;

[0041] One or more memory units;

[0042] and one or more programs,

[0043] The one or more programs are configured to be executed by one or more processors, and the memory includes the storage medium.

[0044] Compared to existing technologies, this invention calculates sustainability parameters by setting sustainability evaluation indicators based on the dirty page ratio and the number of resource handles. It generates a sustainable identifier for tasks that meet the sustainability criteria and innovatively constructs a dynamic matching strategy for hot data storage nodes for sustainable tasks. Combined with the sustainability parameter, it flexibly controls the synchronization time window to ensure the real-time performance and integrity of takeover request data. Furthermore, it obtains a load evaluation value through multi-dimensional analysis of incremental data ratio and IO pressure index. This load evaluation value is combined with time sequence considerations to determine the takeover time point, and the sustainability threshold is adaptively adjusted based on the hot data storage node occupancy ratio. Therefore, this improves fault handling efficiency, ensures reliable fault takeover, and increases storage utilization.

[0045] In particular, this invention considers the dirty page ratio and the number of resource handles as sustainability evaluation indicators. In practice, the dirty page ratio directly reflects the frequency of changes to the data to be synchronized in the processor cache, while the number of resource handles can quantify the complexity of the task's dependence on external resources. When the dirty page ratio increases sharply during task execution, it indicates that the state data is frequently updated. High dirty pages lead to the inability to obtain a consistent memory image, which can easily lead to recovery errors after takeover. Frequent updates cause an explosive increase in the amount of synchronized data, which can easily cause takeover congestion. A surge in the number of resource handles indicates that the dependency relationship has become more complex, increasing the difficulty of takeover. Furthermore, the combined effect of these two factors affects the sustainability of the task. Therefore, this invention calculates sustainability parameters to characterize the sustainability of the task, providing a basis for setting sustainability flags in the future, facilitating adaptive takeover in the future, thereby improving fault handling efficiency, ensuring fault takeover reliability, and improving storage utilization.

[0046] In particular, this invention identifies the reproducibility of tasks and matches hot data storage nodes to task nodes. In practice, distributed processing systems include multiple task nodes. Therefore, considering the reproducibility of tasks executed by different nodes, hot data storage nodes are matched only for task nodes with reproducibility tags, and caching operations are performed to improve the utilization rate of hot data storage nodes.

[0047] In particular, this invention considers the key characteristics of state changes during task execution and selects the persistence representation parameter as the basis for determining the synchronization time window. In practice, a high persistence representation parameter indicates frequent task state changes, requiring an extended synchronization time window to accommodate more check points and ensure the complete capture of the state of high-change tasks; conversely, a low persistence representation parameter shortens the synchronization window to avoid redundant checkpoints in low-change tasks, reducing computational waste and improving storage utilization.

[0048] In particular, this invention addresses the task's reproducibility flag, accurately selecting the takeover time and acquiring corresponding takeover requirement data when a task needs to be taken over due to abnormal situations such as task node failure. In practice, the available node responsible for takeover, based on the acquired takeover requirement data, can effectively restore and continue the task's computational state without having to re-execute the entire computation process from the initial state, thereby improving fault handling efficiency and effectively reducing the waste of computing resources.

[0049] In particular, this invention analyzes load evaluation values ​​from two dimensions: incremental data ratio and IO pressure index. The incremental data ratio reflects the transmission overhead required for state synchronization by measuring the proportion of data change between adjacent checkpoints; the IO pressure index characterizes the load on task nodes by obtaining the frequency and latency of read / write operations. In fault takeover scenarios, a high incremental data ratio necessitates avoiding large-scale data transmission, while a high IO pressure index requires delaying takeover operations. By inputting the incremental data ratio and IO pressure index into a preset load assessment model, a comprehensive consideration of the takeover data is achieved, thereby selecting the optimal checkpoint for takeover, preventing cascading failures caused by storage node overload, ensuring takeover reliability, and improving fault handling efficiency.

[0050] In particular, this invention considers adjusting the continue running threshold based on the hot data storage node occupancy ratio. When the node occupancy ratio is lower than the critical value, the continue running threshold is reduced and the number of continue running flags is increased to make full use of idle storage resources and avoid idle computing power. When the occupancy ratio is higher than the critical value, the continue running threshold is increased and the continue running conditions are tightened to prevent the system stability from being reduced due to insufficient storage resources under high load. This dynamic threshold mechanism achieves a synergistic improvement in storage resource utilization and system stability. Attached Figure Description

[0051] Figure 1 The method for distributed task fault takeover in an embodiment of the invention can be illustrated with a diagram of the steps for taking over a continuing task.

[0052] Figure 2 This is a logic block diagram illustrating task takeover based on a continue-running flag, as described in an embodiment of the invention.

[0053] Figure 3 A logic block diagram for determining whether a task can be continued based on a continued running threshold, as shown in the embodiment of the invention;

[0054] Figure 4 The following is a logic block diagram showing that the continued running threshold for this embodiment of the invention needs to be adjusted based on the occupancy ratio of each hot data storage node. Detailed Implementation

[0055] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0056] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0057] Please see Figure 1The diagram illustrates the steps of a distributed task fault takeover method according to an embodiment of the present invention. The distributed task fault takeover method according to an embodiment of the present invention includes:

[0058] Step S1: In response to the task node executing the task, perform a persistence analysis on the task node, including analyzing the cache to determine the proportion of dirty pages and determining the number of resource handles corresponding to the task.

[0059] Step S2: Calculate the persistence characteristics of the task based on the dirty page ratio and the number of resource handles, and combine them with the persistence threshold to determine whether to set a persistence flag for the task.

[0060] Step S3: For the task's reproducibility flag, match a hot data storage node for the task node to perform caching operations on the task node, including determining the synchronization time window based on the persistence parameter and storing the takeover requirement data of the task node within the synchronization time window through the hot data storage node.

[0061] Step S4: In response to a task node exception, takeover is initiated based on the continue-run flag of the task being executed by the task node, including:

[0062] Determine the takeover load characteristics corresponding to several inspection time points within the synchronization time window, sort each inspection time point based on the takeover load characteristics and the inspection time point, select the takeover time point, obtain the takeover requirement data corresponding to the takeover time point and send it to the available nodes for task takeover.

[0063] The takeover load characteristics include the incremental data ratio of the takeover demand data for each node and the IO pressure index. The continued operation threshold needs to be adjusted based on the occupancy ratio of each hot data storage node.

[0064] Specifically, in implementation, several hot data storage nodes are set up to form a hot data storage node set for caching operations of multiple task nodes, which will not be elaborated further.

[0065] Specifically, there are no restrictions on the methods for obtaining the dirty page ratio and the number of resource handles. The dirty page ratio measures the proportion of cached pages in memory that have been modified but not written back to disk. A resource handle is an abstract access credential given to a process by the operating system or runtime environment. In essence, it is a reference identifier for system resources (such as files, networks, devices, etc.). Existing dirty page detection tools and resource handle detection tools can be used to determine the dirty page ratio and the number of resource handles, which will not be elaborated further here.

[0066] Specifically, takeover requirement data refers to various computational data generated when a task node processes task data, which allows subsequent task nodes taking over the task to continue executing the task based on the previous computation process. This includes data such as register states, processor cache data, memory mappings, task progress, and resource handles. Those skilled in the art can adjust the scope of the takeover requirement data according to the type of computational task, the operating system of the computer system, etc., which will not be elaborated further here.

[0067] Please see Figure 2 As shown, Figure 2 This is a logic block diagram illustrating task takeover based on a continue-running flag, according to an embodiment of the invention. Specifically, in implementation, different takeover methods are used depending on whether a task node has a continue-running flag set.

[0068] In response to a task node anomaly, takeover is initiated based on the task being set to a continue running flag. Then, takeover load characteristics corresponding to several check time points within the synchronization time window are determined. Based on the takeover load characteristics, each check time point is sorted to select a takeover time point. Takeover requirement data corresponding to the takeover time point is obtained and sent to available nodes for task takeover.

[0069] In response to a task node anomaly, take over the task node and restart it if the task being executed by the task node is not set to a continue-run flag.

[0070] Specifically, the process of calculating persistence parameters based on the dirty page ratio and the number of resource handles includes:

[0071] The ratio of the proportion of dirty pages in the processor cache to the dirty page threshold is defined as the dirty page parameter;

[0072] The ratio of the number of resource handles to the handle threshold is determined as the handle parameter;

[0073] The dirty page parameter and the handle parameter are weighted and summed to obtain the survival characterization parameter.

[0074] Specifically, the dirty page threshold is predetermined. The proportion of dirty pages corresponding to the execution of tasks without any abnormalities during the historical period is recorded in advance. The average dirty page proportion is calculated to reflect the dirty page proportion under normal conditions. The purpose of setting the dirty page threshold is to characterize the situation where the dirty page proportion is high. Therefore, the average dirty page proportion is appropriately amplified to determine the dirty page threshold. In implementation, the dirty page threshold is set to 1.3 times the average dirty page proportion.

[0075] When modern processors employ a write-back strategy for caching, writing dirty pages back to main memory requires memory bandwidth. When the proportion of dirty pages is high, the bandwidth contention caused by write-back operations leads to a significant decrease in cache hit rate, consequently extending task execution cycles. A high proportion of dirty pages causes frequent cache misses; even if a task is marked as executable, it may fail to continue due to incomplete data during takeover.

[0076] Specifically, the handle threshold is positively correlated with the allocatable computing quota of the task node. The handle threshold is determined based on the base handle threshold and the additional handle threshold. A preset mapping relationship between the computing quota of the job node and the handle threshold is established in advance. The base handle threshold corresponding to each 1 CPU core quota is selected within the range of [60, 120], and the additional handle threshold corresponding to each 1 GB memory quota is selected within the range of [15, 25].

[0077] Determine the number of CPU cores and memory quota for each task node, and then determine the sum of the basic handle threshold and the sum of the appended handle threshold. The sum of the basic handle threshold and the sum of the appended handle threshold is then defined as the handle threshold. For example,

[0078] For example, a task node is configured with 8 CPU cores, 32GB of memory, a CPU quota of 100 per CPU core, and an append handle threshold of 20 per GB of memory quota.

[0079] The total base handle threshold is calculated to be 800, the total appended handle threshold is calculated to be 640, and the handle threshold is determined to be 1440.

[0080] Those skilled in the art can adjust the basic handle threshold and the append handle threshold according to the single-core performance of the processor and the memory read / write speed in the computer system, so as to make reasonable use of the performance of the processor and memory and avoid performance waste or congestion due to insufficient performance. This will not be elaborated further.

[0081] In practice, the dirty page ratio directly reflects the frequency of changes to the data to be synchronized in the processor cache. When the dirty page ratio increases sharply during task execution, it indicates that the status data is updated frequently, which has a significant impact on fault takeover. Therefore, the weighting weight of the dirty page parameter is increased, and the weighting weight of the dirty page parameter is set to 0.55, while the weighting weight of the handle parameter is 0.45.

[0082] This invention considers the dirty page ratio and the number of resource handles as sustainability evaluation indicators. In practice, the dirty page ratio directly reflects the frequency of changes to the data to be synchronized in the processor cache, while the number of resource handles can quantify the complexity of the task's dependence on external resources. When the dirty page ratio increases sharply during task execution, it indicates that the state data is frequently updated. High dirty pages lead to the inability to obtain a consistent memory image, which can easily cause recovery errors after takeover. Frequent updates cause an explosive increase in the amount of synchronized data, which can easily cause takeover congestion. A surge in the number of resource handles indicates that the dependency relationship has become more complex, increasing the difficulty of takeover. Furthermore, the combined effect of these two factors affects the sustainability of the task. Therefore, this invention calculates sustainability parameters to characterize the sustainability of the task, providing a basis for setting sustainability flags in the future, facilitating adaptive takeover, thereby improving fault handling efficiency, ensuring fault takeover reliability, and improving storage utilization.

[0083] Please see Figure 3 As shown, Figure 3 The following is a logic block diagram for determining whether a task can be continued by combining a continuation threshold in an embodiment of the invention.

[0084] Specifically, the determination of whether to set a continue-running flag for a task is based on the continue-running threshold, including:

[0085] If the task's persistence parameter is less than the preset continuation threshold, then a continuation flag is set for the task.

[0086] Specifically, the initial continuation threshold is predetermined. The survival representation parameters corresponding to each task node's execution without any abnormalities within a historical period are pre-statistically calculated. The mean of the survival representation parameters is then calculated to reflect the survival representation parameters corresponding to the task nodes under normal conditions. The mean of the survival representation parameters is appropriately amplified to reflect situations where the survival representation parameters are too large. In implementation, it is set to 1.45 times the mean of the survival representation parameters.

[0087] Specifically, a synchronization time window is determined based on the persistence characteristic parameter, and the takeover requirement data of the task nodes within the synchronization time window is stored through hot data storage nodes.

[0088] The length of the determined synchronization time window is positively correlated with the persistence representation parameter;

[0089] Within the synchronization time window, there are several equally spaced inspection time points;

[0090] The end time of the synchronization time window is the current time.

[0091] In implementation, optionally, the length of the synchronization time window is dynamically determined based on the persistence representation parameter, specifically including the following steps:

[0092] Set the base time window. The default base time window length is selected within the range of 300 milliseconds to 800 milliseconds, with a typical value of 500 milliseconds.

[0093] When the persistence representation parameter is in the range [0, 0.6k), the length of the synchronization time window is the base length;

[0094] When the persistence representation parameter is in the range of [0.6k, 0.75k), the synchronization time window is extended to 1.2 to 1.5 times the base length;

[0095] When the persistence representation parameter is in the range [0.75k, k), the synchronization time window is extended to 1.6 to 2.5 times the base length;

[0096] Where k represents the continuation threshold, and the base length is 1000ms.

[0097] In practice, preferably, the interval corresponding to the detection time point can be set to 0.1 times the synchronization time window.

[0098] The start time of the synchronization time window is the point in time that the window length is calculated backward from the current time, and the end time is always the current time.

[0099] It is understandable that the values ​​at both ends of the synchronization time window will change continuously as the current moment changes, and the data outside the synchronization time window that corresponds to the takeover requirement can be deleted to free up storage space.

[0100] This invention considers the key characteristics of state changes during task execution and selects the persistence representation parameter as the basis for determining the synchronization time window. In practice, a high persistence representation parameter indicates frequent task state changes, requiring an extended synchronization time window to accommodate more check points and ensure the complete capture of the state of high-change tasks. Conversely, a low persistence representation parameter shortens the synchronization window to avoid redundant checkpoints in low-change tasks, reducing computational waste and improving storage utilization.

[0101] Specifically, the takeover load characteristics corresponding to several inspection time points within the synchronization time window are determined, including...

[0102] Obtain takeover requirement data corresponding to several inspection time points within the synchronization time window;

[0103] Obtain the change in takeover demand data corresponding to the previous inspection time point that is relatively adjacent to the inspection time point, and calculate the proportion of the change to the total amount of takeover demand data in the reference period as the incremental data ratio.

[0104] Collect the frequency and latency of read and write operations on task nodes to generate an IO pressure index.

[0105] Specifically, the end time of the reference period is the current time. The purpose of setting the reference period is to only consider the total amount of takeover demand data within a certain period of time. The reference period is determined based on the length of the synchronization time window and is set to twice the synchronization time window.

[0106] Specifically, at the inspection time point, the read and write operation frequency and read and write operation latency data of the task node storage device are collected in real time. The read and write operation frequency is divided by a preset frequency threshold to obtain the frequency ratio value, and the latency is divided by a preset latency threshold to obtain the latency ratio value. The frequency ratio value and the latency ratio value are weighted and summed to obtain the IO pressure index.

[0107] In implementation, the contributions of frequency proportion and delay proportion need to be considered comprehensively. Therefore, the weighting weights for both frequency proportion and delay proportion are set to 0.5.

[0108] The incremental data ratio and IO pressure index were identified as characteristics of the takeover load.

[0109] Specifically, the preset frequency threshold is determined based on the type of storage device to fully utilize its performance. In the exemplary configuration, for NVMe SSD devices, it is 75% of the device's nominal maximum IOPS value; for SATA SSD devices, it is 65% of the nominal maximum IOPS value; and for mechanical hard drives, it is 50% of the nominal maximum IOPS value.

[0110] Specifically, the preset latency threshold is determined based on the maximum latency requirement specified in the Service Level Agreement (SLA), and 85% of the maximum latency specified in the SLA is taken as the latency threshold.

[0111] This invention analyzes load evaluation values ​​from two dimensions: incremental data ratio and IO pressure index. The incremental data ratio reflects the transmission overhead required for state synchronization by measuring the proportion of data change between adjacent checkpoints. The IO pressure index characterizes the load on task nodes by obtaining the frequency and latency of read and write operations. In failover scenarios, a high incremental data ratio necessitates avoiding large-scale data transmission, while a high IO pressure index requires delaying failover operations. By inputting the incremental data ratio and IO pressure index into a preset load evaluation model, a comprehensive assessment of failover data is achieved, preventing cascading failures caused by storage node overload, shortening task interruption time during failures, and improving fault handling efficiency.

[0112] Specifically, the inspection time points are sorted based on the characteristics of the takeover load in order to select the takeover time point, including,

[0113] Input the characteristics of the takenover load into the preset load assessment model to generate load evaluation values;

[0114] The sorted sequence is obtained by arranging the load evaluation values ​​in descending order;

[0115] A predetermined proportion of sequence segments are extracted from the beginning of the sorted sequence. The sequence segments are then rearranged in ascending order based on the time length of the inspection time point corresponding to each load evaluation value in the sequence segment relative to the current time, resulting in the final sorted sequence.

[0116] The inspection time point corresponding to the first end of the final sorted sequence is selected as the takeover time point;

[0117] It is understandable that the first end of the final sorting sequence is the first load evaluation value in the sorting sequence, and each inspection time point corresponds to a load evaluation value. Therefore, the corresponding inspection time point can be determined as the takeover time point.

[0118] In implementation, the load assessment model is a weighted summation model, specifically a weighted sum of the IO pressure index and the incremental data ratio. The weight of the incremental data ratio is determined based on the statistics of historical fault takeover events. In the historical operating data of the system, the amount of incremental data change is the main factor affecting the transmission load. A higher weight is given priority to ensure the avoidance of large-scale data transmission and reduce the impact of the takeover process on the cluster bandwidth. Therefore, in implementation, the weight of the incremental data ratio is increased, and the weight of the incremental data ratio is set to 0.6, while the weight of the IO pressure index is set to 0.4.

[0119] In practice, the predetermined proportion is set at 30% to give special consideration to the sequence segments located at the beginning of the sorting sequence.

[0120] Specifically, the continued operation threshold needs to be adjusted based on the occupancy ratio of each hot data storage node, where,

[0121] The continuous running threshold is positively correlated with the occupancy ratio of hot data storage nodes;

[0122] The occupancy ratio of each hot data storage node is the proportion of the number of hot data storage nodes whose storage occupancy rate exceeds a preset occupancy threshold to the total number of hot data storage nodes.

[0123] Please see Figure 4 As shown, Figure 4 The following is a logic block diagram showing that the continued running threshold for this embodiment of the invention needs to be adjusted based on the occupancy ratio of each hot data storage node;

[0124] Storage occupancy rate is the proportion of storage capacity occupied by hot data storage nodes. The purpose of setting a preset occupancy threshold is to characterize the situation where there is a lot of storage capacity in the hot data node. The preset occupancy threshold is set to 60%.

[0125] Specifically, the occupancy status of all hot data storage nodes is monitored in real time to determine the occupancy ratio;

[0126] If the occupancy rate is lower than the first preset adjustment threshold, then the continued running threshold will be adjusted to 0.8 times the initial continued running threshold.

[0127] If the occupancy ratio is greater than or equal to the first preset adjustment threshold and less than the second preset adjustment threshold, then the initial continued running threshold is maintained.

[0128] If the occupancy rate is greater than or equal to the second preset adjustment threshold, then select to adjust the continued running threshold to 1.2 times the initial continued running threshold;

[0129] In this embodiment of the invention, the purpose of setting the first preset adjustment threshold is to characterize the situation where the occupancy ratio is relatively low. When the node occupancy ratio is lower than the critical value, the continued running threshold is reduced to expand the range of tasks that can be continued running, making full use of idle storage resources and avoiding idle computing power. Therefore, in implementation, the first preset adjustment threshold is 30%. The purpose of setting the second preset adjustment threshold is to characterize the situation where the occupancy ratio is relatively high. When the occupancy ratio is higher than the critical value, the continued running threshold is increased to tighten the continued running conditions and prevent the system stability from being reduced due to insufficient storage resources under high load. In implementation, the second preset adjustment threshold is set to 65%.

[0130] The present invention also provides a storage medium storing a computer program, which, when executed by a computer system, can be used to perform a distributed task fault takeover method.

[0131] The present invention also provides a computer system, comprising:

[0132] A computing unit consisting of one or more processors;

[0133] One or more memory units;

[0134] and one or more programs,

[0135] The one or more programs are configured to be executed by one or more processors, and the memory includes the storage medium.

[0136] If the method for distributed task fault takeover of the present invention is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for distributed task fault takeover, characterized in that, include: In response to the execution of a task by a task node, a persistence analysis is performed on the task node, including analyzing the cache to determine the proportion of dirty pages and determining the number of resource handles corresponding to the task. Based on the dirty page ratio and the number of resource handles, calculate the persistence characteristics of the task, and combine them with the persistence threshold to determine whether to set a persistence flag for the task. For the task's reproducibility flag, a hot data storage node is matched for the task node to perform caching operations on the task node, including determining a synchronization time window based on the persistence parameter and storing the takeover requirement data of the task node within the synchronization time window through the hot data storage node. In response to a task node anomaly, the system takes over based on the fact that the task being executed by the task node has a continue-run flag set. include, Determine the takeover load characteristics corresponding to several check time points within the synchronization time window, sort each check time point based on the takeover load characteristics, select the takeover time point, obtain the takeover requirement data corresponding to the takeover time point, and send it to the available nodes for task takeover. The takeover load characteristics include the incremental data ratio of the takeover demand data for each node and the IO pressure index, and the continued operation threshold is adjusted based on the occupancy ratio of each of the hot data storage nodes.

2. The method for distributed task fault takeover according to claim 1, characterized in that, It also includes taking over and restarting the task node in response to a task node anomaly, based on the fact that the task being executed by the task node has not been set with a continue-run flag.

3. The method for distributed task fault takeover according to claim 1, characterized in that, The process of calculating the persistence characteristic parameter based on the dirty page ratio and the number of resource handles includes: The ratio of the dirty page ratio in the processor cache to the dirty page threshold is determined as the dirty page parameter; The ratio of the number of resource handles to the handle threshold is determined as the handle parameter; The survival characterization parameter is obtained by weighted summation of the dirty page parameter and the handle parameter.

4. The method for distributed task fault takeover according to claim 3, characterized in that, Determining whether to set a continue-run flag for a task based on the continue-run threshold includes, If the persistence parameter of the task is less than the preset persistence threshold, then a persistence flag is set for the task.

5. The method for distributed task fault takeover according to claim 1, characterized in that, The synchronization time window is determined based on the persistence characteristic parameter, and the takeover requirement data of the task node within the synchronization time window is stored through the hot data storage node. The length of the determined synchronization time window is positively correlated with the survival characterization parameter; Within the synchronization time window, several equally spaced inspection time points are determined; The end time of the synchronization time window is the current time.

6. The method for distributed task fault takeover according to claim 1, characterized in that, Determine the takeover load characteristics corresponding to several inspection time points within the synchronization time window, including: Obtain takeover requirement data corresponding to several inspection time points within the synchronization time window; Obtain the change in takeover demand data corresponding to the previous inspection time point that is relatively adjacent to the inspection time point, and calculate the proportion of the change to the total amount of takeover demand data in the reference period as the incremental data ratio. Collect the frequency and latency of read and write operations on task nodes to generate an IO pressure index.

7. The method for distributed task fault takeover according to claim 1, characterized in that, Based on the aforementioned takeover load characteristics, the inspection time points are sorted to select the takeover time point, including: Input the characteristics of the takenover load into the preset load assessment model to generate load evaluation values; The sorted sequence is obtained by arranging the load evaluation values ​​in descending order; A predetermined proportion of sequence segments are extracted from the beginning of the sorted sequence. The sequence segments are then rearranged in ascending order based on the time length of the inspection time point corresponding to each load evaluation value in the sequence segment relative to the current time, to obtain the final sorted sequence. The inspection time point corresponding to the first end of the final sorted sequence is selected as the takeover time point.

8. The method for distributed task fault takeover according to claim 1, characterized in that, The continued operation threshold needs to be adjusted based on the occupancy ratio of each of the hot data storage nodes, wherein, The continued running threshold is positively correlated with the occupancy ratio of hot data storage nodes; The occupancy ratio of each hot data storage node is the proportion of the number of hot data storage nodes whose storage occupancy rate exceeds a preset occupancy threshold to the total number of hot data storage nodes.

9. A storage medium, characterized in that, The storage medium stores a computer program for performing the distributed task fault takeover method according to any one of claims 1-8.

10. A computer system, characterized in that, include: A computing unit consisting of one or more processors; One or more memory units; and one or more programs, The one or more programs are configured to be executed by one or more processors, and the memory includes the storage medium as described in claim 9.

Images