A Distributed Task Assignment Method

Abstract

The invention relates to a distributive task allocation method, which comprises the following steps that: a job is subjected to initialization decomposition to obtain a plurality of tasks, and a main control node builds a task operation state table for the tasks; the first task is allocated to a calculation node, and meanwhile, a backup task identical to the first task is allocated to a prediction node; the prediction node calculates the disc space required by the backup task, and feeds back the calculation result to the main control node; the main control node updates the task operation state table according to the calculation result from the prediction node; before the task is allocated to the calculation node, the main control node predicts the task state of the calculation node and estimates available task operation space according to the predicted task state and the task operation state table; and under the condition that the estimated available task operation space is greater than the space required for the task operation, the task is allocated to the calculation node, and otherwise, the calculation node is re-selected.

Description

technical field [0001] The invention relates to a task allocation method in a distributed computing system, in particular to an optimization method capable of preventing distributed task computing time delays. Background technique [0002] In the prior art, in a distributed computing process, a job (job) is usually decomposed into multiple subtasks, which are assigned by the main control node to multiple computing nodes for parallel computing. When each computing subtask (task) runs on the computing node, it needs to write the intermediate data to the local file system. Usually, when multiple subtasks are running on the same computing node, they can write data to the file system through different disk write points, so as to improve the throughput of the disk when the job is running. [0003] If the disk space where a write point is located is insufficient, the task will be denied access because the disk space is full when the file is written halfway, causing the task to fai...

AI Technical Summary

Problems solved by technology

Although this can solve the above problems to a certain extent, the configuration value is fixed and cannot be dynamically adjusted according to the needs of different tasks, which may cause a waste of computing resources

If the configuration value is too low, the task will still fail due to insufficient disk space; if the configuration value is too high, the compute nodes will not be fully utilized

Images