Task processing method and device, computer equipment and storage medium

Abstract

The invention relates to a task processing method and device, computer equipment and a storage medium. The method comprises the steps of obtaining at least one to-be-executed task and corresponding attribute information; inputting each to-be-executed task and corresponding attribute information to the trained resource prediction model, and outputting at least one piece of candidate resource information, the candidate resource information comprising a candidate resource type and a corresponding candidate occupation duration; acquiring current resource information of a plurality of data centers;and determining target resource information of each to-be-executed task according to the candidate resource information of each to-be-executed task and each current resource information. The resources of the to-be-executed task are predicted through the resource prediction model; the corresponding prediction information is obtained, the prediction information and the current resource informationare matched to obtain the target resource information, and the to-be-executed task is executed in the data center corresponding to the target resource information, so that the reasonability of resource allocation is ensured, and the data processing efficiency and the resource utilization rate are improved.

Description

technical field [0001] The present application relates to the field of computer technology, and in particular to a task processing method, device, computer equipment and storage medium. Background technique [0002] More complex deep learning models need to be trained using GPU or CPU clusters of different models, scales, and network configurations. Usually, the total amount of different types of GPU or CPU resources that can be scheduled in a single data center is limited. The center's real-time reading of large batches of data will greatly affect the efficiency of model training. [0003] The current method mainly provides a unified entrance through the global task scheduling center, and screens out the method of running the data center that meets the task requirements; that is, all tasks are scheduled using the same scheduling module, and the scheduling is based on the resources corresponding to each task. Designated, while artificially designated resources are related t...

AI Technical Summary

Problems solved by technology

[0002] More complex deep learning models need to be trained using GPU or CPU clusters of different models, scales, and network configurations. Usually, the total amount of different types of GPU or CPU resources that can be scheduled in a single data center is limited. The center reads large batches of data in real time, which will greatly affect the efficiency of model training

[0003] The current method mainly provides a unified entrance through the global task scheduling center, and screens out the method of running the data center that meets the task requirements; that is, all tasks are scheduled using the same scheduling module, and the scheduling is based on the resources corresponding to each task. Designated, and artificially designated resources are related to the experience of technicians, it is difficult to ensure the rationality of designated resource allocation

Images