Task scheduling method and system, computing equipment and storage medium

Abstract

The invention discloses a task scheduling method and system, computing equipment and a storage medium. The method comprises the steps that a plurality of task processing nodes send registration requests to a scheduling node; the scheduling node queries whether the task scheduling table contains a to-be-executed task, and if the to-be-executed task is queried, the plurality of task processing nodes are registered to generate node registration information; and the scheduling node obtains to-be-processed data corresponding to the to-be-executed task, splits the to-be-processed data according to the number of the to-be-processed data and the node registration information, and allocates the split to-be-processed data to the plurality of task processing nodes for processing. According to the invention, the to-be-executed task is dynamically scheduled to the plurality of task processing nodes through the scheduling node, and the data is concurrently processed through the plurality of task processing nodes, so that the situation that the plurality of task processing nodes compete to process the task or repeatedly process the task is avoided, the data processing efficiency is greatly improved, and the efficiency and robustness of large-batch data processing are improved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a task scheduling method, system, computing device and storage medium. Background technique [0002] During the construction of the cloud operation and maintenance platform, it is necessary to collect and monitor various cloud resources, processes, services and other component instances in real time. The size of data monitored has also grown exponentially. In the prior art, when large-scale data is collected, a scheduled task is used for data synchronization and comparison. There are generally two commonly used methods: the first is that the scheduled task is deployed on a single node, and the data is processed one by one; The second is that the scheduled task is deployed on multiple nodes, and a corresponding mark is marked after processing a piece of data. Before each data processing using the second method, it is necessary to determine whether the data has bee...

AI Technical Summary

Problems solved by technology

[0003] However, for the first method, when the amount of data to be processed is relatively large, the speed will be relatively slow and the efficiency will be low. Before data, it is necessary to verify whether the data has been processed, and there are situations where multiple nodes query a piece of unprocessed data at the same time, which may cause a piece of data to be processed by multiple nodes at the same time

Images