Task scheduling method of Ketle cluster server

Abstract

The invention discloses a task scheduling method of a Ketle cluster server, which comprises the following steps: performing coordination filtering based on the completion time of each task on each server to obtain a recommendation score of each server for each task; obtaining a load condition score of each server based on the state index of each server; based on the recommendation score of each server for each task and the load condition score of each server, obtaining a heuristic factor of each task transferred between any two servers; and performing iterative search in the cluster server based on the heuristic factor of each task transferred between any two servers in combination with an ant colony algorithm to obtain a final task scheduling allocation scheme. the data processing tasks on the Ketle system are scheduled and distributed by utilizing an artificial ant colony algorithm; the resource utilization rate of the Kettle server cluster system can be maximized, the load balance of the servers is achieved, the system performance is improved, the efficiency of the data cleaning, extracting and loading process is improved, then the customer experience is improved, the working efficiency is improved, and the benefits of enterprises are improved.

Description

technical field [0001] The invention relates to the technical field of cluster scheduling, in particular to a method for task scheduling of a Kettle cluster server. Background technique [0002] ETL (Extraction-Transformation-Loading) is a process in which one or more data sources are input, specific processing is performed, and then output to one or more data sources. The scheduling of the ETL cluster refers to the process in which the central scheduling node assigns ETL tasks to the service nodes in the cluster. Kettle, as the open-source ETL (Extraction-Transformation-Loading) tool with the largest number of users, includes two types of tasks: job and conversion, and the conversion includes specific operation components. Its powerful and concise functions are welcomed by the majority of ETL practitioners. On the Kettle platform built on cluster servers, it is often necessary to perform high-load big data extraction, conversion, cleaning, etc., and the number of service ...

AI Technical Summary

Problems solved by technology

[0004] 1. The operator does not always know which server has more free resources. Whenever a new task comes, most of them can only be allocated based on feeling and experience, which can easily lead to unbalanced server cluster load

Idle ones are not being used, while busy ones are busier and the system runs slowly

[0005] 2. Manual designation increases the workload of the staff. While writing the logic program for data conversion, it is also necessary to consider the assignment of tasks, which reduces the efficiency of the Kettle system and the team using the system to complete the work.

Images