Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A task scheduling method for kettle cluster servers

A cluster server and task scheduling technology, applied in the task scheduling field of Kettle cluster servers, can solve problems such as lowering the Kettle system and using system efficiency, unbalanced server cluster load, and increasing the workload of staff, so as to improve customer experience, The effect of improving efficiency and improving efficiency

Active Publication Date: 2022-06-28
CHINA POWER IND INTERNET CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] 1. The operator does not always know which server has more free resources. Whenever a new task comes, most of them can only be allocated based on feeling and experience, which can easily lead to unbalanced server cluster load
Idle ones are not being used, while busy ones are busier and the system runs slowly
[0005] 2. Manual designation increases the workload of the staff. While writing the logic program for data conversion, it is also necessary to consider the assignment of tasks, which reduces the efficiency of the Kettle system and the team using the system to complete the work.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A task scheduling method for kettle cluster servers
  • A task scheduling method for kettle cluster servers
  • A task scheduling method for kettle cluster servers

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0056] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0057] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relationship between various components under a certain posture (as shown in the accompanying drawings). The relative positional relationship, the movement situation, etc., if the specific posture changes, the directional indication also changes accordingly. ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a task scheduling method for a Kettle cluster server, which includes coordinating and filtering based on the completion time of each task on each server to obtain the recommendation score of each server for each task; obtaining the score of each server based on the status index of each server Load status score; based on the recommendation score of each server for each task and the load status score of each server, the heuristic factor for each task to be transferred between any two servers; based on the transfer rate of each task between any two servers The heuristic factor combined with the ant colony algorithm performs iterative search in the cluster server, and obtains the final task scheduling and allocation scheme. Using the artificial ant colony algorithm to schedule and allocate data processing tasks on the Kettle system can maximize the resource utilization of the Kettle server cluster system, achieve server load balancing, improve system performance, and improve the efficiency of data cleaning, extraction, and loading processes, thereby improving customer experience. , Improve work efficiency to enhance the benefits of the enterprise.

Description

technical field [0001] The invention relates to the technical field of cluster scheduling, in particular to a task scheduling method for a Kettle cluster server. Background technique [0002] ETL (Extration Extraction-Transformation Transformation-Loading Loading) is the process of inputting one or more data sources, performing specific processing, and then outputting to one or more data sources. The scheduling of the ETL cluster refers to the process that the central scheduling node allocates ETL tasks to the service nodes in the cluster. Kettle, as the most user-scale open source ETL (Extration extraction-Transformation transformation-Loading loading) tool, includes two types of tasks: job and transformation. The transformation includes specific operation components. Its powerful and concise functions are very popular among ETL practitioners. On the Kettle platform built on cluster servers, it is often necessary to perform high-load big data extraction, transformation, a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/48G06F9/50G06N3/00
CPCG06F9/4881G06F9/5083G06N3/006
Inventor 王子新傅苗胡鹏匡岳锋曹林贺群雄
Owner CHINA POWER IND INTERNET CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com