A task processing method and device

A task processing and task management technology, applied in the field of data processing, can solve problems such as insufficient computing resources for tasks, poor user experience, and impact on cluster task throughput, so as to improve execution efficiency, ensure resource utilization, and improve task concurrency degree of effect

Active Publication Date: 2020-12-22
TAOBAO CHINA SOFTWARE
View PDF2 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In practical applications, many tasks do not take all the fields of the entire table, and many tasks have many special operations, such as LIMIT (limitation), GBY (aggregation), etc. If the default algorithm is still used, it will lead to many tasks Reasonable computing resources cannot be allocated, which will affect the throughput of cluster tasks, causing some tasks to wait for a long time because they cannot obtain computing resources in time, which will have a very bad impact on user experience

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A task processing method and device
  • A task processing method and device
  • A task processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0026] As mentioned in the background technology, in the existing default processing rules, the system considers that the size of the input file is the size of the entire table, and a map (mapping) can handle 256M data. This approach does not take into account the computing cluster quota ( Quota) usage rate does not take into account the amount of data actually read by the task, and does not make full use of the characteristics of the task to adaptively allocate computing resources. However, the technical solution proposed in the embodiment of the present application solves this problem.

[0027] The following is a clear and complete description of the technical solutions of the present application with reference to the accompanying drawings. Apparently, the described embodiments are part of the embodiments of the present application, but not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention discloses a method and device for task management. By application of the technical scheme, by combination with the actual input data size of tasks and current quota use rate of a group of a cluster where the tasks are located, and adaptability adjustment is performed on corresponding resources. Thus, in a large computing cluster, computing resources are distributed individually distributed for running tasks, it is guaranteed that the tasks obtain reasonable resources, task execution efficiency is improved, the resource utilization rate of the computing cluster is ensured, and task concurrency is improved.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular to a task processing method and device. Background technique [0002] For a busy large-scale cluster system, there will be hundreds of thousands of temporary query tasks every day. Among these temporary query tasks, most data retrieval tasks will basically only run once. [0003] In the existing technical solution, by default, the system considers that the size of the input file is the size of the entire table, and a table can handle 256M data, so that the number of tables required for the task can be controlled. [0004] However, the applicant found the following technical problems in the prior art during the process of implementing the application: [0005] In practical applications, many tasks do not take all the fields of the entire table, and many tasks have many special operations, such as LIMIT (limitation), GBY (aggregation), etc. If the default algori...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/50G06F16/2455
CPCG06F9/5005G06F16/2455
Inventor 贾元乔孙伟光苏艳
Owner TAOBAO CHINA SOFTWARE
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap