Flow processing operation scheduling method and system for dynamically adjusting task allocation

A technology of dynamic adjustment and task allocation, applied in transmission systems, electrical components, etc., can solve the problem of waste of cluster resources, achieve the effect of fast and effective adjustment, reduce communication overhead, and minimize impact

Active Publication Date: 2018-01-12
SHANDONG UNIV
View PDF4 Cites 30 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Therefore, in order to cope with the peak processing volume of stream processing applications, it is necessary to pre-cal

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Flow processing operation scheduling method and system for dynamically adjusting task allocation
  • Flow processing operation scheduling method and system for dynamically adjusting task allocation
  • Flow processing operation scheduling method and system for dynamically adjusting task allocation

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0070] It should be pointed out that the following detailed description is exemplary and intended to provide further explanation to the present application. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0071] It should be noted that the terminology used here is only for describing specific implementations, and is not intended to limit the exemplary implementations according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural, and it should also be understood that when the terms "comprising" and / or "comprising" are used in this specification, they mean There are features, steps, operations, devices, logic units and / or combinations thereof.

[0072] The stream processing job scheduling method for dynamically adjusting task allocation of the present i...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a flow processing operation scheduling method and system for dynamically adjusting task allocation. The method comprises the steps of receiving the flow processing operation bya main control node server, and generating an initial task allocation scheme according to a given reference concurrency degree, available resources in a cluster and the self structure of the flow processing operation; during the service operation process of the flow processing operation, automatically reporting the own running state information of each calculation node server to a main control node server by the calculation node server itself, uniformly summarizing the running state information of all calculation node servers by the main control node server, and conducting the corresponding aggregation operation; predicting the data average processing amount and the corresponding resource occupation value of the next time period according to the information after the aggregation operationby the main control node server; further dynamically adjusting the flow processing task instances of all the calculation node servers, finally generating a dynamic allocation scheme of the flow processing operation and distributing the dynamic allocation scheme to corresponding calculation node servers.

Description

technical field [0001] The invention belongs to the field of distributed stream processing, and in particular relates to a stream processing job scheduling method and system for dynamically adjusting task allocation. Background technique [0002] In recent years, with the continuous development of informatization and the Internet, a large amount of data has been generated quickly and in real time. In order to obtain and analyze the information in the data quickly, efficiently and instantly, the stream processing system, as a real-time data processing system, has been widely used. A stream processing job is a long-running job task, and the amount of data processed during the running process usually changes greatly over time. [0003] In distributed stream processing systems, stream processing jobs are usually abstracted as a directed acyclic graph. The points in the graph represent the logical unit of the job, and the edges in the graph represent the data flow and grouping ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): H04L29/08
Inventor 陈岳亭禹晓辉
Owner SHANDONG UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap