Method for managing stream processing system, and related apparatus and system

A stream processing system and management method technology, applied in the stream processing system management method and related equipment and system fields, can solve the problems of insufficient processing of large data streams, resource occupation, waste, etc.

Active Publication Date: 2014-05-07
HUAWEI CLOUD COMPUTING TECH CO LTD
View PDF4 Cites 16 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

For example, during a certain period of time, the data flow input into the stream processing system is relatively large. At this time, the number of concurrent instances of the processing units determined by the initial topology design may not be sufficient to handle the current large data flow, which may affect the flow The throughput of the processing system; when the amount of data input into the stream processing system is small in a certain period of time
At this time, the number of concurrent instances of the processing unit determined by the user in the initial topology diagram design may be relatively large, which may cause resource waste due to the occupation of too many stream processing system resources

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for managing stream processing system, and related apparatus and system
  • Method for managing stream processing system, and related apparatus and system
  • Method for managing stream processing system, and related apparatus and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0064] Embodiments of the present invention provide a management method of a stream processing system and related equipment and systems, in order to improve the matching degree between the number of concurrent instances of a processing unit and the size of the actual data stream to be processed, and further ensure the throughput and stream processing of the stream processing system as much as possible. System resource utilization.

[0065] In the following, specific examples will be used to describe in detail respectively.

[0066] In order to make the purpose, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the following The described embodiments are only some, not all, embodiments of the present invention. Based on the embodime...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present invention provides a method for managing a stream processing system, and a related apparatus and system. The stream processing system comprises N computing nodes, N being a positive integer. The managing method comprises: monitoring the operation state of a processing unit deployed in the first computing node of the N computing notes; and adjusting concurrent instances of the processing unit if it is detected that the operation state of the processing unit meets preset adjusting conditions of the concurrent instances of the processing unit, the operation state including use of resources and / or processing performance. According to the embodiments of the present invention, the matching degree between the number of concurrent instances of the processing unit and the size of an actual data stream to be processed is improved, thereby ensuring the throughput of the stream processing system and the utilization of the resources of the stream processing system as far as possible.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a management method of a stream processing system and related equipment and systems. Background technique [0002] In the stream processing mode, the data is regarded as a stream, and when a new data stream arrives, it will be processed in real time and the required result will be returned. A stream processing system is a data processing system that uses the stream processing mode to process large amounts of data and generate low-latency responses. In order to be able to process large amounts of data in real time, some scenarios use a distributed data stream processing framework, and then through a scalable distributed architecture, the data processing unit can be easily expanded to improve the throughput of the stream processing system. [0003] International Business Machines Corporation (IBM, International Business Machines Corporation) has launched a stream processing syst...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/38
CPCG06F9/50
Inventor 王澜智伟张克星
Owner HUAWEI CLOUD COMPUTING TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap