Stream processing system management method and related equipment and system

A stream processing system and management method technology, applied in the stream processing system management method and related equipment and system fields, can solve problems such as waste, resource occupation, and insufficient processing of large data streams

Active Publication Date: 2017-03-08
HUAWEI CLOUD COMPUTING TECH CO LTD
View PDF4 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

For example, during a certain period of time, the data flow input into the stream processing system is relatively large. At this time, the number of concurrent instances of the processing units determined by the initial topology design may not be sufficient to handle the current large data flow, which may affect the flow The throughput of the processing system; when the amount of data input into the stream processing system is small in a certain period of time
At this time, the number of concurrent instances of the processing unit determined by the user in the initial topology diagram design may be relatively large, which may cause resource waste due to the occupation of too many stream processing system resources

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Stream processing system management method and related equipment and system
  • Stream processing system management method and related equipment and system
  • Stream processing system management method and related equipment and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0070] Embodiments of the present invention provide a management method of a stream processing system and related equipment and systems, in order to improve the matching degree between the number of concurrent instances of a processing unit and the size of the actual data stream to be processed, and further ensure the throughput and stream processing of the stream processing system as much as possible. System resource utilization.

[0071] In the following, specific examples will be used to describe in detail respectively.

[0072] In order to make the purpose, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the following The described embodiments are only some, not all, embodiments of the present invention. Based on the embodime...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

A management method for a stream processing system, and a related device and system. In the management method for a stream processing system, the stream processing system comprises N calculation nodes, where N is a positive integer. The management method may comprise: monitoring a running condition of a processing unit deployed in a first calculation node of the N calculation nodes; and if it is monitored that the running condition of the processing unit conforms to a set concurrence example adjustment condition of the processing unit, adjusting a concurrence example of the processing unit, the running condition comprising a resource usage condition and / or processing performance. The solution in the embodiments of the present invention is beneficial for increasing the matching degree between the number of concurrence examples of a processing unit and the size of data streams actually required to be processed, thus guaranteeing the throughput of a stream processing system and the resource utilization rate of the stream processing system as far as possible.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a management method of a stream processing system and related equipment and systems. Background technique [0002] In the stream processing mode, the data is regarded as a stream, and when a new data stream arrives, it will be processed in real time and the required result will be returned. A stream processing system is a data processing system that uses the stream processing mode to process large amounts of data and generate low-latency responses. In order to be able to process large amounts of data in real time, some scenarios use a distributed data stream processing framework, and then through a scalable distributed architecture, the data processing unit can be easily expanded to improve the throughput of the stream processing system. [0003] International Business Machines Corporation (IBM, International Business Machines Corporation) has launched a stream processing syst...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/38
CPCG06F9/50
Inventor 王澜智伟张克星
Owner HUAWEI CLOUD COMPUTING TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap