Stream data processing method, device and system

A processing method and streaming data technology, applied in the computer field, which can solve problems such as poor streaming computing throughput.

Active Publication Date: 2015-02-04
HUAWEI TECH CO LTD
View PDF3 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] In order to solve the problem of poor throughput of stream computing in the prior art, a stream data processing method provided by the embodiment of the present invention encapsulates the received business data into triplet stream data, and

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Stream data processing method, device and system
  • Stream data processing method, device and system
  • Stream data processing method, device and system

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0078] The embodiment of the present invention provides a stream data processing method, which encapsulates the received service data to form triple stream data, and constructs a small batch data set according to the triple stream data and the initial time-sharing lower limit, so that the While ensuring the reliability of the streaming computing system, it also improves the computing throughput of the streaming computing system. The embodiments of the present invention also provide corresponding devices and systems. Detailed descriptions are given below.

[0079] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be described clearly and completely with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is a part of the embodiments of the present invention, not all the embo...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a stream data processing method. The method is applied to a work node in a steam calculation system, and comprises the following steps of: obtaining the initial time-sharing lower limit value of a source operator; receiving the service data, and packaging the service data stream into triple stream data, wherein the triple stream data carries a time stamp and a data mark uniquely marking the service data; building a small-batch data set mini-batch according to the triple stream data and the initial time-sharing lower limit value of the source operator, wherein the time stamp is used for indicating the data order of the data in the building process, and the unique mark is used for indicating the data uniqueness in the building process; processing the small-batch data set mini-batch, and outputting processed data. The method provided by the embodiment of the invention has the advantages that the reliability of the stream calculation system is realized on the basis of the order and uniqueness of the triple stream data, and the calculation throughput of the stream calculation system can be improved on the basis of the initial time-sharing lower limit value, so the reliability of the stream calculation system is ensured, and meanwhile, the stream calculation throughput capability is also improved.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a method, device and system for stream data processing. Background technique [0002] In recent years, data-intensive services have been more and more widely used. Examples of these services include financial services, network monitoring, telecommunication data management, sensor detection and so on. The data generated by data-intensive services has the characteristics of large data volume, fast speed, and time-varying. The stream computing system starts to process the stream data when it receives it to ensure real-time performance. [0003] Such as figure 1 As shown, a stream computing system usually includes a master control node (Master) and multiple work nodes (worker). The master control node is mainly responsible for scheduling and managing each work node, and the work node is a logical entity that carries actual data processing operations. Specifically, the wor...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/38G06F9/46
Inventor 张松山
Owner HUAWEI TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap