Data processing method and stream computing system

A data processing and data technology, applied in the field of data processing, can solve problems such as avalanche effect, difficulty in satisfying external service, tight coupling of message middleware, etc., and achieve the effect of avoiding retransmission

Active Publication Date: 2019-07-09
ALIBABA GRP HLDG LTD
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The most commonly used flow computing system is Storm, which is often combined with message middleware (such as Kafka) or storage system (such as HBase) in practical applications to obtain data through the pull data mode. The disadvantage is that it is tightly coupled with message middleware. It is difficult to meet the demands of external service
In addition, Storm adopts the "source retransmission" message mechanism during failover. The disadvantage of this method is that the cost of failure recovery is high, and it may cause an avalanche effect in some scenarios. The cluster scale is in terms of horizontal scalability. have more limitations

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and stream computing system
  • Data processing method and stream computing system
  • Data processing method and stream computing system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0019] Embodiments of the present application are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals denote the same or similar modules or modules having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary, and are only for explaining the present application, and should not be construed as limiting the present application. On the contrary, the embodiments of the present application include all changes, modifications and equivalents falling within the spirit and scope of the appended claims.

[0020] figure 1 It is a schematic flowchart of a data processing method proposed in an embodiment of the present application. The method can be applied to a flow computing system for processing flow data. The method includes:

[0021] S11: After receiving the data to be processed, the data receiving module writes the data into the file system, and sends the ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

System and method are disclosed for stream computing. An exemplary method may include receive data from a data processing module and determining whether the received data are effective data that are neither incomplete nor duplicative. The method may also include obtaining the effective data when it is determined that the received data are either incomplete or duplicative. In addition, the method may include storing the effective data in a log file of a file system.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular to a data processing method and a stream computing system. Background technique [0002] Stream computing refers to the use of distributed ideas and methods to process massive "streaming" data in real time, which originates from the demand for mining the "timeliness" value of massive data. The data targeted by stream computing can be called stream data. Stream data is boundless and unknown, while computation is defined (known) in advance. The stream computing system processes stream data according to the defined calculation logic. [0003] The most commonly used flow computing system is Storm, which is often combined with message middleware (such as Kafka) or storage system (such as HBase) in practical applications to obtain data through the pull data mode. The disadvantage is that it is tightly coupled with message middleware. It is difficult to meet the dema...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/182G06F16/17G06F21/53
CPCG06F21/53G06F16/1734G06F16/182G06F16/1748G06F16/24568
Inventor 李妹芳魏蒲萌段培乐李闪
Owner ALIBABA GRP HLDG LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap