Data processing method and data processing system based on STORM streaming computation

A streaming computing and data technology, applied in the direction of electrical digital data processing, special data processing applications, redundant data error detection in computing, etc., can solve data loss and other problems

Active Publication Date: 2019-12-13
南京星云数字技术有限公司
View PDF4 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Based on this, it is necessary to address the above technical problems and provide a data processing method and data processing system based on STORM str

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and data processing system based on STORM streaming computation
  • Data processing method and data processing system based on STORM streaming computation
  • Data processing method and data processing system based on STORM streaming computation

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0039] In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not intended to limit the present application.

[0040] A data processing method based on STORM streaming computing provided by the present application is applied to a data processing system based on STORM streaming computing. Wherein, the data processing system of the STORM streaming computing may be a Storm computing system. In one embodiment, as figure 1 As shown, the data processing system 10 for STORM streaming computing includes a STORM input node 100 and a plurality of STORM working nodes. There may be one or more STORM input nodes 100 . The plurality of STORM working nodes include STORM working node 210 , STOR...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a data processing method and a data processing system based on STORM streaming computing. The method comprises the steps that an STORM input node sends a state data backup instruction to an STORM target working node; when the STORM target working node receives the state data backup instruction, the node state data of the STORM target working node is backed up and stored; when STORM streaming computation is interrupted, the STORM input node recovers the message offset state data backed up for the last time in the STORM input node, and sends a state data recovery instruction to the STORM target working node; when the STORM target working node receives a state data recovery instruction, recovering node state data of the STORM target working node from the backup storage; wherein the message offset state data and the node state data are used for STORM streaming computation. According to the method, data stream recovery can be carried out when the data processing system executes stream-oriented computation, and the problem of data loss in the STORM stream-oriented computation process in the data processing system is solved.

Description

technical field [0001] The present application relates to the technical field of streaming computing, in particular to a data processing method and data processing system based on STORM streaming computing. Background technique [0002] With the development of information technology and the integration of retail and the Internet, big data has become a standard configuration for the development of enterprises in the Internet age. For the e-commerce field, the value of big data lies in providing data support for refined operations and marketing. For the big data department, its main areas are business intelligence and data warehouse, and stream computing is one of the indispensable technical means for data processing in the big data department. Currently, the Storm computing system is mainly used to implement streaming computing. The Storm computing system includes multiple physical machines, multiple virtual machines, and multiple topology tasks. The Storm computing system ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F11/14G06F16/2455
CPCG06F11/1469G06F11/1458G06F16/24568
Inventor 何肖明
Owner 南京星云数字技术有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap