Method and system for achieving data conversion based on storm and ram grid

A data and memory technology, applied in the field of data conversion based on storm and memory grid, can solve problems such as high delay of storm processing nodes, lower data processing efficiency, cumbersome matching data changes, etc., to improve data conversion efficiency and improve data The effect of processing efficiency and high fault tolerance

Active Publication Date: 2017-01-04
中电福富信息科技有限公司
View PDF4 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

When storm cooperates with other traditional matching data sources (such as redis) to perform data conversion operations, due to the fast processing speed of storm, when performing data matching, traditional data sources in data conversion operations cannot meet the high-speed processing capabilities of storm with low lat

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for achieving data conversion based on storm and ram grid
  • Method and system for achieving data conversion based on storm and ram grid
  • Method and system for achieving data conversion based on storm and ram grid

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0034] Such as figure 1 with image 3 As shown, a data conversion method based on storm and memory grid needs to provide storm's multiple launch nodes, multiple processing nodes, multiple memory grid nodes and a warm-up and update node, and each processing node Corresponding to a transmitting node and a memory grid node, the method includes the following steps:

[0035] Step 10, collecting data, and entering the collected data into the message queue in the form of data flow one by one in order;

[0036] Step 20. Bind each processing node to a memory grid node, and the bound processing node and the memory grid node share the memory space, and realize synchronous startup and synchronous shutdown;

[0037] Step 30. Before performing the conversion operation, after the warm-up and update nodes receive the data loading request, they call all processing nodes to obtain the first data to be matched from the associated data source, and store the first data in their respective bindin...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a method for achieving data conversion based on storm and RAM grids. Multiple emission nodes, processing nodes and ram grid nodes and one warm-up and update node of the storm need to be provided, and each processing node corresponds to one emission node and one ram grid node. The method includes the steps that data is acquired in a message queue; each processing node and one corresponding ram grid node are bound, and ram space sharing and synchronous starting and stopping are achieved; after the warm-up and update node receives a data loading request, all the processing nodes are called to acquire to-be-matched first data from an association data source and store the first data in the ram data nodes; the emission nodes read to-be-matched second data in the message queue to the processing nodes; the processing nodes acquire the first data from the ram grid nodes for matching, and the matched data is sent to corresponding output nodes. The invention further provides a system for achieving data conversion based on the storm and the ram grid. Data processing efficiency is improved.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a method and system for realizing data conversion based on a strom and a memory grid. Background technique [0002] As more and more data needs to be processed per unit time, real-time processing has become the primary challenge faced by many organizations. Storm, as a low-latency, high-fault-tolerant stream computing framework, is widely used in more and more projects And in the software system, strom reads data from the message queue, and sends them one by one to the processing node through the transmitting node for data conversion operations. The data conversion operation is mainly to perform corresponding operations according to the preset business, generally operations such as transformation or discarding, etc. The data after the data conversion operation is sent to the output node. When storm cooperates with other traditional matching data sources (such as redis) t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/54
Inventor 连城郭海涛孙而焓
Owner 中电福富信息科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap