A method and system for realizing data conversion based on strom and memory grid

Abstract

The invention provides a method for achieving data conversion based on storm and RAM grids. Multiple emission nodes, processing nodes and ram grid nodes and one warm-up and update node of the storm need to be provided, and each processing node corresponds to one emission node and one ram grid node. The method includes the steps that data is acquired in a message queue; each processing node and one corresponding ram grid node are bound, and ram space sharing and synchronous starting and stopping are achieved; after the warm-up and update node receives a data loading request, all the processing nodes are called to acquire to-be-matched first data from an association data source and store the first data in the ram data nodes; the emission nodes read to-be-matched second data in the message queue to the processing nodes; the processing nodes acquire the first data from the ram grid nodes for matching, and the matched data is sent to corresponding output nodes. The invention further provides a system for achieving data conversion based on the storm and the ram grid. Data processing efficiency is improved.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a method and system for realizing data conversion based on a strom and a memory grid. Background technique [0002] As more and more data needs to be processed per unit time, real-time processing has become the primary challenge faced by many organizations. Storm, as a low-latency, high-fault-tolerant stream computing framework, is widely used in more and more projects And in the software system, strom reads data from the message queue, and sends them one by one to the processing node through the transmitting node for data conversion operations. The data conversion operation is mainly to perform corresponding operations according to the preset business, generally operations such as transformation or discarding, etc. The data after the data conversion operation is sent to the output node. When storm cooperates with other traditional matching data sources (such as redis) t...

AI Technical Summary

Problems solved by technology

When storm cooperates with other traditional matching data sources (such as redis) to perform data conversion operations, due to the fast processing speed of storm, when performing data matching, traditional data sources in data conversion operations cannot meet the high-speed processing capabilities of storm with low latency, which will cause storm The processing node delay is too high, which leads to the problem of data backlog, and when storm cooperates with other traditional matching data sources (such as redis) for data conversion, the change of matching data is very cumbersome, which greatly reduces the efficiency of data processing

Images