Large-scale data processing device and method based on procedure processing frame

Abstract

The invention discloses a large-scale data processing device and method based on a procedure processing frame. The device comprises a topology construction module, a data reading module, a data processing module, an aggregation module and a storage module; the topology construction module is used for constructing data processing topology according to an XML configuration file; the data reading module is used for reading original data from a data source with a label, and loading a logic configuration file according to the label to obtain data of additional logic configuration; the data processing module is used for receiving the data with the additional logic configuration, and dynamically calling a processing method to generate a processing result and conducting shunting; the aggregation module is used for receiving a processing result of shutting and conducting merging; the storage module is used for receiving a merged result, and storing the merging result in a specified storage medium. In the large-scale data processing device and method based on the procedure processing frame, the data processing speed is high, and newly added data can be timely processed; a newly added processing rule can be dynamically called by being configured in Redis, and the insertion mode of the data is diversified; in different scenes, the device and method can be simply utilized with simple configuration modification, and have a certain application prospect.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a large-scale data processing device and method based on a stream processing framework. Background technique [0002] At present, large-scale data is generally processed using a multi-threaded single instance method. This method usually runs on one server and is highly targeted to the business, but the corresponding configuration is less. With the explosive growth of data, traditional data processing methods can no longer meet the speed and performance requirements of large-scale data processing. The main defects are as follows: [0003] 1. Because a single server has performance bottlenecks such as poor network stability and high CPU usage, the data processing speed is not fast enough, and new data cannot be processed in a timely manner. [0004] 2. When processing data, there is no configuration file to intervene in the processing process, and the processing rules cannot be ...

AI Technical Summary

Problems solved by technology

[0003] 1. Because a single server has performance bottlenecks such as poor network stability and high CPU usage, the data processing speed is not fast enough, and new data cannot be processed in time

[0004] 2. When processing data, there is no configuration file to intervene in the processing process, and the processing rules cannot be dynamically configured. Once the processing rules change, the program needs to be restarted

[0005] 3. The statement (such as SQL) inserted in one operation is fixed and cannot be modified dynamically. The data insertion method is single and does not support multiple insertion methods

[0006] 4. One time data processing can only be used in one business scenario, highly coupled with specific business, poor independence, and inconvenient to migrate

Images