Large-scale data processing device and method based on procedure processing frame

A large-scale data, stream processing technology, applied in the computer field, can solve the problems of non-dynamic modification, poor independence, high coupling, and achieve the effect of various insertion methods and fast data processing speed

Inactive Publication Date: 2017-10-03
湖南星汉数智科技有限公司
View PDF0 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] 1. Because a single server has performance bottlenecks such as poor network stability and high CPU usage, the data processing speed is not fast enough, and new data cannot be processed in time
[0004] 2. When processing data, there is no configuration file to intervene in the processing process, and the processing rules cannot be dynamically configured. Once the processing rules chan

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Large-scale data processing device and method based on procedure processing frame
  • Large-scale data processing device and method based on procedure processing frame

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0034] Environment preparation: This embodiment is a large-scale data general-purpose processing device based on a streaming processing framework, which relies on the streaming processing framework, Kafka and Redis message middleware, and media for data storage. This time, Storm is used as the underlying processing framework. Make sure these environments are in place before deploying the appliance.

[0035] Configuration preparation: when constructing the topology, load the XML configuration file and Storm-related configuration in Redis, determine the loaded modules and the operating parameters of Storm (such as the number of jobs and tasks of Storm); when building the topology, data sources and storage media When connecting, load the Properties configuration file; during data processing, dynamically load the configuration file in Redis, and modify the running threshold in real time (such as: batch submission data size, waiting timeout time, data processing rules, etc.).

[00...

Embodiment 2

[0051] refer to figure 2 , this embodiment is based on the large-scale data processing method of the stream processing framework, including the following process:

[0052] Step 1: The topology construction module constructs the data processing topology on the stream processing framework according to the XML configuration file, and establishes the connection between the data processing topology and the data source and storage medium at the same time;

[0053] Step 2: The data reading module reads the tagged raw data from the data source, loads the corresponding logical configuration file according to the tag, obtains the data of the additional logical configuration and sends it to the data processing module; the logical configuration file contains the processing logic , processing method and storage logic content;

[0054] Step 3: The data processing module receives the data of the additional logical configuration, and dynamically invokes the corresponding processing method a...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a large-scale data processing device and method based on a procedure processing frame. The device comprises a topology construction module, a data reading module, a data processing module, an aggregation module and a storage module; the topology construction module is used for constructing data processing topology according to an XML configuration file; the data reading module is used for reading original data from a data source with a label, and loading a logic configuration file according to the label to obtain data of additional logic configuration; the data processing module is used for receiving the data with the additional logic configuration, and dynamically calling a processing method to generate a processing result and conducting shunting; the aggregation module is used for receiving a processing result of shutting and conducting merging; the storage module is used for receiving a merged result, and storing the merging result in a specified storage medium. In the large-scale data processing device and method based on the procedure processing frame, the data processing speed is high, and newly added data can be timely processed; a newly added processing rule can be dynamically called by being configured in Redis, and the insertion mode of the data is diversified; in different scenes, the device and method can be simply utilized with simple configuration modification, and have a certain application prospect.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a large-scale data processing device and method based on a stream processing framework. Background technique [0002] At present, large-scale data is generally processed using a multi-threaded single instance method. This method usually runs on one server and is highly targeted to the business, but the corresponding configuration is less. With the explosive growth of data, traditional data processing methods can no longer meet the speed and performance requirements of large-scale data processing. The main defects are as follows: [0003] 1. Because a single server has performance bottlenecks such as poor network stability and high CPU usage, the data processing speed is not fast enough, and new data cannot be processed in a timely manner. [0004] 2. When processing data, there is no configuration file to intervene in the processing process, and the processing rules cannot be ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/22G06F16/284
Inventor 曾琰陈敏张圣栋
Owner 湖南星汉数智科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap