Fast data processing method and device based on flume system
A data processing device and data processing technology, applied in the information field, can solve the problems of production machine burden and occupation of production machine resources, etc.
Pending Publication Date: 2022-02-25
武汉众邦银行股份有限公司
View PDF0 Cites 1 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0012] The purpose of the present invention is to solve the problem that the Flume system is attached to the production machine, and the cleaning work of the interceptor will occupy the resources of the production machine and cause a burden on the production machine
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0046] In order to overcome the deficiencies of the prior art, the purpose of the present invention is: 1. To solve the problem that flume collects and cleans data and affects the operation of production machines. 2. Solve the problem that flume cannot satisfy complex scene data processing. 3. Solve the efficiency problem of flume data collection and cleaning. 4. Solve the problem that flume cannot cache large batches of data. 5. Solve the problem that the data collected by flume cannot support repeated consumption by multiple consumer groups. like Figure II The concrete implementation shown is:
[0047] The source layer of flume collects raw data, and sends the collected raw data to the distributed message middleware module in real time.
[0048] The distributed data processing module consumes the data collection message queue data of the distributed message middleware module in real time for data cleaning processing, and then sends the processed data to the result messa...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention relates to the technical field of information, provides a fast data processing method and device based on a flume system, and aims to solve the problems that the Flume system is attached to a production machine, cleaning work of an interceptor occupies resources of the production machine, and burden is caused to the production machine. According to the main scheme, the method comprises the steps that a source layer of flume collects original data and sends the collected original data to a data collection queue of a distributed message middleware module in real time; a distributed data processing module consumes the data in the acquisition queue of the distributed message middleware module in real time to perform data cleaning processing; the distributed data processing module sends the processed data to a result message queue of the distributed message middleware module in real time; and the Sink layer of the flume obtains the processed data from the result message queue of the distributed message middleware module and sends the processed data to a receiver.
Description
technical field [0001] The invention relates to the field of information technology, and provides a fast data processing method and device based on a flume system. Background technique [0002] In order to better understand this application, it is necessary to understand the following basic technologies: [0003] Data cleaning: Data cleaning is the process of re-examining and verifying data, with the purpose of removing duplicate information, correcting existing errors, and providing data consistency. [0004] In the data collection phase, we generally use Flume as the data collection tool. Flume is a highly available, highly reliable and distributed mass data collection, aggregation and transmission system provided by Cloudera. Flume supports customizing various data senders in the system to collect data. At the same time, Flume can perform simple data processing on data. And write about the capabilities of various data recipients. [0005] like figure 1 As shown, a con...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/215G06F16/2457G06F16/2458G06F9/54
CPCG06F16/215G06F16/2471G06F16/24578G06F9/546G06F2209/548G06F2209/547
Inventor 徐浩李耀田骏石龙宋圣杰
Owner 武汉众邦银行股份有限公司