Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Flume-based online multi-party aggregation method

A multi-party, scheduler technology, applied in the field of data analysis, can solve problems such as increased operation and maintenance costs, unpredictable manual configuration, difficult authority control, etc., and achieves strong scalability

Pending Publication Date: 2021-03-19
TIANYI ELECTRONICS COMMERCE
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide an online multi-party aggregation method based on Flume. Forwarding, etc., can not only solve the unpredictable risks of manual configuration, but also the difficulty of controlling the authority of automatic remote configuration of the program, and the increase of operation and maintenance costs.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Flume-based online multi-party aggregation method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0020] Such as figure 1 As shown, the present invention provides a Flume-based online multi-party aggregation method, which expands and adds a flume scheduling layer on the basis of the original flume process, so that the data analysis platform can support the online management of the flume agent. The specific steps are as follows:

[0021] S1. Deploy the flume scheduler on the machine where flume is located;

[0022] S2. The data analysis platform calls the flume scheduler to create an agent with flume;

[0023] The S3.flume scheduler generates the agent configuration according to the corresponding parameters;

[0024] S4. Then execute the flume creation command, and feed back the result to the flume scheduling layer.

[0025] Existing technical solutions:

[0026] Technology 1: Manual configuration (low efficiency, difficult to expand, error-prone)

[0027] Technology 2: The program is configured after logging in through ssh (permissions are not easy to control, and oper...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a Flume-based online multi-party aggregation method, which is characterized in that a Flume scheduling layer is extended and added on the basis of the original process of Flume, so that a data analysis platform can support online management of a Flume agent, and the method comprises the following specific steps: S1, a Flume scheduler is deployed on a machine where the Flumeis located; S2, the data analysis platform calls a flume scheduler to carry out flume creation for agent creation; S3, the flume scheduler carries out agent configuration generation according to thecorresponding parameters; and S4, a flume creation command is executed, and a result is reported to the flume scheduling layer. Online configuration of flume agents can be realized, and the method andthe system are superior to a configuration mode of manually logging in a production server; an account password or password-free login configuration of a machine where flume is located does not needto be configured; the invention is high in expansibility, the data analysis platform does not depend on flume, and flume can be replaced with other data aggregation components through calling of the middle layer.

Description

technical field [0001] The invention relates to the technical field of data analysis, in particular to an online multi-party aggregation method based on Flume. Background technique [0002] In traditional flume-based multi-party data aggregation solutions, there are generally two methods. [0003] 1. Manually configuring the Flume agent is inefficient, difficult to expand, and error-prone. [0004] 2. Logging in to the flume machine through the program SSH for automatic configuration, the authority is not easy to control, requires a lot of configuration in the early stage, and is not easy to maintain. Contents of the invention [0005] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide an online multi-party aggregation method based on Flume. Forwarding, etc., can not only solve the unpredictable risks of manual configuration, but also the difficulty of controlling the authority of automatic remote configura...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/445
CPCG06F9/44505
Inventor 吴朝亮刘剑群于传富刘奇彭大祥王佳伟
Owner TIANYI ELECTRONICS COMMERCE
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com