Unlock instant, AI-driven research and patent intelligence for your innovation.

A management method, device and system for a flume cluster

A management method and cluster technology, applied in the field of big data, can solve problems such as poor management performance of Flume clusters, and achieve the effect of improving management performance

Active Publication Date: 2022-05-13
CHINA MOBILE SUZHOU SOFTWARE TECH CO LTD +1
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] Embodiments of the present invention provide a management method, device and system of a Flume cluster, which are used to solve the technical problem of poor management performance of the Flume cluster in the prior art, and improve the management performance of the Flume cluster

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A management method, device and system for a flume cluster
  • A management method, device and system for a flume cluster
  • A management method, device and system for a flume cluster

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0090] Please refer to Figure 4 , the embodiment of the present invention provides a management method of a Flume cluster, which is applied to the server side, and the method includes:

[0091] S101: Receive a start request for starting a Flume task sent by the web interface;

[0092] S102: According to the current resource usage of the Flume cluster, the running resources of the Flume task, and the priority of the Flume task, assign the Flume task to the first machine in the Flume cluster;

[0093] S103: Generate a startup program for the Flume task;

[0094] S104: According to the startup program, control the first machine to start the Flume task, and limit the running resources of the Flume task through Container containerization technology, so that the first machine can start multiple Flume tasks.

[0095] In the specific implementation process, the specific implementation process of step S101 to step S104 is as follows:

[0096] First, the server side receives the sta...

Embodiment 2

[0124] Based on the same inventive concept as Embodiment 1 of the present invention, please refer to Figure 11 , the embodiment of the present invention also provides a management device for a Flume cluster, which is applied to the server side, including:

[0125] The receiving unit 10 is used to receive the start request for starting the Flume task sent by the web interface;

[0126] The assignment unit 20 is configured to assign the Flume task to the first machine in the Flume cluster according to the current resource usage of the Flume cluster, the running resources of the Flume task, and the priority of the Flume task;

[0127] The first generation unit 30 is used to generate the startup program for the Flume task;

[0128] The processing unit 40, according to the startup program, controls the first machine to start the Flume task, and limits the running resources of the Flume task through Container containerization technology, so that the first machine can start a plura...

Embodiment 3

[0144] Based on the same inventive concept as Embodiment 1 of the present invention, please refer to Figure 12 , the embodiment of the present invention also provides a management system of a Flume cluster, the system includes:

[0145] The web interface end 50 is used to send a start request for starting the Flume task;

[0146] The server end 60 is configured to receive the startup request; assign the Flume task to the Flume according to the current resource usage of the Flume cluster, the running resources of the Flume task, and the priority of the Flume task The first machine in the cluster; generating a start-up program for the Flume task; according to the start-up program, controlling the first machine to start the Flume task, and limiting the operation resources of the Flume task through Container containerization technology, So that the first machine can start multiple Flume tasks. Wherein, the server end 60 includes each unit module described in the second embodime...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The present invention discloses a management method, device and system of a Flume cluster, wherein the method includes: receiving a startup request sent by a web interface terminal for starting a Flume task; according to the current resource usage of the Flume cluster, the The running resources of the Flume task and the priority of the Flume task, the Flume task is assigned to the first machine in the Flume cluster; the startup program for the Flume task is generated; according to the startup program, the control The first machine starts the Flume task, and defines the running resources of the Flume task through Container containerization technology, so that the first machine can start multiple Flume tasks. It is used to solve the technical problem of poor management performance of the Flume cluster in the prior art, and improves the management performance of the Flume cluster.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a management method, device and system of a Flume cluster. Background technique [0002] Flume is a highly available, highly reliable, and distributed massive log collection, aggregation, and transmission log collection system provided by Cloudera. It supports customizing various data senders (such as Kafka, HDFS, etc.) in the log system, which is convenient Data collection. It has a simple and flexible architecture based on streaming data flow, reliable failover capability and fault tolerance, and good robustness. [0003] In the existing technology, it is impossible to automatically manage the Flume cluster. For the resources of the entire cluster, at present, only manual calculations can be used to allocate resources for new Flume tasks. Moreover, the management and operation of the start and stop of Flume tasks can only be done artificially. Executed manually on ea...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): H04L67/10H04L67/1001H04L67/61H04L67/02H04L67/30H04L41/0253H04L41/0894H04L41/5054
CPCH04L67/10H04L67/02H04L67/30H04L41/0253H04L41/0893H04L41/5054H04L67/1001H04L67/61
Inventor 张永曦王玉雷廖光贤李宏源张巧龙黄海波
Owner CHINA MOBILE SUZHOU SOFTWARE TECH CO LTD