Data management method, middleware and data management system

A technology of data management and middleware, applied in the field of middleware, which can solve problems such as unable to write logs into Kafka

Inactive Publication Date: 2018-06-22
FLYING FOX INFORMATION TECH TIANJIN CO LTD
View PDF5 Cites 19 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In view of this, the purpose of the present invention is to provide a data management method, middleware and data management system, which is used to solve the problem that early Flume versions such as Flume OG do not have a Kafka plug-in, and the collected logs cannot be written into Kafka

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data management method, middleware and data management system
  • Data management method, middleware and data management system
  • Data management method, middleware and data management system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0049] For the sake of reference and clarity, the technical terms, abbreviations or abbreviations used in the following text are summarized as follows:

[0050] Flume: Flume is a highly available, highly reliable, and distributed massive log collection, aggregation, and transmission system provided by Cloudera. Flume supports customizing various data senders in the log system for data collection; at the same time, Flume Provides the ability to simply process data and write to various data recipients (customizable).

[0051] Kafka: Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action streaming data in consumer-scale websites. Such actions (web browsing, searching and other user actions) are a key factor in many social functions on the modern web. These data are usually addressed by processing logs and log aggregation due to throughput requirements. This is a viable solution for systems like Hadoop that log data and analyze offli...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data management method, middleware and a data management system. The method and the middleware obtain the collection data of a data collection party to generate the theme information of the collection data and package the collection data and the theme information, the collection data which packages the theme information is sent to a data storage party (to enable the data storage party to store the collection data to a storage position corresponding to the corresponding theme according to the theme information of the collection data), the collection data of the data collection party is written into the data storage party, so that the scheme of the invention can be used for writing the collection data of data collection parties including Flume OG and the like into data storage parties including Kafka and the like through the middleware, and the problem in the prior art that the collected log can not be written into Kafka since an early Flume version is not provided with a Kafka plugin is solved.

Description

technical field [0001] The invention belongs to the technical field of middleware, and in particular relates to a data management method, middleware and a data management system. Background technique [0002] Flume OG is a highly available, highly reliable, and distributed massive log collection, aggregation, and transmission system provided by Cloudera. [0003] Flume supports customizing various data senders in the log system to collect data. The collected data needs to be written to storage systems such as Kafka for real-time computing and data cleaning. However, since early Flume versions such as Flume OG do not have Kafka plug-ins, The collected logs cannot be written into storage systems such as Kafka, so a middleware needs to be developed to receive Flume logs and write them into storage systems such as Kafka. Contents of the invention [0004] In view of this, the object of the present invention is to provide a data management method, middleware and data managemen...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30G06F9/50
CPCG06F9/505G06F16/116G06F16/1815
Inventor 颜健
Owner FLYING FOX INFORMATION TECH TIANJIN CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap