Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method and system for acquiring log data

A log and data technology, applied in the field of data communication, can solve problems such as low throughput and poor log data flexibility, and achieve the effect of improving flexibility

Inactive Publication Date: 2013-11-20
GUANGZHOU VIPSHOP INFORMATION TECH
View PDF0 Cites 57 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Based on this, it is necessary to provide a method and system for obtaining log data for the above-mentioned problems of poor flexibility and low throughput of log data collected by Flume

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for acquiring log data
  • Method and system for acquiring log data
  • Method and system for acquiring log data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0018] see figure 1 , figure 1 is a schematic flowchart of the first embodiment of the method for acquiring log data in the present invention.

[0019] The method for obtaining log data in this embodiment includes the following steps:

[0020] Step 101, the first Flume obtains log data from an application server.

[0021] Step 102, the first Flume transmits the acquired log data to Kafka, and the Kafka converts the received log data into a Kafka message queue.

[0022] The method for obtaining log data described in this implementation mode transmits the log data in the application server to Kafka through the first Flume, and converts the log data into a Kafka message queue through Kafka. When users obtain log data from Kafka, they only need to connect to Kafka is enough, without the need for cumbersome restart and insertion operations, which can improve the flexibility of obtaining log data.

[0023] Wherein, for step 101, the Flume preferably can capture log data from the...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method and a system for acquiring log data. The method comprises the following steps: first Flume acquires log data from an application server; the first Flume transmits the acquired log data to Kafka, and the Kafka converts the received log data into a Kafka message queue. According to the method and the system for acquiring log data, the log data in the application server are transmitted to the Kafka via the first Flume, the log data are converted into the Kafka message queue via the Kafka, and a user only needs to link to the Kafka during acquisition of the log data from the Kafka, so that complex restart and insertion operation is not needed, and the acquisition flexibility of the log data can be enhanced.

Description

technical field [0001] The invention relates to the technical field of data communication, in particular to a method and system for acquiring log data. Background technique [0002] With the development of e-commerce technology, the pressure on the back-end server of the network is also increasing. At the same time, the "data" that needs to be processed is also increasing geometrically. Real-time and accurate collection, transmission, and calculation of massive logs have become an e-commerce urgent demands. In the prior art, the flume-ng technology involved in collecting logs by twitter is mainly used. Flume is a distributed, reliable, high-performance tool for collecting, aggregating, and transferring large amounts of log data from different data sources to a central data source. Flume has three important concepts, source, channel, and sink. These three logical concepts form a Flume agent. Source defines the source of data (such as a file), Sink defines the export of dat...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08H04L12/58
Inventor 姚仁捷
Owner GUANGZHOU VIPSHOP INFORMATION TECH
Features
  • Generate Ideas
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com