A method for data transmission based on logstash + kafka

A data transmission method and data technology, applied in transmission systems, electrical components, etc., can solve problems such as poor scalability, interface development workload, and inability to guarantee the integrity of transmission data, and achieve the effect of achieving performance.

Inactive Publication Date: 2018-12-28
INSPUR SOFTWARE CO LTD
View PDF5 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0008] The main disadvantages of using FTP for file transfer are: the integrity of the transferred data cannot be guaranteed; the scalability is poor;
However, most of the middleware transmission methods currently have the disadvantages of low efficiency, corresponding interface development workload, and inflexible data extraction.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method for data transmission based on logstash + kafka

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0038] Configuration device:

[0039] A data transmission device based on logstash+kafka, including a business data production end and a business data consumption end;

[0040] The business data production end is used to collect business data generated by the business system; the business data production end includes: an input module and an output module; the input module is for data collection, and automatically collects required business in the business database data; the output module is data transmission, which converts the collected data into messages and sends them to the kafka queue of the data production end.

[0041] The business data consumption end is used to process the collected data and centralize the data into a unified data platform. Described business data consumption end comprises consumption end kafka data middleware and consumption end logstash; Described consumption end kafka data middleware is used for transmitting and accepting the business data message...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data transmission method based on logstash + kafka, which comprises operation of service data producing end and operation of service data consuming end. 1) automatically collecting the service data in the service system database through the service data production terminal logstash, and transmitting the service data to the middleware kafka; 2) the middleware kafka collects the data transmitted by each service data producing end and uniformly sends the data to the service data consuming end logstash; 3) the service data consuming terminal logstash receives the messageof the middleware kafka, converts the message into a data file, and stores the message in the target database. Compared with the prior art, the invention has the advantages of simple configuration andoperation, high processing speed, and can effectively realize the operation of collecting, processing, storing and the like of the dispersed service data, and can quickly and conveniently build the transmission channel between the service data production end and the service data consumption end, and has good expansibility, and reduces the difficulty of development and operation and maintenance. The invention has the advantages of simple configuration and operation, high processing speed, and effectively realizes the operations of collecting, processing, storing, etc.

Description

technical field [0001] The invention relates to the application field of high-speed data collection, in particular to a data transmission method based on logstash+kafka. Background technique [0002] In the process of enterprise application, each information enterprise has its own information system, but the data information between each enterprise cannot be shared, and there is also a lack of a unified data platform to integrate, process, mine and analyze these business data. Seriously hindered the overall process of enterprise information construction. In order to solve this problem, people began to study various data transmission methods, trying to efficiently and safely collect data from different systems, and further mining and analyzing data on the basis of a unified and centralized data platform. rules to assist in decision-making. The existing data transmission methods are mainly divided into the following types: [0003] 1) CDN technology [0004] By adding a ne...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08
CPCH04L67/06H04L67/30
Inventor 颜朋
Owner INSPUR SOFTWARE CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap