Unlock instant, AI-driven research and patent intelligence for your innovation.

A portable high-throughput big data acquisition method and system

A high-throughput, acquisition method technology, applied in transmission systems, electrical digital data processing, special data processing applications, etc., to achieve the effect of easy operation and convenient high-throughput data acquisition

Active Publication Date: 2019-05-10
INSPUR SOFTWARE CO LTD
View PDF3 Cites 15 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The technical task of the present invention is to provide a convenient high-throughput large data acquisition method and system to solve the problem of how to instantly collect and process data from different databases and different data structures

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A portable high-throughput big data acquisition method and system
  • A portable high-throughput big data acquisition method and system
  • A portable high-throughput big data acquisition method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0058] A kind of convenient high-throughput big data collection method of the present invention, this method is to send instruction by central server, each cluster server starts logstash, reads configuration parameters through the etcd component of datatrains, automatically generates the configuration file of response, Logstash reads Configuration file, collect various database data according to the configuration file, and send it to each server through Kafka in the form of a message after sorting out. Kafka and logstash cooperate to temporarily store the collected data according to the corresponding format at the consumer end; the central server calls related components to process The relevant data collected; the specific steps are as follows:

[0059] S1. The server generates different parameter message packets according to different business requirements, and sends the parameter message packets through scheduled tasks or manual operations;

[0060] S2. After receiving the m...

Embodiment 2

[0081] as attached figure 1 As shown, the provincial bureau, national bureau and 101 server are taken as examples. Among them, there are two ways to start the task: ①, the scheduled task of the provincial bureau; ②, receive the mq dispatch of the national bureau.

[0082] The specific steps for the provincial bureau to start the data collection task are as follows:

[0083] (1), the provincial bureau starts the data collection task through the scheduled task or receives the mq dispatch of the national bureau;

[0084] (2), data compression, encryption;

[0085] (3) Send a message to the central server;

[0086] (4), Timer polls for five minutes and asks the national bureau for transmission permission;

[0087] (5), whether to obtain the national bureau transmission license:

[0088] ①. If the transmission permission is obtained, Ftp uploads the data to the 242 storage server of the National Bureau;

[0089] (6) Determine whether the upload is successful:

[0090] ①. If...

Embodiment 3

[0103] The short-cut high-throughput large data acquisition system of the present invention includes a server end and a consumer end; the server end is used to generate different parameter message packets according to different business requirements, and send parameter message packets through timing tasks or manual operations. The data collection compression package performs data decryption and decompression operations to obtain the decompressed .csv file; the consumer side is used to analyze after receiving the message package sent by the server side, obtain the corresponding parameters, and then collect data according to the parameters, and Complete data collection through the dtp transmission channel, write the collected data into a .csv file, and perform compression and encryption operations on the .csv file, produce a data collection compressed package, and upload the data collection compressed package to the server through FTP or SFTP ( The two transmission methods are ma...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a portable high-throughput big data acquisition method and system, and belongs to data acquisition field. The technical problem to be solved by the invention is how to collectand process data of different data structures of different databases instantly. The adopted technical scheme is as follows: the device comprises a base, the invention discloses a portable high-throughput big data acquisition method. According to the method, a central server sends an instruction; each cluster server starts the logstash; reading the configuration parameters through an etcd componentof the data rains; the method comprises the steps that a response configuration file is automatically generated, Logstash reads the configuration file, various database data are collected according to the configuration file, the database data are sent to all servers in a message mode after being arranged, and the kafka and the Logstash are matched at a consumer side to temporarily store the collected data according to a corresponding format; and the central server calls related components and processes the collected related data. The invention also discloses a portable high-throughput big data acquisition system.

Description

technical field [0001] The invention relates to the field of data collection, in particular to a convenient high-throughput big data collection method and system. Background technique [0002] Information is the core basis for decision-making, and it is extremely important to collect and process data in a timely and effective manner. Due to the dispersion of data and the inconsistency of the structure, the data collection method of the transmission is very time-consuming and laborious. Therefore, how to instantly collect and process data from different databases and different data structures is a technical problem that needs to be solved urgently; [0003] In order to solve the above technical problems, people began to pay attention to the research of enterprise data integration. Try to reprocess the data in different systems to form an integrated, analysis-oriented environment, so as to be able to mine rules, extract knowledge, and assist decision-making from these massiv...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/174H04L29/08
Inventor 张晨光
Owner INSPUR SOFTWARE CO LTD