Data acquisition method and device based on Redis and Logstash

A collection method and data collection technology, applied in database indexing, structured data retrieval, database management system, etc., can solve the problems of increasing Redis application complexity, instability, Redis performance degradation, etc., to save time and labor costs , reduce data selection and conversion operations, and speed up processing effects

Active Publication Date: 2019-03-19
北京明朝万达科技股份有限公司
View PDF9 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In order to ensure its efficiency, Redis caches data in memory, and the storage space of memory is often limited, which requires Redis to periodically write data to different disks. When Redis is applied to data collection , because data collection needs to be stored in different disks for different data, a large amount of complex control logic needs to be written for Redis, which increases the

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data acquisition method and device based on Redis and Logstash
  • Data acquisition method and device based on Redis and Logstash
  • Data acquisition method and device based on Redis and Logstash

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0039] The technical solutions of the present invention will be further described below in conjunction with the accompanying drawings and through specific implementation methods.

[0040] In order to better illustrate the present invention and facilitate understanding of the technical solutions of the present invention, typical but non-limiting examples of the present invention are as follows: What needs to be particularly explained here is that the implementations listed in the description of the present invention are only for the convenience of illustrating the problem. The given exemplary implementation method should not be interpreted as the only correct implementation of the present invention, nor should it be interpreted as a limiting description of the protection scope of the present invention.

[0041] Data acquisition and storage in big data applications is the most basic function in big data applications. Its characteristic is to obtain enough diverse data as much as p...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data acquisition method based on Redis and Logstash. The data acquisition method comprises the steps that a data receiving server receives data pushed by an external system; storing the pushed data into a file system through the Redis; the Logstash collects the data from the file system; and the Logstash converts the data and sends the converted data to an application system. A high-performance storage mechanism of Redis is utilized, data processing can be accelerated, and through decoupling operation of different services, the stability of the services is improved, and the error probability is reduced.

Description

technical field [0001] The invention relates to the field of data collection and storage, in particular to a data collection method and equipment based on Redis and Logstash. Background technique [0002] In open source data systems, Redis, Memcached, etc. are commonly used. Among them, Redis is a key-value storage system. Compared with the distributed cache system Memcached, it supports relatively more value types, including string (string), list (linked list), set (collection), zset (sorted set--ordered collection) and hash (hash type). These data types support push / pop, add / remove, intersection, union, difference and richer operations, and these operations are atomic. On this basis, Redis supports sorting in various ways. Like Memcached, data is cached in memory to ensure efficiency. But the difference is that Redis will periodically write updated data to disk or write modification operations to additional record files, and on this basis, master-slave (master-slave) s...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/25G06F16/22
Inventor 董晓斌喻波王志海董爱华安鹏
Owner 北京明朝万达科技股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap