Data acquisition method and device based on Redis and Logstash

Abstract

The invention provides a data acquisition method based on Redis and Logstash. The data acquisition method comprises the steps that a data receiving server receives data pushed by an external system; storing the pushed data into a file system through the Redis; the Logstash collects the data from the file system; and the Logstash converts the data and sends the converted data to an application system. A high-performance storage mechanism of Redis is utilized, data processing can be accelerated, and through decoupling operation of different services, the stability of the services is improved, and the error probability is reduced.

Description

technical field [0001] The invention relates to the field of data collection and storage, in particular to a data collection method and equipment based on Redis and Logstash. Background technique [0002] In open source data systems, Redis, Memcached, etc. are commonly used. Among them, Redis is a key-value storage system. Compared with the distributed cache system Memcached, it supports relatively more value types, including string (string), list (linked list), set (collection), zset (sorted set--ordered collection) and hash (hash type). These data types support push / pop, add / remove, intersection, union, difference and richer operations, and these operations are atomic. On this basis, Redis supports sorting in various ways. Like Memcached, data is cached in memory to ensure efficiency. But the difference is that Redis will periodically write updated data to disk or write modification operations to additional record files, and on this basis, master-slave (master-slave) s...

AI Technical Summary

Problems solved by technology

[0004] In order to ensure its efficiency, Redis caches data in memory, and the storage space of memory is often limited, which requires Redis to periodically write data to different disks. When Redis is applied to data collection , because data collection needs to be stored in different disks for different data, a large amount of complex control logic needs to be written for Redis, which increases the application complexity of Redis, and a high application degree usually leads to Redis performance degradation and instability The phenomenon

The log analysis system uses Logstash to realize the data collection function of Redis, but it is limited to the storage and collection of log data types, and the logical coupling of the two is complicated, so that its application has relatively large limitations

Images