Log data processing method, system and device and storage medium

A data processing and log technology, applied in the field of information processing, can solve problems such as inability to support large-scale data and low coverage

Pending Publication Date: 2019-04-26
PINGAN PUHUI ENTERPRISE MANAGEMENT CO LTD
View PDF9 Cites 22 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The main purpose of the present invention is to provide a log data processing method, system, device and storage medium, aiming to solve the problem that the existing log analysis platform has low coverage and cannot support large-scale data when collecting and analyzing log data question

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Log data processing method, system and device and storage medium
  • Log data processing method, system and device and storage medium
  • Log data processing method, system and device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0046] It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0047] refer to figure 1 , figure 1 It is a schematic structural diagram of a log data processing device in the hardware operating environment involved in the solution of the embodiment of the present invention.

[0048] Such as figure 1 As shown, the log data processing device may include: a processor 1001 , such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002 , a user interface 1003 , a network interface 1004 , and a memory 1005 . Wherein, the communication bus 1002 is used to realize connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 10...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a log data processing method, System, device and storage medium, The method comprises the following steps: a data transmission component determines a target data source and a data type of data in the target data source according to a data acquisition request, calls a preset script according to the data type to perform data acquisition, performs initialization processing onacquired log data and sends the log data to a preset message cluster; The preset message cluster generates a message queue according to the initialization log data and sends the message queue to the data transmission component; The data transmission component determines a consumption sequence according to the message queue; and consuming the log data stored in the preset message cluster accordingto the consumption sequence. According to the log data collecting method and device, the data source log data are collected through the data searching assembly and the consumed data are stored throughthe data searching assembly, and the corresponding scripts are called according to the data types of the data stored in the data source to collect the data source log data, so that the coverage of data collection is increased, and diversified collection of the log data is achieved.

Description

technical field [0001] The present invention relates to the technical field of information processing, in particular to a log data processing method, system, device and storage medium. Background technique [0002] The ELK platform is generally referred to as a log analysis platform, where E, L, and K are the initials of the names of three open source products (Elasticsearch, Logstash, and Kibana). ElasticSerach is a distributed search engine based on an open source search engine, and also provides data storage functions; Logstash is mainly used for filtering, modifying and collecting log data, and supports various data acquisition methods; Kibana mainly It is used to present the above two to the user through the Hypertext Transfer Protocol (HyperText Transfer Protocol, HTTP) protocol, so that the user can use it. When the existing ELK platform collects and analyzes data from external system databases or application logs and other data sources, it often only collects, proce...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2458G06F16/2455G06F16/95H04L12/24
CPCH04L41/069
Inventor 冉茂树卢荣荣冯欣苏丹
Owner PINGAN PUHUI ENTERPRISE MANAGEMENT CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap