Unlock instant, AI-driven research and patent intelligence for your innovation.

Multi-source log data processing system and method in cloud environment

A data processing system and data preprocessing technology, applied in the field of multi-source log data processing systems, achieves the effects of high scalability, reduced software operation and maintenance workload, and simple log processing process

Pending Publication Date: 2020-01-24
G CLOUD TECH
View PDF0 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0009] The technical problem solved by the present invention is to provide a multi-source log data processing system and method in a cloud environment; realize secondary processing of multi-source log data, and avoid problems such as high coupling and insufficient scalability

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multi-source log data processing system and method in cloud environment
  • Multi-source log data processing system and method in cloud environment
  • Multi-source log data processing system and method in cloud environment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0032] See figure 1 As shown, the system for secondary processing of multi-source log data streams of the present invention includes 4 functional modules: 1. Log source input module, which provides the function of multi-source log access; 2. Data preprocessing module, which provides data classification function; 3. 1. Data processing module, which provides flexible log processing plug-in management functions; 4. Data storage module, which provides back-end data storage management functions.

[0033] 1. Log source access module

[0034] The log source access module connects text format logs from different sources to the log data stream processing platform. At this point, what is accessed is the original and heterogeneous logs generated by different service components.

[0035] 2. Data preprocessing module

[0036] The data preprocessing module unifies and centrally manages logs from different sources and identifies tags at the log receiving end. In the present invention, a ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a multi-source log data processing system and method in a cloud environment. A log source input module of the system provides a multi-source log access function; a data preprocessing module provides a data classification function; a data processing module provides a log processing plug-in management function; and a data storage module provides a back-end data storage management function. The method comprises the following steps of: performing tagging processing on a text log data stream through a two-dimensional vector (ip, path); editing a plugin_playbook. yml file, defining a log data stream processing link, and determining a processing sequence of logs among different plugs; scanning and loading all plug-ins under a plugins catalogue; analyzing the plug-in_playbook.yml file, constructing a complete data stream analysis link, and gradually streaming an original log file to different plug-ins for processing in sequence according to a link identifier. Accordingto the method, the problems of high coupling, insufficient expansibility and the like of multi-source log processing are solved; the method can be used for multi-source log data processing.

Description

technical field [0001] The invention relates to the technical field of log data processing, in particular to a multi-source log data processing system and method in a cloud environment. Background technique [0002] With the rapid development of various distributed technologies and the maturity of rich open source distributed frameworks, traditional large-scale single programs are gradually deconstructed and transformed into service-oriented architecture (SOA). Among them, the microservice architecture is a typical representative. However, there is an obvious problem with this architecture. Since all service components are deployed in a distributed manner, when an exception occurs in the system, the workload of troubleshooting the exception log will be very heavy. Therefore, in the face of this kind of multi-source and heterogeneous log data, it is very necessary to analyze and store it in a structured way to support later operation and maintenance work. Therefore, a unifi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F11/34
CPCG06F11/3476
Inventor 罗平季统凯
Owner G CLOUD TECH