Log data processing method and system

A data processing system and data processing technology, applied in the computer field, can solve problems affecting system scalability, large input data volume, small output data volume, etc., to avoid performance bottlenecks, realize dynamic load balancing, and good scalability Effect

Active Publication Date: 2013-11-20
INST OF INFORMATION ENG CHINESE ACAD OF SCI
View PDF2 Cites 24 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the data characteristics of log analysis are large input data volume (that is, request data volume) and small output data volume (that is, response data volume). The forwarding mode of the load balancer cannot adapt to this characteristic, which will cause a large amount of log data to pass through the load balancer. , forming a single-point bottleneck, seriously affecting the scalability of the system

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Log data processing method and system
  • Log data processing method and system
  • Log data processing method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0062] The principles and features of the present invention are described below in conjunction with the accompanying drawings, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.

[0063] figure 2 It is a schematic diagram of the principle of the log data processing method of the present invention. Such as figure 2 As shown, the log data processing method of the present invention uses a cross-connection mode to perform load balancing instead of forwarding by a load balancer. The so-called "cross-interconnection mode" refers to the cross-connection between the sender and the server. Each sender is connected to all servers. Similarly, each server is also connected to all senders. The log data is directly transmitted from the sender to the server. server. The log analysis system using the log data processing method of the present invention includes multiple sending ends, multiple servers, a ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a log data processing method and system, wherein the log data processing method comprises the following steps: 1, each sending end is set to be connected with all servers; 2, the sending ends distribute a log identifier to each log self-generated; 3, the sending ends inquire whether the log identifier of the log to be sent in a sending record table or not, if so, the sending ends obtain a server identifier corresponding to the log identifier of the log to be sent from the sending record table, wherein the server corresponding to the server identifier is the selected server; and 4, the sending ends directly send the log data to be sent to the selected server. The method and system provided by the invention prevent a load balancer from becoming performance bottleneck, so that the system applying the log data processing method has good expandability, and better adapts the characteristics of large input data size and small output data size in log analysis.

Description

technical field [0001] The invention relates to the field of computers, in particular to a log data processing method and system. Background technique [0002] Logging is a relatively broad concept in the computer field. Any program may output logs, such as cluster monitoring programs, intrusion detection programs, and Web server programs. The specific format of logs varies widely, but they are essentially records of events or behaviors. Logs usually include event subjects, time of occurrence, event content, etc. Log analysis is to extract valuable information from log data, such as detecting abnormal events and making corresponding adjustments. [0003] In most scenarios, logs have the characteristics of large data volume, strong real-time performance, and dynamic fluctuations. In order to process logs in a timely and efficient manner, log analysis systems mostly adopt a distributed architecture, that is, there are multiple senders and multiple back-end servers in the syst...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/08
Inventor 韩冀中王旻李勇张章孟丹
Owner INST OF INFORMATION ENG CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap