Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Log collection server, data packet delivering and log clustering methods and network

A technology for collecting servers and data packets, applied in data exchange networks, digital transmission systems, electrical components, etc., can solve the problem of poor scalability, low efficiency of data collection and clustering, and processing delays under large-scale network logs when a single node fails Major and other issues

Inactive Publication Date: 2012-10-10
BEIJING UNIV OF POSTS & TELECOMM
View PDF4 Cites 31 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0012] In order to solve the problem that the network data collection and clustering scheme in the prior art cannot solve the problems of single node failure and large-scale network log processing delay, etc., there are problems of low data collection and clustering efficiency and poor scalability, the present invention implements The example provides a log collection server, data package distribution, log clustering method and network

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Log collection server, data packet delivering and log clustering methods and network
  • Log collection server, data packet delivering and log clustering methods and network
  • Log collection server, data packet delivering and log clustering methods and network

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0068] Such as figure 1 As shown, it is a schematic diagram of a DHT-based distributed network data collection and clustering model (hereinafter referred to as "model") provided by Embodiment 1 of the present invention, wherein,

[0069] DHT network: The DHT network consists of "log collection servers". In this embodiment 1, the "log collection server" is mapped to a point in the Chord ring through a consistent hash function such as SHA-1, and the label of the "log collection server" in the DHT network is represented by ServerID(i). The main function of the DHT network is to receive, store and distribute network data packets under the distributed network structure. Once a "log collection server" leaves or a new "log collection server" joins the network, the DHT network needs to update its own information.

[0070] Log collection server: Each "log collection server" is connected to the export device (such as router / switch) that needs to collect data. The log collection serve...

Embodiment 2

[0074] Such as figure 2 As shown, it is a schematic structural diagram of the log collection server provided by Embodiment 2 of the present invention, including a detector unit 11, an analysis unit 12, a Hash distribution unit 13, a clustering and log forwarding unit 14, and a database unit 15, wherein,

[0075] The detector unit 11 is used for bypass monitoring of data flow and data collection.

[0076] As described in Embodiment 1, each log collection server is connected to an egress device (such as a router / switch) that needs to collect data, that is, the log collection server needs to bypass monitor the data stream and collect the corresponding data , the detector unit 11 is used to collect data.

[0077] The parsing unit 12 is used to analyze the data in real time, extract the key field of the data packet, and send it to the Hash distribution unit 13.

[0078] The main job of the parsing unit 12 is to analyze the data collected by the detector unit 11 in real time, and...

Embodiment 3

[0091] Such as image 3 Shown is a flowchart of the principle of the data packet distribution method provided by Embodiment 3 of the present invention, wherein,

[0092] Step 21, the log collection server captures and analyzes the data packets.

[0093] The log collection server is located in the DHT network, and captures and analyzes data packets by monitoring the data stream. One of the purposes of parsing is to obtain the key field of the data packet. The key field of the data packet includes the source address IP, destination address IP, source port, destination port and transport layer protocol quintuple information of the data packet.

[0094] Step 22, according to the key field of the data packet, run the Hash function to obtain the Key value of the data packet.

[0095] Here, the Key value of the packet needs to be calculated according to the following formula:

[0096] Key=hash(SrcIP+DstIP+SrcPort+DstPort+Proto)

[0097] Among them, SrcIP is the source address IP...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a log collection server which comprises a detector unit, an analytical unit, a Hash shunt unit, a cluster and log forwarding unit and a database unit. The detector unit is used for data flow bypass monitoring and data acquiring; the analytical unit is used for analyzing data in real time, extracting key domains of data packets and sending the key domains of the data packets to the Hash shunt unit; the Hash shunt unit is used for performing a Hash calculation on the key domains of the data packets and delivering the data packets; the cluster and log forwarding unit is used for identifying the cluster to which log records of the log collection server belong and calculating cluster center of the log collection server; and the database unit is used for storing log records that belong to the log collection server. In the invention, fast shunt of data packets, distributed storage of network logs, load balancing of log servers are realized by performing Hash calculation on the key domains of data packets, and efficiency and scalability of data acquiring and clustering are improved.

Description

technical field [0001] The invention relates to the technical field of computer network communication, in particular to a log collection server, data package distribution, log clustering method and network. Background technique [0002] Data collection is the basis of network management. Currently, network monitoring technology is generally used in network management, mainly because a single monitoring agent is responsible for data collection for each monitoring network. Data collection technologies for IP networks are generally divided into three categories: based on Simple Network Management Protocol (SNMP), based on Net Flow, and based on full traffic mirroring. With the increasing scale of network service requests and the diversification of network services, facing the explosive growth of network data traffic and rapid changes in network service information, the operation of the network service information security supervision system requires a large amount of data. At ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): H04L12/24H04L29/08
Inventor 许长桥关建峰张宏科曹远龙权伟刘诗维文新赵付涛
Owner BEIJING UNIV OF POSTS & TELECOMM
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com