Streaming Method and System for Processing Network Metadata

Abstract

An improved method and system for processing network metadata is described. Network metadata may be processed by dynamically instantiated executable software modules which make policy-based decisions about the character of the network metadata and about presentation of the network metadata to consumers of the information carried by the network metadata. The network metadata may be type classified and each subclass within a type may be mapped to a definition by a unique fingerprint value. The fingerprint value may be used for matching the network metadata subclasses against relevant policies and transformation rules. For template-based network metadata such as NetFlow v9, an embodiment of the invention can constantly monitor network traffic for unknown templates, capture template definitions, and informs administrators about templates for which custom policies and conversion rules do not exist. Conversion modules can efficiently convert selected types and / or subclasses of network metadata into alternative metadata formats.

Description

CROSS REFERENCE TO RELATED APPLICATIONS[0001]This non-provisional application claims the benefit of provisional application No. 61 / 751,243 filed on Jan. 10, 2013, entitled “An Improved Streaming Method and System for Processing Network Metadata”, which application is incorporated herein in its entirety by this reference.[0002]This application is related to provisional application No. 61 / 556,817 filed on Nov. 7, 2011, entitled “A Streaming Method and System for Processing Network Metadata”, which application is incorporated herein in its entirety by this reference.[0003]This application also claims the benefit of provisional application No. 61 / 699,823, filed Sep. 11, 2012, entitled “A Streaming Method and System for Processing Network Metadata”, which application is incorporated herein in its entirety by this reference.[0004]This continuation-in-part application also claims the benefit of application Ser. No. 13 / 669,235, filed Nov. 5, 2012, entitled “A Streaming Method and System for...

AI Technical Summary

Benefits of technology

The present invention provides a system and method for analyzing large amounts of metadata in real-time and reducing the volume of incoming data through real-time data reduction techniques. The invention also allows for the identification and description of network metadata processing and conversion rules, resulting in a consistent policy across multiple modules. The system is capable of handling large volumes of network metadata at faster rates than other processing environments.

Problems solved by technology

Some of the issues created by the Big Data problem include an inability to analyze and store massive amounts of machine-generated data that often exists in different formats and structures.

1. Too much data to analyze in real time to acquire timely insight into network conditions.

2. Data arrives in different formats from different device types on a network, making correlation of data from different device types difficult and slow; and

3. Too much data to store (e.g., for later analysis and / or for compliance with data retention requirements).

Images