Streaming Method and System for Processing Network Metadata

Abstract

An improved method and system for processing network metadata is described. Network metadata may be processed by dynamically instantiated executable software modules which make policy-based decisions about the character of the network metadata and about presentation of the network metadata to consumers of the information carried by the network metadata. The network metadata may be type classified and each subclass within a type may be mapped to a definition by a unique fingerprint value. The fingerprint value may be used for matching the network metadata subclasses against relevant policies and transformation rules. For template-based network metadata such as NetFlow v9, an embodiment of the invention can constantly monitor network traffic for unknown templates, capture template definitions, and informs administrators about templates for which custom policies and conversion rules do not exist. Conversion modules can efficiently convert selected types and/or subclasses of network metadata into alternative metadata formats.

Description

CROSS REFERENCE TO RELATED APPLICATIONS[0001]This non-provisional application claims the benefit of provisional application No. 61 / 751,243 filed on Jan. 10, 2013, entitled “An Improved Streaming Method and System for Processing Network Metadata”, which application is incorporated herein in its entirety by this reference.[0002]This application is related to provisional application No. 61 / 556,817 filed on Nov. 7, 2011, entitled “A Streaming Method and System for Processing Network Metadata”, which application is incorporated herein in its entirety by this reference.[0003]This application also claims the benefit of provisional application No. 61 / 699,823, filed Sep. 11, 2012, entitled “A Streaming Method and System for Processing Network Metadata”, which application is incorporated herein in its entirety by this reference.[0004]This continuation-in-part application also claims the benefit of application Ser. No. 13 / 669,235, filed Nov. 5, 2012, entitled “A Streaming Method and System for...

AI Technical Summary

Benefits of technology

[0028]According to another embodiment of the present invention, the method and system may be implemented in a streaming fashion, i.e., processing the input network metadata as it arrives (“in real-time or near-real-time”) without the need to resort to persistent storage of the network metadata. This embodiment of the invention allows deployment of the system and method on a computer with limited memory and storage capacity, which makes the embodiment especially well suited for deployments in a computing cloud.

[0029]After processing a class member instance according to a policy or a plurality of policies, an embodiment of the present invention may provide an efficient method for converting the results of the policies' application into zero, one or more representations (“converter”) suitable for further processing by recipients of the converted network metadata or the original network metadata. As a result, the system and method disclosed herein is exceptionally well suited for deployments in existing environments where its output may be directed towards existing diverse components such as SIEM systems adapted for use with syslog metadata.

[0030]An embodiment of the invention provides a plurality of converters that may be customized for a particular class or classes of network metadata and / or output format, thereby increasing throughput of the system to better enable real-time or near-real-time services on the network. Further, in response to a heavy volume of a particular class or subclass of network metadata, multiple instances of the customized working thread and / or conversion modules can be instantiated to operate in parallel to further enhance system performance and throughput.

[0031]Furthermore, an embodiment of the present invention is able to ensure integrity of the converted network metadata by appending message authentication codes. This embodiment of the invention enables sophisticated network metadata recipients to verify authenticity of the received information.

[0032]Yet another embodiment of this invention is the ability to deploy the system and method in a fashion transparent to the existing network ecosystem. This embodiment does not require any change in the existing network components' configuration.

[0033]Another embodiment of the present invention provides a method and apparatus for describing network metadata processing and conversion rules either in visual or in textual terms or a combination thereof. Once the policies' description is complete and verified to be non-contradicting, the policies and converters applicable to a class member subject to the rules may be instantiated as one or a plurality of executable modules simultaneously derived from one or a plurality of the network metadata processing and conversion rules definitions. As a result, systemic policy consistency is achieved across a plurality of modules. Furthermore, the binary nature of the modules implementing the policies and conversion rules makes the system capable of handling the input network metadata at rates significantly exceeding processing rates in environments which interpret comparable processing rules.

Problems solved by technology

Some of the issues created by the Big Data problem include an inability to analyze and store massive amounts of machine-generated data that often exists in different formats and structures.

1. Too much data to analyze in real time to acquire timely insight into network conditions.

2. Data arrives in different formats from different device types on a network, making correlation of data from different device types difficult and slow; and

3. Too much data to store (e.g., for later analysis and / or for compliance with data retention requirements).

Images