Method and apparatus for data network sampling

Abstract

Disclosed is an informed sampling technique for biasing a sample data set toward network data of interest for a particular application. Network data received at a network node (for example at a rate which is greater than a sampling rate for which the network node is configured) is chosen to be included in a sample set based on one or more predetermined signatures which are chosen to bias the sample set toward network data of interest for a particular application. For example, the sample set may be biased to include data of interest for fraud detection, spam detection, and intrusion detection. The particular signature(s) may be predefined by a user, or may be automatically generated by another network application. The invention may be implemented at various levels and nodes of a network. For example, the informed sampling may be implemented at a traffic monitoring function of a network router, a flow collector which receives network flow data from the router, or both.

Description

[0001] This application claims the benefit of U.S. Provisional Application No. 60 / 702,100 filed Jul. 22, 2005, which is incorporated herein by reference.BACKGROUND OF THE INVENTION [0002] The present invention relates generally to data sampling, and more particularly to improved sampling in data networks. [0003] Data networks, such as the Internet, transport large amounts of data, often in the form of data packets. As is well known, data packets are transmitted through a network via routers. Routers are network nodes that receive data packets on a network interface, inspect the destination address of the data packets, determine next hop routing, and output data packets on an appropriate interface for further routing through the network. The router also buffers received data packets from the time the packet is received until the time the packet is output from the router. A data packet may traverse multiple routers during its traversal of the network from a source node to a destinatio...

AI Technical Summary

Benefits of technology

[0014] In one embodiment, the data packets in the sample set may be aggregated and the router may generate network flow data based on the sample set. This network flow data may be a summary of the data packets communicated within particular network flows being handled by the router. This network flow data may be received by another network node (e.g., a network flow collector), and the informed sampling technique of the present invention may be applied at the network flow collector as well. Such an embodiment is advantageous, for example, when network flow data is generated by the router at a rate greater than the network flow collector can handle. Again, rather than sampling the network flow data at a pure 1 / n rate as known in the art, the flow collector chooses flow data to be included in a sample set based on one or more predetermined signatures which are chosen such that the sample set is biased toward flow data of interest for a particular application.

Problems solved by technology

However, with the dramatic increase in worldwide network traffic, even the fastest routers have difficulty just keeping up with their primary function of routing network data.

The addition of traffic monitoring to a router's functionality imposes an overhead cost, over and above the cost of the router's main routing function.

Another problem exists with respect to retrieval of the flow data from the router by the flow collector.

If the bandwidth of the connection between the router and the flow collector is insufficient to transfer all the flow data, some of the flow data may be lost.

Even if the bandwidth is sufficient to support the data transfer, the storage system of the flow collector may be incapable of keeping up with the transferred data and again, some of the flow data may be lost.

However, the use of 1 / n sampling, while heretofore required for acceptable router performance, generally renders the resulting flow data unusable for these additional purposes.

Since this type of network traffic monitoring must make inferences based on the network traffic, it is likely that certain packets required for such inferences will be lost during the 1 / n sampling, resulting in an unacceptable data set upon which to perform the required inferencing.

However, the problem with network sniffers is that they are an additional network element, and as such they are expensive to implement within a network.

Images