Distributed network traffic retrieval method and device

Abstract

The invention belongs to the technical field of real-time capture, storage and retrieval of network traffic, and particularly relates to a distributed network traffic retrieval method and device. The method comprises: acquiring data; recombining a session and extracting metadata; writing the session into a disk; and performing a distributed retrieval algorithm DSearch and data storage. The invention provides a distributed index establishment algorithm DIndex and a distributed retrieval algorithm DSearch, which can support distributed real-time high-speed network full-traffic storage, indexing and retrieval services, and satisfy the use requirements of users for high bandwidth and multiple branches.

Description

technical field [0001] The invention belongs to the technical field of real-time capture, storage and retrieval of network traffic, and in particular relates to a distributed network traffic retrieval method and device. Background technique [0002] In recent years, with the rapid development of the Internet, network bandwidth has increased dramatically, and the development of the Internet has brought more challenges to network security. Similar to surveillance cameras and surveillance video storage, query, and forensics equipment, real-time monitoring, storage analysis, and source traceability of network traffic have become a key method to ensure network security. In this way, network administrators can retrospectively analyze the abnormal network behavior and application communication data that have occurred, which is widely used in financial transactions, network forensics, network security and other fields. [0003] For the field of real-time network traffic storage and...

AI Technical Summary

Problems solved by technology

However, there are the following problems. This application is not suitable for large traffic or multi-branch scenarios. These scenarios require multiple nodes to form a cluster, and the index algorithm proposed in this application is a single-host algorithm.

In addition, the index of this application is stored on the SSD array, which is mainly cold data, imported into memory on demand, and lacks the design of cold and hot separation, resulting in high I / O access to the index when the user's search changes, and a certain degree of query time. loss

Finally, the application also has certain limitations in terms of retrieval, such as the lack of capabilities such as IP V6 query, mask retrieval, range retrieval, and Payload retrieval.

[0006] Due to the shortcomings of the above methods in real-time storage and retrieval of network traffic, these methods cannot be really applied to high-bandwidth links and multi-branch user environments.

Images