Compression method for collecting data in high-speed HTTP network

Abstract

The invention discloses a compression method for collecting data in a high-speed HTTP network. The compression method for collecting data in the high-speed HTTP network comprises the following steps:a network data collector performs HTTP data collection and reception, and judges whether the HTTP data is locally cached or not; if the HTTP data is locally cached, the load of the data is deleted, and a cache identifier is used to replace the load content; if the HTTP data is not locally cached, the HTTP load is cached and the cache identifier is added; and the HTTP data is added to a processingqueue to be compressed, so as to perform compression on the data except the cached load data. By means of the compression method for collecting data in the high-speed HTTP network, the amount of the HTTP data is greatly reduced through the caching mode; and good compression rate can be obtained through the lossless compression algorithm for compression by means of the repeated redundancy of mass HTTP collected data in the compression processing queue, therefore, the compression requirements for collecting data in the high-speed HTTP network are satisfied.

Description

technical field [0001] The invention relates to the technical fields of computers and networks, in particular to a compression method for collecting data from a high-speed HTTP network. Background technique [0002] The HTTP protocol is the most widely used protocol on the Internet, and the HTTP data in the network is an important data source for network security applications such as network security audit and network traffic analysis. Through the analysis and statistics of HTTP in the wired network, its traffic accounts for 24.7% of the total traffic. At present, when the bandwidth of network edge nodes reaches 10G, and the bandwidth of network core nodes reaches 40G and 100G, the collected HTTP network data traffic and concurrency are extremely large. For the application scenario where the network data collector transmits HTTP network collection data to the remote cloud analysis system for subsequent analysis and processing, the required collection data transmission requi...

AI Technical Summary

Problems solved by technology

[0003] The purpose of the present invention is to overcome the problem that the current high-speed HTTP network collection data compression is difficult to realize, provide a kind of compression method facing the high-speed HTTP network collection data, utilize HTTP service content, HTTP protocol field and IP Baotou etc. Cache content loads such as HTTP audio, video, and text, and perform lossless compression on multiple HTTP concurrent IP packets in the processing queue, effectively reducing the amount of data collected by HTTP networks, especially suitable for network operation and maintenance audits, banking telecommunications and other services Web application scenarios with fixed service content such as website network behavior auditing

Images