Fpga-based message deduplication method, device and fpga chip

Abstract

This application discloses a message de-duplication method based on FPGA. In this method, the look-up table based on the hash value as the reading index address is called to perform repeated matching and comparison of messages based on pipeline operation, which improves the efficiency of de-duplication comparison. ; Since the hash value of traffic data is greatly shortened compared to the data length of traffic data itself, matching and comparison based on hash value can improve the efficiency of matching comparison; and in this method, hash value is used as the read index address, according to Values ​​can be directly read from the historical entries in the lookup table for matching and comparison, avoiding traversal matching of historical traffic data, significantly improving the efficiency of repeated comparisons, and realizing fast packet deduplication. The present application also provides an FPGA-based message deduplication device, equipment and a readable storage medium, which have the above beneficial effects.

Description

technical field [0001] The present application relates to the technical field of image processing, in particular to an FPGA-based message deduplication method, device and an FPGA chip. Background technique [0002] In the field of network analysis and monitoring, the role of network visualization collection and distribution equipment is becoming more and more important. In the network visualization collection, the flow splitter collects the flow data of multiple links, aggregates and splits the output, and introduces the data flow in the link to the back-end analysis server, which plays a role in the visualization platform and user data link. A "Logistics" role for filtering categories. [0003] Since the collection and distribution equipment usually completes the function of converging and distributing output of multiple links, and multiple links usually correspond to the Ethernet split points of multiple links, sometimes there are a large number of identical Ethernet dat...

AI Technical Summary

Problems solved by technology

[0003] Since the collection and distribution equipment usually completes the function of converging and distributing output of multiple links, and multiple links usually correspond to the Ethernet split points of multiple links, sometimes there are a large number of identical Ethernet data traffic, these repetitive traffic will increase the processing pressure on the background analysis server and increase the complexity of background processing

[0004] In order to solve the problem of link duplicate message filtering, a kind of duplicate message detection method based on CPU platform is provided in the related art, this traditional deduplication method adopts the method of quintuple flow table search, this method deduplication comparison speed Low, and the comparison accuracy is not high, it is difficult to meet the needs of efficient data processing, and in the face of large data traffic, it will generate a large processing burden for the duplicate message detection end

Images