Message duplicate removal processing method and device
By analyzing the layer 2 protocol header information of data packets and generating digest identifiers, the high pressure and complexity problems caused by the marking processing of source equipment in the prior art are solved, and efficient and accurate message deduplication is achieved.
Patent Information
- Application Number
- CN202510652921.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-20
- Publication Date
- 2025-08-12
AI Technical Summary
The existing packet deduplication processing method requires the use of source equipment to mark the original packet, which results in both the source equipment and the receiving system need to adapt to the parsing marking label scheme, which increases the processing pressure and service processing complexity of the packet deduplication equipment.
By analyzing the layer 2 protocol header information of the data packet, determining the message type, and obtaining the digest information based on the message characteristics, using MD5 operations to generate a fixed-length digest identifier, and determining whether the same digest information exists in the message deduplication list to determine whether to discard duplicate messages, avoiding marking processing of the source device.
It reduces the processing pressure and service processing complexity of the message deduplication device, improves the deduplication efficiency, reduces storage and computing overhead, and ensures the accuracy of deduplication, especially the recognition ability of sharded messages.
Smart Images
Figure CN120475087A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of network communication technology, and in particular to a method and device for processing message deduplication. Background Art
[0002] Packet deduplication refers to the removal of duplicate packets during data transmission and processing to improve data processing efficiency and reduce resource usage. Visual analysis of network traffic requires the collection and capture of real-time data packets transmitted across the network. Due to the location of traffic collection, switches may mirror the same traffic flow at different transmission nodes to the network traffic analysis system. To prevent duplicate packets from increasing processing pressure and complexity on back-end analysis tools, network traffic analysis systems must perform packet deduplication.
[0003] Current methods for deduplicating messages work by assigning each message a unique transaction identifier or message ID. Upon receiving a message, the receiving system checks whether the message ID or transaction identifier in the message header already exists in the database. If so, it marks the message as a duplicate and removes the corresponding message.
[0004] The disadvantage of the above solution is that it requires the use of source equipment to label the original message, so both the source device and the receiving system need to adapt to the parsing labeling solution, which increases the processing pressure of the message deduplication equipment and the complexity of business processing. Summary of the Invention
[0005] The present application provides a method and device for deduplicating messages to solve the technical problem that the existing deduplication processing method requires the use of a source device to label the original message, so both the source device and the receiving system need to adapt to the parsing label scheme, which increases the processing pressure of the message deduplication device and the complexity of business processing.
[0006] The first aspect of the present application provides a method for deduplicating packets, comprising:
[0007] Get data packets;
[0008] Parsing the data message to obtain layer 2 protocol header information;
[0009] Determine the data message type according to the layer 2 protocol header information; the data message type includes: layer 2 protocol message and layer 3 protocol message;
[0010] Obtaining the data message characteristics according to the data message type; and obtaining summary information of the data message according to the data message characteristics;
[0011] It is determined whether the same summary information exists in the message deduplication linked table. If so, the data message corresponding to the summary information is discarded; the message deduplication linked table is configured to store the summary information of the data message.
[0012] In some embodiments, after the step of parsing the data message to obtain the layer 2 protocol header information, the following steps are included:
[0013] Determine whether the layer 2 protocol header information includes a network label header; the network label header includes: a vlan label header and an mpls label header;
[0014] If so, the network tag header is stripped from the layer 2 protocol header information.
[0015] In some embodiments, the step of obtaining the data message characteristics according to the data message type; and obtaining summary information of the data message according to the data message characteristics includes:
[0016] If the data message type is the layer 2 protocol message, then obtain the layer 2 protocol message characteristics; the layer 2 protocol message characteristics include: the layer 2 protocol message protocol number and the payload data length and content;
[0017] Performing an MD5 operation on the layer 2 protocol message feature to obtain summary information of the layer 2 protocol message.
[0018] In some embodiments, the step of obtaining the data message characteristics according to the data message type; and obtaining summary information of the data message according to the data message characteristics further includes:
[0019] If the data packet type is the three-layer protocol packet, determine whether the three-layer protocol packet is an IPv6 packet;
[0020] If yes, obtain IPv6 message features; the IPv6 message features include: IPv6 message header information, transport layer header information, and total length of the IPv6 message;
[0021] Perform an MD5 operation on the IPv6 message feature to obtain summary information of the IPv6 message.
[0022] In some embodiments, if the data packet type is the three-layer protocol packet, after the step of determining whether the three-layer protocol packet is an IPv6 packet, the method further includes:
[0023] If not, the three-layer protocol message is an IPv4 message;
[0024] Obtain the IPv4 message features; the IPv4 message features include: IP header ID, five-tuple information, total length, and fragmentation flag value of the IPv4 message; the five-tuple information includes: source IP address, destination IP address, protocol number, source port, and destination port;
[0025] Performing an MD5 operation on the IPv4 message feature to obtain summary information of the IPv4 message;
[0026] The fragment flag bit value is configured to indicate whether the IPv4 message is a fragment message; if the IPv4 message is a fragment message, the bit value of the IPv4 message is 1;
[0027] The bit value of the last fragment message is not 1.
[0028] In some embodiments, after the step of obtaining the data message feature according to the data message type, the following steps are included:
[0029] Performing XOR processing on the data message features.
[0030] In some embodiments, after the step of determining whether the same summary information exists in the message deduplication linked table, the process is further configured as follows:
[0031] If so, the data message corresponding to the summary information is discarded, and the summary information and the number of repetitions of the summary information are recorded.
[0032] In some embodiments, the message deduplication linked table includes: a layer 2 protocol message deduplication linked table and a layer 3 protocol message deduplication linked table;
[0033] The layer 2 protocol message deduplication linked table is configured to store summary information of the layer 2 protocol message; the layer 3 protocol message deduplication linked table is configured to store summary information of the layer 3 protocol message.
[0034] In some embodiments, after the step of determining whether the same summary information exists in the message deduplication linked table, the method further includes:
[0035] If not, the summary information is stored in the message deduplication linked table, and it is determined whether the summary information matches the same summary information within a preset time; if not, the summary information is deleted from the message deduplication linked table.
[0036] A second aspect of the present application provides a message deduplication processing device, comprising:
[0037] at least one processor;
[0038] and at least one memory, which is coupled to the at least one processor and stores a computer program for execution by the at least one processor, and when the computer program is executed by the at least one processor, enables the device to perform a message deduplication processing method as described in any one of the first aspects.
[0039] The present application provides a method and device for processing message deduplication, the method comprising: obtaining a data message; parsing the data message to obtain layer 2 protocol header information; determining the data message type according to the layer 2 protocol header information; the data message type comprising: layer 2 protocol message and layer 3 protocol message; obtaining the data message characteristics according to the data message type; and obtaining summary information of the data message according to the data message characteristics; judging whether the same summary information exists in a message deduplication linked table, and if so, discarding the data message corresponding to the summary information; the message deduplication linked table is configured to store the summary information of the data message, so as to solve the problem that the current message deduplication processing method needs to use the source device to label the original message, so both the source device and the receiving system need to adapt to the parsing labeling scheme, which increases the processing pressure of the message deduplication device and the complexity of business processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0041] Figure 1 This is a flow chart of the method for deduplicating messages in this application;
[0042] Figure 2 A flowchart of the method for obtaining abstract information in this application;
[0043] Figure 3 This is a diagram of the current usage scenario of message deduplication. DETAILED DESCRIPTION
[0044] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.
[0045] like Figure 3The figure shows the current usage scenario of message deduplication.
[0046] For example, the core switch mirrors all traffic to the traffic analysis system, which then analyzes all traffic. As can be seen from the deployment, traffic between departments 1 and 2 accessing hosts and servers, as well as accessing the network, may mirror the same traffic multiple times. This results in many duplicate packets being counted when analyzing the same session on the traffic analysis system. Therefore, deduplication of duplicate packets within the same session is necessary.
[0047] Ethernet messages are generated by source device software, encapsulated using the application layer protocol according to the corresponding software program, and then repacked into data frames by the network card. These frames are then transmitted to the designated destination via the network transmission medium. Ethernet messages are encapsulated according to the TCP / IP protocol model, which, from bottom to top, comprises the link layer, network layer, transport layer, and application layer.
[0048] For example, the current application scenarios of message deduplication are as follows:
[0049] 1. Network traffic analysis:
[0050] Visualizing network traffic analysis requires collecting and capturing data packets transmitted in real time. Due to the location of traffic collection, switches may mirror the same traffic flow at different transmission nodes to the network traffic analysis system. To prevent duplicate packets from increasing processing pressure and complexity on back-end analysis tools, the network traffic analysis system must perform packet deduplication.
[0051] 2. Electronic Data Interchange (EDI) System:
[0052] In EDI systems, message deduplication is required to prevent duplicate processing of messages and ensure data accuracy and consistency.
[0053] 3. Data storage and backup:
[0054] During the data storage and backup process, duplicate messages may appear. Deduplication can reduce storage space usage and improve storage efficiency.
[0055] For example, the current method for deduplicating packets has the following drawbacks:
[0056] 1. Message deduplication based on unique identifier:
[0057] Each message is assigned a unique transaction identifier or message ID. After receiving the message, the receiving system checks whether the message ID or transaction identifier in the message header already exists in the database. If so, it is marked as a duplicate message. The disadvantage of this solution is that it requires the source device to tag the original message. Therefore, both the source device and the receiving system need to be adapted to parse the tagging scheme. It is usually impossible to deduplicate messages on a single device, and it increases the processing pressure on the message deduplication equipment and the complexity of business processing.
[0058] 2. Message deduplication based on content hash:
[0059] The message content is hashed to generate a unique hash value. Duplicate messages are determined by comparing hash values. To improve comparison efficiency, a hash summary of the payload is typically created. However, hashing can result in a small probability of collision, meaning different payloads may have the same hash value. Therefore, information from the message header must be combined to avoid false deduplication. This solution also creates a hash summary for the entire payload, which incurs significant system overhead and can result in false hits when calculating and matching fragmented messages.
[0060] 3. Message deduplication based on sliding window cache:
[0061] A certain number of data packets are cached in memory (a sliding window). New incoming data packets are compared with the cached packets. If they match a data packet in the sliding window, they are considered duplicates and discarded. If they do not match, they are considered non-duplicates and placed in the sliding window, removing the oldest packet. This solution places high demands on device performance and can put significant pressure on system resources when the cached data volume is large.
[0062] Since in some technologies, the message deduplication processing method requires the source device to tag the original message, both the source device and the receiving system need to adapt and parse the tagging scheme, which increases the processing pressure of the message deduplication device and the complexity of business processing. In order to solve this technical problem, the present application provides a message deduplication processing method and device. The message deduplication processing method and device are described below:
[0063] like Figure 1 As shown, this is a flow chart of the message deduplication processing method in this application.
[0064] The first aspect of the present application provides a method for deduplicating packets, comprising the following steps:
[0065] S100: Obtain data packets.
[0066] S200: Parse the data message to obtain layer 2 protocol header information.
[0067] Specifically, network traffic is captured using a network packet capture tool (such as Wireshark or tcpdump) to obtain data packets. The packet capture tool displays the complete hierarchical structure of the data packet, including the Layer 2 (data link layer) protocol header. The packet capture tool locates the Layer 2 protocol header and extracts the header fields to obtain the Layer 2 protocol header information. The Layer 2 protocol header information includes the destination MAC address, source MAC address, protocol type and length, etc.
[0068] After the step of parsing the data message to obtain the layer 2 protocol header information, the following steps are included:
[0069] S210: Determine whether the layer 2 protocol header information includes a network label header; the network label header includes: a vlan label header and an mpls label header.
[0070] S220: If yes, strip the network label header from the layer 2 protocol header information.
[0071] Specifically, stripping the label header reduces differences in message characteristics and improves deduplication efficiency. Label headers increase message length: VLAN and MPLS label headers increase message length and complexity, causing messages with identical content to be misidentified as different due to different labels. After stripping the label header, only the original content is retained. Messages with identical content have consistent characteristics, making it easier for deduplication algorithms to identify and filter duplicate messages.
[0072] Specifically, stripping the tag header reduces storage and computational overhead. Deduplication systems typically need to store the message's fingerprint, and stripping the tag header reduces the fingerprint length, lowering storage requirements. Furthermore, stripping the tag header simplifies the message content, reducing feature extraction and hash calculation time, and improving deduplication efficiency.
[0073] Specifically, by stripping the tag header, we avoid deduplication failures caused by dynamic label changes. During network transmission, VLAN or MPLS labels may change due to routing changes or policy adjustments, resulting in packets with identical content being unable to be deduplicated due to different labels. After stripping the tag header, the message characteristics are solely related to the content and are not affected by dynamic label changes, ensuring effective deduplication.
[0074] S300: Determine the data message type based on the second-layer protocol header information; the data message types include: second-layer protocol messages and third-layer protocol messages; second-layer protocol messages (data link layer), the data link layer is responsible for encapsulating the original bit stream into logical frames and realizing reliable transmission between devices in the same local area network; third-layer protocol messages (network layer), the network layer is responsible for routing data packets from the source host to the target host across different networks.
[0075] like Figure 2 As shown, based on Figure 1 Flowchart of the steps for obtaining summary information.
[0076] S400: Acquire the data message characteristics according to the data message type; and acquire summary information of the data message according to the data message characteristics; the summary information is used to quickly identify or analyze the data message content.
[0077] For example, the summary information is a fixed-length string (or binary data) generated by performing an irreversible mathematical transformation on the original data through a hash algorithm (such as MD5, SHA-1, SHA-256, etc.). Its core features include: Fixed length: Regardless of the size of the input data, the summary length remains unchanged (such as MD5 outputs 128 bits / 16 bytes). Uniqueness: In theory, the summaries generated by different data should be different. Irreversibility: The original data cannot be deduced from the summary. In summary, whether the data packets are the same can be determined based on the summary information, which has a certain degree of reliability.
[0078] After the step of obtaining the data message feature according to the data message type, the following steps are included:
[0079] S401: Perform XOR processing on the data message features.
[0080] Specifically, features for XOR processing are selected in the data message; the features are preprocessed, features of different lengths are padded or truncated to a uniform length (such as 64 bits), and the hexadecimal or decimal addresses are converted to binary; the binary values of all features are bitwise XORed to generate a fixed-length summary; the final XOR result, that is, the fixed-length summary, is converted into a hexadecimal or decimal representation as the unique identifier of the message.
[0081] The step of obtaining the data message characteristics according to the data message type; and obtaining summary information of the data message according to the data message characteristics, comprises the following steps:
[0082] S410: If the data message type is the layer 2 protocol message, obtain the layer 2 protocol message characteristics; the layer 2 protocol message characteristics include: the layer 2 protocol message protocol number and the payload data length and content; when the payload data is a layer 2 protocol message, obtain the data content of the layer 2 protocol carried after the MAC header, including the layer 2 protocol header information and data.
[0083] S420: Perform MD5 operation on the layer 2 protocol message feature to obtain summary information of the layer 2 protocol message. MD5 is a widely used hash function that can convert input of any length into summary information of a fixed length (128 bits / 16 bytes).
[0084] The MD5 digest uniquely identifies the Layer 2 protocol message for quick comparison or deduplication. If the Layer 2 feature MD5 digests of two messages are the same, they can be determined to be the same message.
[0085] The step of obtaining the data message characteristics according to the data message type; and obtaining the summary information of the data message according to the data message characteristics, further includes the following steps:
[0086] S430: If the data message type is the layer-3 protocol message, determine whether the layer-3 protocol message is an IPv6 message.
[0087] If the data message type is the three-layer protocol message, after the step of determining whether the three-layer protocol message is an IPv6 message, the following steps are included:
[0088] S431: If not, the three-layer protocol message is an IPv4 message; IPv4 messages are longer and may be fragmented, so the fragmentation of IPv4 messages needs to be considered to avoid the system identifying fragmented messages as duplicate messages for deduplication processing.
[0089] S432: Obtain the IPv4 message features; the IPv4 message features include: the IP header ID, five-tuple information, total length, and fragmentation flag value of the IPv4 message; the five-tuple information includes: source IP address, destination IP address, protocol number, source port, and destination port.
[0090] The IP ID field in an IP message is used to identify an IP packet. IP packets sent by the same host have different IDs, so it can be used to identify a specific packet within a session. Secondly, if a message is fragmented during forwarding due to link MTU limitations, all fragments of the same message have the same IP ID. However, Layer 2 protocol messages and IPv6 messages are smaller in length and therefore do not require fragmentation. Therefore, this field value is missing from these messages.
[0091] The total length of an IP packet includes the IP header and data. Typically, for one or more sessions generated by an application, the total length of an IP packet is fixed due to the data content or transmission mechanism. To a certain extent, this field, combined with information such as the IP ID, can identify a data packet. However, this field does not exist for Layer 2 protocol packets or IPv6 packets; instead, the IPv6 header contains the payload length.
[0092] The fragment flag value is configured to indicate whether the IPv4 message is a fragmented message; if the IPv4 message is a fragmented message, the bit value of the IPv4 message is 1; the flag value of the IP data packet sent by the sender is a unique value, which is copied to each fragment during fragmentation. The bit value of the fragmented message located in the last fragment is not 1. By adding the characteristics of the fragment flag value, the system can distinguish whether the message is a fragmented message, thereby avoiding the system identifying the fragmented message as a duplicate message and performing deduplication processing.
[0093] S433: Perform MD5 operation on the IPv4 message feature to obtain summary information of the IPv4 message; wherein, the MD5 summary uniquely identifies the layer 2 protocol message for fast comparison or deduplication; if the layer 2 feature MD5 summaries of two messages are the same, they can be determined to be the same message.
[0094] S440: If yes, obtain IPv6 message features; the IPv6 message features include: IPv6 message header information, transport layer header information, and total length of the IPv6 message.
[0095] S450: Performing an MD5 operation on the IPv6 message features to obtain summary information of the IPv6 message. The MD5 summary uniquely identifies the Layer 2 protocol message for rapid comparison or deduplication. If the Layer 2 MD5 summary of two messages is the same, they are determined to be the same message.
[0096] S500: Determine whether the same summary information exists in the message deduplication linked table. If so, discard the data message corresponding to the summary information. The message deduplication linked table is configured to store the summary information of the data message.
[0097] After the step of determining whether the same summary information exists in the message deduplication linked table, the process is further configured as follows:
[0098] If so, the data message corresponding to the summary information is discarded, and the summary information and the number of repetitions of the summary information are recorded.
[0099] For example, by recording the summary information of duplicate data packets and the number of repetitions of the summary information, subsequent analysis and processing of duplicate data packets (such as calculating packet loss and retransmissions) can be avoided, and the user can be informed of the deployment location of the duplicate data packets. Duplicate packets are often caused by improper deployment, resulting in different nodes mirroring the same traffic. By recording the summary information of duplicate data packets and the number of repetitions of the summary information, improper packet deployment can be avoided, thereby improving network resource utilization.
[0100] The message deduplication linked table includes: a Layer 2 protocol message deduplication linked table and a Layer 3 protocol message deduplication linked table; the Layer 2 protocol message deduplication linked table is configured to store summary information of Layer 2 protocol messages; the Layer 3 protocol message deduplication linked table is configured to store summary information of Layer 3 protocol messages. After the system is started, a Layer 2 protocol message deduplication HASH linked table and a Layer 3 protocol message deduplication HASH linked table are established respectively, and the summary information of the Layer 2 protocol messages and the summary information of the Layer 3 protocol messages are stored in real time.
[0101] After the step of determining whether the same summary information exists in the message deduplication linked table, the following steps are included:
[0102] S510: If not, the summary information is stored in the message deduplication linked table, and it is determined whether the summary information matches the same summary information within a preset time. If not, the summary information is deleted from the message deduplication linked table. The data in the message deduplication linked table is aged regularly. For example, the aging time of the data added to the message deduplication linked table is 100ms. If a match is hit before aging, the corresponding summary information will be restarted to enter the aging timer; if the data aging time exceeds the data aging time, the corresponding summary information will be deleted.
[0103] For example, by setting a data aging time, we can dynamically manage the deduplication list resources, preventing the list from growing indefinitely due to long-term storage of useless summaries and excessive memory usage. Only recently active traffic summaries are retained to ensure deduplication efficiency. This reduces the system's false positive rate. If a summary does not reappear within the aging time, it is considered sporadic traffic and does not need to be retained long-term, reducing the possibility of mismatches in subsequent messages.
[0104] The present application adopts a message deduplication processing method, which confirms whether it is a duplicate message through a certain judgment mechanism through relevant fields such as IP ID identification, IP fragment identification value, five-tuple information and total length of IP message, payload, etc., without the help of source equipment to mark the original message, thereby reducing the processing pressure of the message deduplication device and the complexity of business processing; and duplicate messages are not included in the calculation of related session statistics, and the number of duplicate messages in the session is calculated at the same time; message deduplication logic processing is performed on both Layer 2 protocol messages and Layer 3 protocol messages; in the message deduplication logic processing, by reducing the original data for calculating the message feature value, the consumption of system resources is reduced, which helps to improve the performance of message deduplication; compared with the current message deduplication processing method, the present application takes into account the scenario of fragmented message transmission during implementation, which can avoid fragmented messages being identified as duplicate messages during statistical processing.
[0105] A second aspect of the present application provides a message deduplication processing device, comprising:
[0106] at least one processor;
[0107] and at least one memory, which is coupled to the at least one processor and stores a computer program for execution by the at least one processor, and when the computer program is executed by the at least one processor, enables the device to perform a message deduplication processing method described in any of the above embodiments.
[0108] Exemplarily, the computer program can be divided into one or more units, which are stored in the memory and executed by the memory to complete the message deduplication processing method provided by the present application. The one or more units can be a series of computer program instruction segments capable of completing specific functions, and the computer program instruction segments are used to describe the execution process of the computer program in the message deduplication device.
[0109] The above specific implementation methods further explain in detail the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementation methods of the embodiments of the present application and are not intended to limit the scope of protection of the embodiments of the present application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the scope of protection of the embodiments of the present application.
Claims
1. A method for deduplicating messages, characterized in that: include: Get data packets; Parsing the data message to obtain layer 2 protocol header information; Determining the data message type according to the layer 2 protocol header information; The data message types include: layer 2 protocol messages and layer 3 protocol messages; Obtaining the data message characteristics according to the data message type; and obtaining summary information of the data message according to the data message characteristics; It is determined whether the same summary information exists in the message deduplication linked table. If so, the data message corresponding to the summary information is discarded; the message deduplication linked table is configured to store the summary information of the data message.
2. A message deduplication processing method according to claim 1, characterized in that: After the step of parsing the data message to obtain the layer 2 protocol header information, the method includes: Determine whether the layer 2 protocol header information includes a network label header; the network label header includes: a vlan label header and an mpls label header; If so, the network tag header is stripped from the layer 2 protocol header information.
3. A method for deduplicating messages according to claim 1, characterized in that: acquiring the data message characteristics according to the data message type; The step of obtaining summary information of the data message according to the data message characteristics includes: If the data message type is the layer 2 protocol message, then obtain the layer 2 protocol message characteristics; the layer 2 protocol message characteristics include: the layer 2 protocol message protocol number and the payload data length and content; Performing an MD5 operation on the layer 2 protocol message feature to obtain summary information of the layer 2 protocol message.
4. A message deduplication processing method according to claim 1, characterized in that: acquiring the data message characteristics according to the data message type; The step of obtaining summary information of the data message according to the data message characteristics further includes: If the data packet type is the three-layer protocol packet, determine whether the three-layer protocol packet is an IPv6 packet; If yes, obtain IPv6 message features; the IPv6 message features include: IPv6 message header information, transport layer header information, and total length of the IPv6 message; Perform an MD5 operation on the IPv6 message feature to obtain summary information of the IPv6 message.
5. A message deduplication processing method according to claim 4, characterized in that: If the data message type is the three-layer protocol message, after the step of determining whether the three-layer protocol message is an IPv6 message, the method further includes: If not, the three-layer protocol message is an IPv4 message; Obtain the IPv4 message features; the IPv4 message features include: IP header ID, five-tuple information, total length, and fragmentation flag value of the IPv4 message; the five-tuple information includes: source IP address, destination IP address, protocol number, source port, and destination port; Performing an MD5 operation on the IPv4 message feature to obtain summary information of the IPv4 message; The fragment flag bit value is configured to indicate whether the IPv4 message is a fragment message; if the IPv4 message is a fragment message, the bit value of the IPv4 message is 1; The bit value of the last fragment message is not 1.
6. A method for deduplicating packets according to claim 1, characterized in that: After the step of obtaining the data message feature according to the data message type, the method further includes: Performing XOR processing on the data message features.
7. A message deduplication processing method according to claim 1, characterized in that: After the step of determining whether the same summary information exists in the message deduplication linked table, the method is further configured as follows: If so, the data message corresponding to the summary information is discarded, and the summary information and the number of repetitions of the summary information are recorded.
8. The method for deduplicating packets according to claim 1, wherein: The message deduplication chain table includes: a layer 2 protocol message deduplication chain table and a layer 3 protocol message deduplication chain table; The layer 2 protocol message deduplication linked table is configured to store summary information of the layer 2 protocol message; the layer 3 protocol message deduplication linked table is configured to store summary information of the layer 3 protocol message.
9. The method for deduplicating packets according to claim 1, wherein: After the step of determining whether the same summary information exists in the message deduplication linked table, the following steps are included: If not, the summary information is stored in the message deduplication linked table, and it is determined whether the summary information matches the same summary information within a preset time; if not, the summary information is deleted from the message deduplication linked table.
10. A message deduplication processing device, characterized in that: include: at least one processor; and at least one memory, which is coupled to the at least one processor and stores a computer program for execution by the at least one processor, and when the computer program is executed by the at least one processor, enables the device to perform a message deduplication processing method as described in any one of claims 1 to 9.