Method and device for judging message repetition and electronic equipment

Abstract

The invention relates to a message duplication judgment method and device and electronic equipment, and belongs to the technical field of network, and the method comprises the following steps: storing a source IP of a received first message into a first RBM table, storing a destination IP of the received first message into a second RBM table, calculating a first KEY value of transmission data of the received first message, and storing the first KEY value into a third RBM table; when a second message is received, calculating an obtained temporary KEY value based on transmission data of the second message; and when the source IP of the second message exists in the first RBM table, the destination IP of the second message exists in the second RBM table, and the temporary KEY value exists in the third RBM table, determining that the second message is a repeated message. According to the invention, the full-flow equipment can count and accurately calculate various index data in the actual flow, and the real network condition is reflected.

Description

Technical Field

[0001] The present invention relates to the field of network technology, and in particular to a method, device and electronic device for determining message duplication. Background Art

[0002] A full-flow device is a device deployed in a bypass mode in the network topology of a computer room. It is generally used to receive mirrored or split traffic from firewalls, switches, splitters and other devices. There is more or less repetitive traffic in these mirrored or split traffic.

[0003] The full-flow device outputs relevant logs based on the session quintuple. After these repetitive flows reach the full-flow device, if they are not distinguished, the following situation will occur: Flow 1 and Flow 2 are exactly the same. After the flow 1 data reaches the full-flow device, before the session data is completely ended (the FIN message or RST message is received), the flow 2 data also reaches the full-flow device. At this time, the full-flow device will continue to count the quintuple data of flow 2 into the session data corresponding to flow 1, causing data confusion and calculation errors of performance indicators related to the output session data (retransmission, delay, etc.), which will affect the system users' network troubleshooting on the full-flow device.

[0004] In summary, the prior art lacks a method for repeated judgment of traffic based on full-flow equipment, so that the full-flow equipment can count and accurately calculate various indicator data in the actual traffic to reflect the real network situation. Summary of the invention

[0005] In view of this, it is necessary to provide a method, device and electronic device for determining message duplication, so as to enable full-flow equipment to count and calculate various accurate indicator data in actual flow and reflect the actual network situation.

[0006] In order to achieve the above object, the present invention provides a method for determining message duplication, comprising: The source IP address of the received first message is stored in the first RBM table, the destination IP address of the received first message is stored in the second RBM table, a first KEY value of the transmission data of the received first message is calculated, and the first KEY value is stored in the third RBM table; When receiving the second message, a temporary KEY value calculated based on the transmission data of the second message; When the source IP of the second message exists in the first RBM table, the destination IP of the second message exists in the second RBM table, and the temporary KEY value exists in the third RBM table, it is determined that the second message is a duplicate message.

[0007] In a possible implementation, transmitting data includes: Link ID, source MAC address, destination MAC address, data packet IP layer identifier, maximum number of hops the data packet has passed, virtual LAN identifier, data packet checksum, data packet payload, data packet payload length, source port, and destination port.

[0008] In a possible implementation manner, the temporary KEY value calculated based on the transmission data of the second message includes: The temporary KEY value is obtained by calculating the transmission data of the second message based on the hash algorithm.

[0009] In a possible implementation manner, the temporary KEY value obtained by calculating the transmission data of the second message based on the hash algorithm includes: Calculate a first KEY value of the link ID based on a hash algorithm; Calculate a second KEY value of the source MAC address based on a hash algorithm; Calculate a third KEY value of the destination MAC address based on a hash algorithm; Calculate the fourth KEY value of the IP layer identifier of the data packet based on the hash algorithm; Calculate a fifth KEY value of the maximum number of hops that the data packet has passed through based on a hash algorithm; Calculate a sixth KEY value of the virtual local area network identifier based on a hash algorithm; Calculate a seventh KEY value of the data packet checksum based on a hash algorithm; Calculate an eighth KEY value of the data packet payload based on a hash algorithm; Calculate a ninth KEY value of the data packet payload length based on a hash algorithm; Calculate the tenth KEY value of the source port based on a hash algorithm; Calculate the eleventh KEY value of the destination port based on a hash algorithm; A temporary KEY value is obtained based on the first KEY value, the second KEY value, the third KEY value, the fourth KEY value, the fifth KEY value, the sixth KEY value, the seventh KEY value, the eighth KEY value, the ninth KEY value, the tenth KEY value and the eleventh KEY value.

[0010] In a possible implementation manner, calculating the eighth KEY value of the data packet payload based on a hash algorithm includes: Dividing the data packet payload into a plurality of original data according to the length of the data packet payload; The eighth KEY value is obtained by performing a mixed operation on the length of the plurality of original data and the length of the data packet payload.

[0011] In a possible implementation, the method further includes: When it is determined that the second message is not a duplicate message, the source IP of the second message is stored in the first RBM table, the destination IP of the second message is stored in the second RBM table, the second KEY value of the transmission data of the second message is calculated, and the second KEY value is stored in the third RBM table.

[0012] In a possible implementation manner, the first RBM table, the second RBM table, and the third RBM table are respectively in different threads.

[0013] In a possible implementation, the first RBM table includes a primary RBM table and a backup RBM table; When the life cycle of the main RBM table exceeds a first preset value, storing the received message in the main RBM table and the standby RBM table; When the life cycle of the main RBM table exceeds a second preset value, the main RBM table is deleted and the backup RBM table is used as a temporary main RBM table, and a temporary backup RBM table is created at the same time.

[0014] On the other hand, the present invention also provides a device for determining message duplication, comprising: A message data storage module, used for storing the source IP of the received first message into the first RBM table, storing the destination IP of the received first message into the second RBM table, calculating the first KEY value of the transmission data of the received first message, and storing the first KEY value into the third RBM table; A temporary KEY value acquisition module, used for calculating a temporary KEY value based on the transmission data of the second message when receiving the second message; The message duplication determination module is used to determine that the second message is a duplicate message when the source IP of the second message exists in the first RBM table, the destination IP of the second message exists in the second RBM table, and the temporary KEY value exists in the third RBM table.

[0015] On the other hand, the present invention also provides an electronic device, including a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in a method for determining message duplication described in any one of the above implementations.

[0016] The beneficial effect of the present invention is as follows: the present invention provides a method for determining message duplication, firstly storing the source IP of the first message in the first RBM table, storing the destination IP of the received message in the second RBM table, calculating the first KEY value of the transmission data of the first message, storing the first KEY value in the third RBM table, classifying the first message according to the source address, the destination address and the transmission data of the message to establish an RBM table, when the second message is received, the temporary KEY value calculated based on the transmission data of the second message, when the source IP of the second message exists in the first RBM table, and the destination IP of the second message is in the second RBM table, and the temporary KEY value exists in the third RBM table, the second message is determined to be a duplicate message. The present invention classifies the first message according to the source address, the destination address and the transmission data of the first message to establish an RBM table, and then matches the second message with the three RBM tables to determine the duplicate message, so that the full flow device can count and accurately calculate the various indicator data in the actual flow, reflecting the real network situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A structural method flow chart of an embodiment of a method for determining message duplication provided by the present invention; Figure 2 A schematic diagram of the RBM life cycle in an embodiment of a method for determining message duplication provided by the present invention; Figure 3 A flowchart of a message entering a data packet processing function in an embodiment of a method for determining message duplication provided by the present invention; Figure 4 A schematic flow chart of an embodiment of a device for determining message duplication provided by the present invention; Figure 5 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0018] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0019] Before presenting the embodiments, the following terms are explained.

[0020] RoaringBitmap is an efficient compressed bitmap, referred to as RBM. RoaringBitmap is a data structure used to store large sets of 32-bit integers. It can efficiently represent bitmaps and support fast operations such as intersection, union, and difference. The main advantage of RoaringBitmap is that it can compress data to a certain extent, and modifications to the set are immutable, which makes it superior to traditional set representation methods in terms of memory and processing speed.

[0021] The core principle of RoaringBitmap is efficient bit array storage, which uses high-bits-are-sorted, bitmap and interval compression techniques to reduce storage space and increase processing speed.

[0022] The present invention provides a method, device, electronic device and medium for determining message duplication, which are described below respectively.

[0023] Figure 1 A flow chart of an embodiment of a method for determining message duplication provided by the present invention is shown as follows: Figure 1 As shown, the method for determining message duplication includes: S101, storing the source IP of the received first message into the first RBM table, storing the destination IP of the received first message into the second RBM table, calculating the first KEY value of the transmission data of the received first message, and storing the first KEY value into the third RBM table; S102, when receiving the second message, calculating a temporary KEY value based on the transmission data of the second message; S103. When the source IP of the second message exists in the first RBM table, the destination IP of the second message exists in the second RBM table, and the temporary KEY value exists in the third RBM table, it is determined that the second message is a duplicate message.

[0024] Compared with the prior art, the present embodiment provides a method for determining message duplication, firstly storing the source IP of the first message in the first RBM table, storing the destination IP of the received message in the second RBM table, calculating the first KEY value of the transmission data of the first message, storing the first KEY value in the third RBM table, classifying the first message according to the source address, the destination address and the transmission data of the message to establish an RBM table, when the second message is received, the temporary KEY value calculated based on the transmission data of the second message, when the source IP of the second message exists in the first RBM table, and the destination IP of the second message is in the second RBM table, and the temporary KEY value exists in the third RBM table, the second message is determined to be a duplicate message. The present invention classifies the first message according to the source address, the destination address and the transmission data of the first message to establish an RBM table, and then matches the second message with the three RBM tables to determine the duplicate message, so that the full flow device can count and accurately calculate the various indicator data in the actual flow, reflecting the real network situation.

[0025] It should be noted that the present invention is applied to full-flow equipment, but it can be understood that it can also be applied to equipment of other flow processing types.

[0026] In a specific embodiment of the present invention, a full-flow device captures data packets of one or more applications, each of which contains many data messages. The data messages in these data packets are judged one by one whether they are repeated. Non-duplicate messages are stored in the RBM, and duplicate messages are not included in the statistical performance indicators.

[0027] In a specific embodiment of the present invention, when it is determined that the second message is not a duplicate message, the source IP of the second message is stored in the first RBM table, the destination IP of the second message is stored in the second RBM table, the second KEY value of the transmission data of the second message is calculated, and the second KEY value is stored in the third RBM table.

[0028] In a specific embodiment of the present invention, when performing message deduplication search, first search in the corresponding RBM, i.e., the first RBM table and the second RBM table, according to the source and destination IP of the message. If both can be found, then a 32-bit KEY value is calculated according to other fields and searched in the other RBM, i.e., the third RBM table. If it can be found, it is determined to be a duplicate message.

[0029] Three RBM tables are used for message deduplication. Three keys—source IP key, destination IP key, and message summary key—are calculated for each message, and each key is searched in the three RBMs. Only when all three keys exist at the same time can it be determined as a duplicate message.

[0030] In some embodiments of the present invention, transmitting data includes: Link ID, source MAC address, destination MAC address, data packet IP layer identifier, maximum number of hops the data packet has passed, virtual LAN identifier, data packet checksum, data packet payload, data packet payload length, source port, and destination port.

[0031] In some embodiments of the present invention, the temporary KEY value calculated based on the transmission data of the second message includes: The temporary KEY value is obtained by calculating the transmission data of the second message based on the hash algorithm.

[0032] In some embodiments of the present invention, the temporary KEY value obtained by calculating the transmission data of the second message based on the hash algorithm includes: Calculate a first KEY value of the link ID based on a hash algorithm; Calculate a second KEY value of the source MAC address based on a hash algorithm; Calculate a third KEY value of the destination MAC address based on a hash algorithm; Calculate the fourth KEY value of the IP layer identifier of the data packet based on the hash algorithm; Calculate a fifth KEY value of the maximum number of hops that the data packet has passed through based on a hash algorithm; Calculate a sixth KEY value of the virtual local area network identifier based on a hash algorithm; Calculate a seventh KEY value of the data packet checksum based on a hash algorithm; Calculate an eighth KEY value of the data packet payload based on a hash algorithm; Calculate a ninth KEY value of the data packet payload length based on a hash algorithm; Calculate the tenth KEY value of the source port based on a hash algorithm; Calculate the eleventh KEY value of the destination port based on a hash algorithm; A temporary KEY value is obtained based on the first KEY value, the second KEY value, the third KEY value, the fourth KEY value, the fifth KEY value, the sixth KEY value, the seventh KEY value, the eighth KEY value, the ninth KEY value, the tenth KEY value and the eleventh KEY value.

[0033] In some embodiments of the present invention, the calculating the eighth KEY value of the data packet payload based on the hash algorithm includes: Dividing the data packet payload into a plurality of original data according to the length of the data packet payload; The eighth KEY value is obtained by performing a mixed operation on the length of the plurality of original data and the length of the data packet payload.

[0034] In the specific embodiment of the present invention, the source ip RBM, i.e., the first RBM table, and the destination ip RBM, i.e., the second RBM table, are used to add and search information. The key can be directly taken from the ip of the message, and no further description is required. The information addition in the other RBM table, i.e., the third RBM table, i.e., the method for filling and searching the key of the summary information of the message is as follows: When certain fields are set to be ignored, these fields will not be included in the calculation of the final key value. Since deduplication is based on the link, the link ID must also be used as a calculation element. By default, when the field to be ignored is not checked, the fields involved in the calculation are link_id (link ID), smac (source mac address), dmac (destination mac address), ipid (data packet IP layer identifier ID), ttl (maximum number of hops for the data packet), vlan (virtual LAN identifier), checksum (data packet checksum), payload (data packet load), payload_len (data packet load length), sport (source port), and dport (destination port).

[0035] The summary information of the message payload is an important distinguishing element of the message. The value is selected by using a simple and effective algorithm as much as possible. First, extract the payload and length, take a part of the original message data in the front, middle and back segments according to the different lengths, and perform mixed operations to obtain a summary value. For example, if the payload length is 768, take the sum of the 0-4 bytes, 768 / 2=384-388 bytes, and the last 4 bytes, and add the payload_len to get a 32-bit summary value.

[0036] The key algorithm uses the jhash algorithm. Practice has proven that the jhash algorithm is an algorithm with high efficiency and good distribution balance of hash results.

[0037] It should be noted that RBM needs to consider the aging mechanism, because message deduplication is time-sensitive, and message information from too long ago cannot be stored in an RBM. Therefore, the RBM table must be eliminated and updated regularly to ensure the effectiveness of deduplication.

[0038] In some embodiments of the present invention, the first RBM table, the second RBM table and the third RBM table are respectively in different threads.

[0039] In some embodiments of the present invention, the first RBM table includes a primary RBM table and a backup RBM table; When the life cycle of the main RBM table exceeds a first preset value, storing the received message in the main RBM table and the standby RBM table; When the life cycle of the main RBM table exceeds a second preset value, the main RBM table is deleted and the backup RBM table is used as a temporary main RBM table, and a temporary backup RBM table is created at the same time.

[0040] In a specific embodiment of the present invention, since deduplication and search of messages are performed in the packet processing flow, operations such as adding, searching, and deleting the RBM table are all performed in the packet processing thread. There are generally multiple packet processing threads. If the RBM table is shared between threads, this involves the operation protection of the RBM table. The simplest method is to lock the operation, but this will greatly affect the efficiency of packet reception and reduce system performance. The RBM table is created and maintained based on the packet processing thread, and each RBM table is exclusively owned by a single thread, so there is no need to lock the RBM operation.

[0041] Each RBM table has a life cycle and cannot exist for a long time, because the cache records only need to record messages within a very short period of time (within 1 second). It is meaningless to judge deduplication after that. Continuously adding elements to the RBM table will eventually fill up the RBM and lose its deduplication function. The RBM aging mechanism needs to be considered.

[0042] Since the table operation of RBM runs in the packet processing thread, it is driven by the message and performs various operations in the message processing flow (within the flow_queue_loop function).

[0043] In the absence of traffic, the packet processing function cannot be called. In this case, the timestamp of the RBM can be checked outside the flow_queue_loop, and the aging operation can be performed if it times out.

[0044] In order to ensure that there are enough RBMs available at all times, so that there is no temporary lack of available data for deduplication after each cleanup, a mechanism for switching between the primary and backup RBMs is designed. The timing of the primary and backup switching is when the primary RBM is cleaned. The details are as follows: During initialization, each thread creates RBM-A and records the current timestamp; When thread 1 receives the first packet, it records the current timestamp and then puts the key value of the message into it; When receiving subsequent packets, the key value is searched in RBM-A. If it cannot be found, it means it is not a duplicate message, and the key is added to RBM-A. When the life cycle of RBM-A exceeds 1 second, RBM-B is created, and the subsequent message keys are added to both A and B. When the life cycle of RBM-A exceeds 2 seconds, RBM-A is deleted. At this time, RBM-B has saved message data for more than 1 second and is used to determine deduplication. At the same time, RBM-C is created, and the message key is stored in both B and C. Similarly, when the life cycle of the active RBM exceeds 2 seconds, or the life cycle of the standby RBM exceeds 1 second, the active RBM is deleted, and then the standby RBM is upgraded to the active one, and a new RBM is created as the standby. This ensures that there are active and standby RBMs at all times, and the active RBM also saves enough information for message deduplication. Ideally, the life cycle of an RBM is as follows: Figure 2 As shown. Except for A, the valid message period of each subsequent RBM is between 1s and 2s; at each moment, there are two RBMs, one active and one standby. The key value of the message is stored in both the active and standby RBMs.

[0045] If the current timestamp of the received packet is 2700ms, the active RBM-B of the thread has stored the previous 1700ms of message data, and the standby RBM-C has stored 700ms of message data.

[0046] The entire message deduplication process is performed in each packet processing thread, after the message enters the data packet processing function, such as Figure 3 As shown in the figure, the specific steps for the message to enter the data packet processing are: (1) Determine whether the active RBM life cycle of the current link has expired and whether the standby RBM has been upgraded; (2) Calculate the message’s KEY1 and KEY2, and search for them in the primary RBM. (3) Determine whether the message is a duplicate and add the current message KEY to the primary and backup RBMs; (4) Duplicate messages are marked and the subsequent process is skipped; non-duplicate messages continue with the subsequent process.

[0047] The full-flow device in the embodiment of the present invention will separately count the repetitive data packets detected in the network, and can intuitively and visually display the number of repetitive data packet flows and their proportions in the traffic, providing relevant data support for system users. In addition, repetitive traffic is not included in the calculation of relevant indicators, which can reduce the error calculation rate of relevant indicators, provide real data for system users to perform network troubleshooting, and make the troubleshooting results more accurate.

[0048] In order to better implement a method for determining message duplication in an embodiment of the present invention, based on a method for determining message duplication, correspondingly, as follows: Figure 4 As shown, an embodiment of the present invention further provides a message deduplication device, a message deduplication device 400 includes: The message data storage module 401 is used to store the source IP of the received first message into the first RBM table, store the destination IP of the received first message into the second RBM table, calculate the first KEY value of the transmission data of the received first message, and store the first KEY value into the third RBM table; A temporary KEY value acquisition module 402 is used to calculate a temporary KEY value based on the transmission data of the second message when receiving the second message; The message duplication determination module 403 is used to determine that the second message is a duplicate message when the source IP of the second message exists in the first RBM table, the destination IP of the second message exists in the second RBM table, and the temporary KEY value exists in the third RBM table.

[0049] The message deduplication device 400 provided in the above embodiment can implement the technical solution described in the above embodiment of a method for determining message duplication. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above embodiment of a method for determining message duplication, which will not be repeated here.

[0050] like Figure 5 As shown, the present invention also provides an electronic device 500. The electronic device 500 includes a processor 501, a memory 502 and a display 503. Figure 5 Only some components of the electronic device 500 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0051] In some embodiments, the processor 501 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 502, such as a method for determining message duplication in the present invention.

[0052] In some embodiments, the processor 501 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, the processor 501 may be local or remote. In some embodiments, the processor 501 may be implemented in a cloud platform. In some embodiments, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination thereof.

[0053] In some embodiments, the memory 502 may be an internal storage unit of the electronic device 500, such as a hard disk or memory of the electronic device 500. In other embodiments, the memory 502 may also be an external storage device of the electronic device 500, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 500.

[0054] Furthermore, the memory 502 may include both an internal storage unit of the electronic device 500 and an external storage device. The memory 502 is used to store application software installed in the electronic device 500 and various data.

[0055] In some embodiments, the display 503 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 503 is used to display information on the electronic device 500 and to display a visual user interface. The components 501-503 of the electronic device 500 communicate with each other via a system bus.

[0056] In one embodiment, when the processor 501 executes a message deduplication program in the memory 502, the following steps may be implemented: The source IP address of the received first message is stored in the first RBM table, the destination IP address of the received first message is stored in the second RBM table, a first KEY value of the transmission data of the received first message is calculated, and the first KEY value is stored in the third RBM table; When receiving the second message, a temporary KEY value calculated based on the transmission data of the second message; When the source IP of the second message exists in the first RBM table, the destination IP of the second message exists in the second RBM table, and the temporary KEY value exists in the third RBM table, it is determined that the second message is a duplicate message.

[0057] It should be understood that: when the processor 501 executes a message deduplication program in the memory 502, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0058] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 500 mentioned, and the electronic device 500 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 500 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0059] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0060] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for determining message duplication, characterized in that: include: The source IP address of the received first message is stored in the first RBM table, the destination IP address of the received first message is stored in the second RBM table, a first KEY value of the transmission data of the received first message is calculated, and the first KEY value is stored in the third RBM table; When receiving the second message, a temporary KEY value calculated based on the transmission data of the second message; When the source IP of the second message exists in the first RBM table, the destination IP of the second message exists in the second RBM table, and the temporary KEY value exists in the third RBM table, it is determined that the second message is a duplicate message.

2. The method for determining message duplication according to claim 1, characterized in that: The transmission data includes: Link ID, source MAC address, destination MAC address, data packet IP layer identifier, maximum number of hops the data packet has passed, virtual LAN identifier, data packet checksum, data packet payload, data packet payload length, source port, and destination port.

3. The method for determining message duplication according to claim 2, wherein: The temporary KEY value calculated based on the transmission data of the second message includes: The temporary KEY value is obtained by calculating the transmission data of the second message based on the hash algorithm.

4. The method for determining message duplication according to claim 3, characterized in that: The temporary KEY value obtained by calculating the transmission data of the second message based on the hash algorithm includes: Calculate a first KEY value of the link ID based on a hash algorithm; Calculate a second KEY value of the source MAC address based on a hash algorithm; Calculate a third KEY value of the destination MAC address based on a hash algorithm; Calculate the fourth KEY value of the IP layer identifier of the data packet based on a hash algorithm; Calculate a fifth KEY value of the maximum number of hops that the data packet has passed through based on a hash algorithm; Calculate a sixth KEY value of the virtual local area network identifier based on a hash algorithm; Calculate a seventh KEY value of the data packet checksum based on a hash algorithm; Calculate an eighth KEY value of the data packet payload based on a hash algorithm; Calculate a ninth KEY value of the data packet payload length based on a hash algorithm; Calculate the tenth KEY value of the source port based on a hash algorithm; Calculate the eleventh KEY value of the destination port based on a hash algorithm; A temporary KEY value is obtained based on the first KEY value, the second KEY value, the third KEY value, the fourth KEY value, the fifth KEY value, the sixth KEY value, the seventh KEY value, the eighth KEY value, the ninth KEY value, the tenth KEY value and the eleventh KEY value.

5. The method for determining message duplication according to claim 4, characterized in that: The calculating the eighth KEY value of the data packet payload based on the hash algorithm includes: Dividing the data packet payload into a plurality of original data according to the length of the data packet payload; The eighth KEY value is obtained by performing a mixed operation on the length of the plurality of original data and the length of the data packet payload.

6. The method for determining message duplication according to claim 1, characterized in that: Also includes: When it is determined that the second message is not a duplicate message, the source IP of the second message is stored in the first RBM table, the destination IP of the second message is stored in the second RBM table, the second KEY value of the transmission data of the second message is calculated, and the second KEY value is stored in the third RBM table.

7. The method for determining message duplication according to claim 1, characterized in that: The first RBM table, the second RBM table and the third RBM table are respectively in different threads.

8. The method for determining message duplication according to claim 7, characterized in that: The first RBM table includes a primary RBM table and a backup RBM table; When the life cycle of the main RBM table exceeds a first preset value, storing the received message in the main RBM table and the standby RBM table; When the life cycle of the main RBM table exceeds a second preset value, the main RBM table is deleted and the backup RBM table is used as a temporary main RBM table, and a temporary backup RBM table is created at the same time.

9. A device for determining message duplication, characterized in that: include: A message data storage module, used for storing the source IP of the received first message into the first RBM table, storing the destination IP of the received first message into the second RBM table, calculating the first KEY value of the transmission data of the received first message, and storing the first KEY value into the third RBM table; A temporary KEY value acquisition module, used for calculating a temporary KEY value based on the transmission data of the second message when receiving the second message; The message duplication determination module is used to determine that the second message is a duplicate message when the source IP of the second message exists in the first RBM table, the destination IP of the second message exists in the second RBM table, and the temporary KEY value exists in the third RBM table.

10. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the method for determining message duplication as described in any one of claims 1 to 8 above.

Images