A method and device for deduplication of traffic statistics

Abstract

The embodiment of the present application provides a method and device for deduplicating traffic statistics, which relates to the field of communication technology. The method for deduplicating traffic statistics includes: obtaining data flow information of a target session; parsing the data flow information to obtain parsed data; judging whether the data flow information is duplicate traffic based on a preset hash table and the parsed data; if not, caching the data flow information; and when the target session ends, storing the cached data flow information on a disk. It can be seen that the implementation of this implementation method can deduplicate traffic, avoid multiple repeated statistics of the same traffic, improve the accuracy of traffic statistics, and thus improve traffic query performance.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a method and device for deduplication of traffic statistics. Background Art

[0002] With the development and application of cloud computing technology, new security threats and challenges have been brought. In response to malicious attacks that consume limited network resources or occupy the system, thereby destroying the system's external services, the industry has proposed a method to detect network data flows to determine network anomalies and attacks. At the same time, it allows network administrators to view the status of the entire network, and with the help of real-time and historical traffic records, it is convenient to sort out business to ensure efficient and reliable operation of the network. Existing traffic statistics methods usually deploy virtualized firewalls on the host machine in an agentless manner, embed traffic probes, redirect virtual machine traffic to the virtual firewall, detect it as safe traffic, and then release it for forwarding. However, in practice, it is found that there is a flow that passes through two or even more flow probes, resulting in the same flow being counted repeatedly. When it is aggregated to the management platform, inaccurate traffic statistics will occur, resulting in reduced traffic query performance. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method and device for deduplicating traffic statistics, which can deduplicate traffic, avoid the same traffic being counted multiple times, improve the accuracy of traffic statistics, and thus improve traffic query performance.

[0004] A first aspect of an embodiment of the present application provides a traffic statistics deduplication method, including:

[0005] Get the data flow information of the target session;

[0006] Parsing the data stream information to obtain parsed data;

[0007] Determining whether the data flow information is repeated traffic according to a preset hash table and the parsed data;

[0008] If not, caching the data stream information;

[0009] When the target session ends, the cached data stream information is stored on disk.

[0010] In the above implementation process, the method can first obtain data flow information, then parse the data flow information to obtain parsed data, and then determine whether the data flow information is repeated traffic based on the preset hash table and parsed data. If the data flow information is not repeated traffic, the data flow information is cached and stored in the database when the target session ends. It can be seen that the method can deduplicate traffic, avoid the same traffic being counted repeatedly, improve the accuracy of traffic statistics, and thus improve traffic query performance.

[0011] Further, judging whether the data flow information is duplicate traffic according to the preset hash table and the parsed data includes:

[0012] Obtaining tuple information in the parsed data;

[0013] Determine whether a target hash record matching the tuple information can be found in a pre-stored hash table;

[0014] If yes, determining that the data flow information is repeated traffic;

[0015] If not, it is determined that the data flow information is not duplicate traffic.

[0016] Furthermore, after caching the data stream information, the method further includes:

[0017] Obtaining the communication address of the virtual firewall that sends the data flow information;

[0018] A new hash record is generated according to the communication address and the parsed data, and the hash table is updated by the new hash record.

[0019] Furthermore, the method further comprises:

[0020] When it is determined that the data flow information is duplicate traffic, obtaining address information of a virtual firewall that sends the data flow information;

[0021] Determine whether the address information of the virtual firewall matches the target hash record;

[0022] If there is a match, obtaining the data packet information of the data flow information;

[0023] The target hash record is updated according to the data packet information.

[0024] Furthermore, the method further comprises:

[0025] If the address information of the virtual firewall does not match the target hash record, obtaining the session flag field in the target hash record;

[0026] Determining a session direction according to the session flag field;

[0027] The session direction and the tuple information are sent to the distributed firewall corresponding to the target session, so that the distributed firewall marks the traffic of the target session according to the session direction and the tuple information, and does not perform traffic statistics on the marked traffic.

[0028] Furthermore, the method further comprises:

[0029] Obtain all data flow information and statistical requirements stored on disk;

[0030] Performing statistical processing on all the data flow information according to the statistical requirements to obtain flow statistical information;

[0031] Output the traffic statistics information.

[0032] A second aspect of an embodiment of the present application provides a traffic statistics deduplication device, the traffic statistics deduplication device comprising:

[0033] An acquisition unit, used for acquiring data flow information of a target session;

[0034] A parsing unit, used for parsing the data stream information to obtain parsed data;

[0035] A judging unit, configured to judge whether the data flow information is duplicate traffic according to a preset hash table and the parsed data;

[0036] A cache unit, configured to cache the data flow information when it is determined that the data flow information is not duplicate traffic;

[0037] The storage unit is used to store the cached data stream information on a disk when the target session ends.

[0038] In the above implementation process, the traffic statistics deduplication device can parse the received data flow information and identify the parsed data through the hash table, so as to determine whether the data flow information is repeated traffic, and cache the data flow information when it is not repeated traffic, and store the data flow information on the disk when the target session ends. It can be seen that the device can deduplicate the traffic, avoid the same traffic being counted repeatedly, improve the accuracy of traffic statistics, and thus improve the traffic query performance.

[0039] Furthermore, the judging unit includes:

[0040] An acquisition subunit, used for acquiring tuple information in the parsed data;

[0041] A judging subunit, used to judge whether a target hash record matching the tuple information can be found in a pre-stored hash table;

[0042] A determination subunit is used to determine that the data flow information is duplicate traffic when it is determined that the target hash record can be queried; and to determine that the data flow information is not duplicate traffic when it is determined that the target hash record cannot be queried.

[0043] A third aspect of an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the traffic statistics deduplication method described in any one of the first aspect of the embodiment of the present application.

[0044] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the traffic statistics deduplication method described in any one of the first aspect of the embodiment of the present application is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flow chart of a traffic statistics deduplication method provided in an embodiment of the present application;

[0047] Figure 2 A flow chart of another method for deduplication of traffic statistics provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of the structure of a traffic statistics deduplication device provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of the structure of another traffic statistics deduplication device provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of a method for deduplication of traffic statistics based on a virtualized distributed firewall provided in an embodiment of the present application;

[0051] Figure 6 A schematic diagram of an implementation scheme of a traffic statistics deduplication method provided in an embodiment of the present application.

[0052] Illustration: VM1-virtual machine 1, VM2-virtual machine 2, VM3-virtual machine 3, VM4-virtual machine 4. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0054] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0055] Example 1

[0056] Please see Figure 1 , Figure 1 A flow chart of a traffic statistics deduplication method is provided for an embodiment of the present application. The traffic statistics deduplication method includes:

[0057] S101. Obtain data flow information of a target session.

[0058] In an embodiment of the present application, the data flow information includes Netflow data or Netflow messages.

[0059] In the embodiment of the present application, the traffic statistics deduplication method is applied to a virtualized distributed firewall in a non-agent mode. In the virtualized distributed firewall, four virtual machines are included, namely, virtual machine 1 (ie, VM1), virtual machine 2 (ie, VM2), virtual machine 3 (ie, VM3), and virtual machine 4 (ie, VM4). The traffic communication between the virtual machines is divided into Figure 5 Two cases are shown:

[0060] (1) Communication between virtual machines on the same server: Figure 5 As shown, when virtual machine 1 (ie, VM1) communicates with virtual machine 2 (ie, VM2), the data packet is first redirected to virtual firewall 1 and then forwarded to virtual machine 2 (ie, VM2).

[0061] (2) Communication between virtual machines on different servers: Figure 5 As shown, when virtual machine 1 (i.e. VM1) communicates with virtual machine 4 (i.e. VM4), the data packet is first redirected to virtual firewall 1 on server 1, and then detected by virtual firewall 1. It is sent by server 1 to a router or switch and other devices. After being forwarded to server 2, the data packet first enters virtual firewall 2 on server 2 and is then forwarded to the destination virtual machine 4 (i.e. VM4); at this time, the same data packet has been counted twice by the traffic probes on virtual firewall 1 and virtual firewall 2.

[0062] like Figure 5 As shown, a Netflow v9 flow probe can be embedded in the virtualized distributed firewall to collect flow information passing through the distributed firewall to obtain data flow information of the target session.

[0063] In the embodiment of the present application, the data stream information includes field information of a preset template. Specifically, the preset template is shown in the following table:

[0064] Table 1

[0065]

[0066] As shown in Table 1, the Netflow v9 flow probe supports an extensible Netflow data output format and uses statistical data output based on preset templates.

[0067] In the embodiment of the present application, the flow probe implemented on the distributed firewall uses the Netflow v9 preset template to associate the data packet with the session state for statistics.

[0068] S102: parse the data stream information to obtain parsed data.

[0069] In the embodiment of the present application, the data flow information can be parsed according to the preset Netflow v9 protocol and the preset template shown in Table 1.

[0070] S103, judging whether the data flow information is repeated traffic according to the preset hash table and the parsed data, if so, ending the process; if not, executing steps S104 to S105.

[0071] In the embodiment of the present application, a hash table (i.e., a hash table) can be generated based on the five-tuple as the key to store the data of each field of the current flow. The data format of each hash record in the hash table is shown in Table 2 below. After receiving the data flow information, the five-tuple of the data flow information is first used as the key to query the hash table. If no record is found, it means that it is not a duplicate flow. Then a new data structure is applied to store the data flow information and the IP of the virtual firewall that sends the data flow information (the field in the data format of the hash table is hostIP), and inserted into the hash table.

[0072] Table 2

[0073] Fields describe srcip The IPv4 address of the source. dstip The IPv4 address of the destination. npkt Packets in a traffic flow. nbyte The total number of Layer 3 bytes in the packets of the information flow. first_time SysUptime when the flow started. last_time SysUptime when the last packet of the flow was received. srcport The TCP / UDP source port number or equivalent. dstport The TCP / UDP destination port number or equivalent. categoryID Application groups in a stream. applicationID Applications in information flow. sessionFlag Information flow direction and end mark proto IP protocol (e.g., 6 = TCP, 17 = UDP) hostIP Record the IP address of the virtual firewall that sends Netflow messages

[0074] If a record is found, it means that it is duplicate traffic. We will further determine whether the IP of the virtual firewall that sent the data flow information is consistent with the hostIP in the queried target hash record. If they are consistent, we will update the number of packets and bytes in the target hash record. If they are inconsistent, it means that the current data flow information has been recorded by the previous distributed firewall. In this case, we will discard the current data flow information and no longer record it.

[0075] S104: Cache the data stream information.

[0076] In the embodiment of the present application, a new data structure may be applied for to store the data stream information.

[0077] S105: When the target session ends, the cached data stream information is stored on disk.

[0078] In an embodiment of the present application, whether the current target session has ended can be determined based on the sessionFlag field in the Netflow data. When it is determined based on the sessionFlag field that the virtual firewall has marked the current target session as ended, the cached data flow information is stored on disk.

[0079] In the embodiment of the present application, when the storage is performed on the disk, it is stored in the target database. Specifically, the target database may be an Elasticsearch database, etc., which is not limited in the embodiment of the present application.

[0080] In the embodiment of the present application, the method is implemented, based on the distributed firewall, and the traffic can be deduplicated and counted without changing the original network topology in a virtualized environment; the server configuration in the virtualized environment can also be used to forward messages inside the server, giving full play to the advantages of the virtualized cloud platform;

[0081] In the embodiment of the present application, the implementation of this method can perform statistics on the same message only once, thereby improving the accuracy of traffic statistical information; at the same time, it greatly reduces the space occupied by storage of traffic statistical information.

[0082] In the embodiment of the present application, the execution subject of the method can be a computing device such as a computer or a server, and no limitation is made in this embodiment.

[0083] In the embodiment of the present application, the execution subject of the method can also be a smart device such as a smart phone, a tablet computer, etc., and there is no limitation on this in the present embodiment.

[0084] It can be seen that the method for deduplicating traffic statistics described in this embodiment can be implemented to obtain data flow information first, and then parse the data flow information to obtain parsed data, and then determine whether the data flow information is duplicate traffic based on the preset hash table and the parsed data. If the data flow information is not duplicate traffic, the data flow information is cached, and the data flow information is stored in the database when the target session ends. It can be seen that by implementing this implementation method, the method can deduplicate traffic, avoid multiple repeated statistics of the same traffic, improve the accuracy of traffic statistics, and thus improve traffic query performance.

[0085] Example 2

[0086] Please see Figure 2 , Figure 2 A flow chart of another method for deduplication of traffic statistics provided in an embodiment of the present application. Figure 2 As shown, the traffic statistics deduplication method includes:

[0087] S201. Obtain data flow information of a target session.

[0088] S202: parse the data stream information to obtain parsed data.

[0089] S203: Obtain tuple information in the parsed data.

[0090] In an embodiment of the present application, tuple information can be obtained from the parsed data. Specifically, the tuple information includes parsed data such as source IP, destination IP, protocol number, source port number, destination port number, packet length, timestamp, application category identifier, application identifier, and session status information, which is not limited to the embodiment of the present application.

[0091] In the embodiment of the present application, the identified application category identifier, application identifier, and session status information can be obtained from the target session (i.e., target session) of the virtualized firewall. When the target session ages or reaches the set firewall statistics time interval, the virtualized firewall will send the firewall data to the centralized management platform of the virtualized distributed firewall.

[0092] S204. Determine whether a target hash record matching the tuple information can be found in the pre-stored hash table. If so, determine that the data flow information is duplicate traffic, and execute steps S208 to S209. If not, determine that the data flow information is not duplicate traffic, and execute steps S205 to S207 and steps S215 to S218.

[0093] In an embodiment of the present application, a method for cleaning traffic statistics data of a traffic probe hostIP is implemented in a traffic collector by combining a quintuple and a hash table.

[0094] S205: Cache the data stream information.

[0095] S206: Obtain the communication address of the virtual firewall that sends the data flow information.

[0096] S207, generating a new hash record according to the communication address and the parsed data, and updating the hash table through the new hash record, and steps S215 to S218.

[0097] S208: Acquire the address information of the virtual firewall that sends the data flow information.

[0098] In the embodiment of the present application, the address information of the virtual firewall may specifically be the IP (Internet Protocol) information of the virtual firewall.

[0099] S209. Determine whether the address information of the virtual firewall matches the target hash record. If yes, execute steps S210 to S211 and S215 to S218; if not, execute steps S212 to S218.

[0100] S210: Obtain data packet information of data flow information.

[0101] In the embodiment of the present application, the data packet information includes the number of packets, the number of bytes, etc., which is not limited to the embodiment of the present application.

[0102] S211. Update the target hash record according to the data packet information, and execute steps S215 to S218.

[0103] S212. Obtain the session flag field in the target hash record.

[0104] In the embodiment of the present application, the session flag field is the sessionFlag field shown in Table 2.

[0105] S213: Determine the session direction according to the session flag field.

[0106] In the embodiment of the present application, the information flow direction can be determined according to the sessionFlag field, thereby determining the session direction (ie, the conversation direction).

[0107] S214: Send the session direction and tuple information to the distributed firewall corresponding to the target session, so that the distributed firewall marks the traffic of the target session according to the session direction and tuple information, and does not perform traffic statistics on the marked traffic.

[0108] In an embodiment of the present application, when a traffic collector finds duplicate traffic statistics, it notifies a corresponding subsequent traffic probe to stop collecting statistics on the flow.

[0109] In an embodiment of the present application, the session direction and tuple information are sent to the distributed firewall corresponding to the target session, and the corresponding distributed firewall is notified to find the corresponding target session according to the session direction and tuple information, and mark the target session. The current virtual firewall will no longer perform traffic statistics on the marked target session and will no longer send the Netflow data of the current data flow information.

[0110] In the embodiment of the present application, when duplicate Netflow data is received, the virtual firewall can be dynamically notified to mark the target session, fundamentally realizing that the counted traffic will not be counted again, so that duplicate Netflow data will not be sent, and the CPU computing resources and network bandwidth of the virtual firewall will be saved. Compared with the method of receiving duplicate Netflow data in the flow collector and then removing duplicates, the performance overhead of the flow probe and the flow collector is greatly reduced.

[0111] In the embodiment of the present application, the method is implemented to achieve the upstream and downstream separation of traffic data by combining Netflow data with the session status in the virtual firewall session, and also achieves the separation of real-time traffic information and historical traffic data, which can be applied to different business scenarios.

[0112] S215: When the target session ends, the cached data stream information is stored on disk.

[0113] In an embodiment of the present application, whether the current target session has ended can be determined based on the sessionFlag field in the Netflow data. When it is determined based on the sessionFlag field that the virtual firewall has marked the current target session as ended, the cached data flow information is stored on disk.

[0114] In the embodiment of the present application, when the storage is performed on the disk, it is stored in the target database. Specifically, the target database may be an Elasticsearch database, etc., which is not limited in the embodiment of the present application.

[0115] S216: Obtain all data flow information and statistical requirements stored on disk.

[0116] In the embodiment of the present application, on-demand query of different business scenarios is achieved by caching real-time traffic statistics in the traffic collector and storing historical traffic statistics on disk.

[0117] S217: Perform statistical processing on all data flow information according to statistical requirements to obtain flow statistical information.

[0118] S218: Output traffic statistics information.

[0119] In an embodiment of the present application, when the management platform wants to query the rankings of various dimensions such as traffic IP, port, protocol, trend charts, and generate traffic topology, the method can obtain all traffic information by querying the real-time traffic cache and historical traffic stored in Elasticsearch according to the business scenario.

[0120] Please also read Figure 6 , Figure 6 Schematic diagram of a traffic statistics deduplication method provided in an embodiment of the present application. Figure 6 As shown, it can be parsed through the Netflow v9 protocol and the preset template. It is also possible to generate a hash table with the five-tuple as the key to store the data of each field of the current flow. After receiving the data flow information, first query the hash table based on the five-tuple as the key. If no record is found, a new data structure is applied to store the data flow information and the IP of the virtual firewall that sends the data flow information (the field in the data structure is hostIP), and insert it into the hash table. If a record is found, it is determined whether the IP of the virtual firewall that sent the data flow information is consistent with the recorded hostIP. If they are consistent, the number of packets and bytes on the hash table node are updated. If they are inconsistent, it means that the current data packet information has been recorded by the previous distributed firewall, and the current data flow information is discarded and no longer recorded.

[0121] It can be seen that the implementation of the traffic statistics deduplication method described in this embodiment can solve the problem of repeated statistics of traffic between virtual machines in the scenario of distributed firewalls in a virtualized environment. Specifically, the implementation of this implementation method can perform traffic statistics deduplication based on distributed firewalls, so that traffic can be counted without changing the original network topology in a virtualized environment; at the same time, the server configuration in the virtualized environment can be used to forward messages inside the server, giving full play to the advantages of the virtualized cloud platform; in addition, the method can also be combined with the session mechanism on the virtual firewall through Netflow data in a virtualized environment, and then according to the data flow direction on the session, when receiving duplicate Netflow data, the virtual firewall can be dynamically notified to mark the session, so that the method can fundamentally achieve the effect of no repeated statistics of the traffic that has been counted, thereby achieving the effect of not sending duplicate Netflow data. On this basis, the method can also save the CPU computing resources and network bandwidth of the virtual firewall. Compared with the method of receiving duplicate Netflow data in the traffic collector and then deduplicating, this can greatly reduce the performance overhead of the traffic probe and the traffic collector.

[0122] In addition, this method only counts the same message once, which can improve the accuracy of traffic statistics information; and greatly reduce the space occupied by traffic statistics information storage; at the same time, the use of Netflow data combined with the session status in the virtual firewall session can also achieve upstream and downstream separation of traffic data, thereby achieving separation of real-time traffic information and historical traffic data, making this method applicable to different business scenarios.

[0123] Example 3

[0124] Please see Figure 3 , Figure 3 A schematic diagram of the structure of a traffic statistics deduplication device provided in an embodiment of the present application. Figure 3 As shown, the traffic statistics deduplication device includes:

[0125] An acquisition unit 310 is used to acquire data flow information of a target session;

[0126] The parsing unit 320 is used to parse the data stream information to obtain parsed data;

[0127] A determination unit 330, configured to determine whether the data flow information is duplicate traffic according to a preset hash table and parsed data;

[0128] The cache unit 340 is used to cache the data flow information when it is determined that the data flow information is not duplicate traffic;

[0129] The storage unit 350 is used to store the cached data stream information on a disk when the target session ends.

[0130] In the embodiments of the present application, the explanation of the traffic statistics deduplication device can refer to the description in Embodiment 1 or Embodiment 2, and will not be further elaborated in this embodiment.

[0131] It can be seen that the traffic statistics deduplication device described in this embodiment can parse the received data flow information and identify the parsed data through the hash table, thereby determining whether the data flow information is duplicate traffic, and caching the data flow information when it is not duplicate traffic, and storing the data flow information on the disk when the target session ends. It can be seen that the device can deduplicate traffic, avoid the same traffic being counted repeatedly, improve the accuracy of traffic statistics, and thus improve traffic query performance.

[0132] Example 4

[0133] Please also read Figure 4 , Figure 4: is a schematic diagram of the structure of another traffic statistics deduplication device provided in an embodiment of the present application. Figure 4 The traffic statistics deduplication device shown is composed of Figure 3 The traffic statistics deduplication device shown is optimized. Figure 4 As shown, the judging unit 330 includes:

[0134] The acquisition subunit 331 is used to acquire tuple information in the parsed data;

[0135] A judging subunit 332 is used to judge whether a target hash record matching the tuple information can be found in a pre-stored hash table;

[0136] The determination subunit 333 is used to determine that the data flow information is duplicate traffic when it is determined that the target hash record can be queried; and to determine that the data flow information is not duplicate traffic when it is determined that the target hash record cannot be queried.

[0137] As an optional implementation manner, the traffic statistics deduplication device further includes:

[0138] The acquisition unit 310 is further used to acquire the communication address of the virtual firewall that sends the data flow information;

[0139] The updating unit 360 is used to generate a new hash record according to the communication address and the parsed data, and update the hash table through the new hash record.

[0140] As an optional implementation, the acquisition unit 310 is further configured to acquire the address information of the virtual firewall that sends the data flow information when it is determined that the data flow information is duplicate traffic;

[0141] The judging unit 330 is further used to judge whether the address information of the virtual firewall matches the target hash record;

[0142] The acquisition unit 310 is further used to acquire data packet information of the data flow information when the address information of the virtual firewall matches the target hash record;

[0143] The updating unit 360 is also used to update the target hash record according to the data packet information.

[0144] As an optional implementation manner, the traffic statistics deduplication device further includes:

[0145] The acquisition unit 310 is further configured to acquire a session flag field in a target hash record when the address information of the virtual firewall does not match the target hash record;

[0146] A determination unit 370, configured to determine a session direction according to a session flag field;

[0147] The sending unit 380 is used to send the session direction and tuple information to the distributed firewall corresponding to the target session, so that the distributed firewall marks the traffic of the target session according to the session direction and tuple information, and does not perform traffic statistics on the marked traffic.

[0148] As an optional implementation manner, the traffic statistics deduplication device further includes:

[0149] The acquisition unit 310 is also used to acquire all data flow information and statistical requirements stored on the disk;

[0150] The processing unit 390 is used to perform statistical processing on all data flow information according to statistical requirements to obtain flow statistical information; and output the flow statistical information.

[0151] In the embodiments of the present application, the explanation of the traffic statistics deduplication device can refer to the description in Embodiment 1 or Embodiment 2, and will not be further elaborated in this embodiment.

[0152] It can be seen that the traffic statistics deduplication device described in this embodiment can parse the received data flow information and identify the parsed data through the hash table, thereby determining whether the data flow information is duplicate traffic, and caching the data flow information when it is not duplicate traffic, and storing the data flow information on the disk when the target session ends. It can be seen that the device can deduplicate traffic, avoid the same traffic being counted repeatedly, improve the accuracy of traffic statistics, and thus improve traffic query performance.

[0153] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute any traffic statistics deduplication method in Embodiment 1 or Embodiment 2 of the present application.

[0154] An embodiment of the present application provides a computer-readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, any traffic statistics deduplication method in Embodiment 1 or Embodiment 2 of the present application is executed.

[0155] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0156] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0157] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0158] The above description is only an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0159] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0160] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

Claims

1. A traffic statistics deduplication method, characterized in that: include: Get the data flow information of the target session; Parsing the data stream information to obtain parsed data; Obtaining tuple information in the parsed data; Determine whether a target hash record matching the tuple information can be found in a pre-stored hash table; If yes, determining that the data flow information is repeated traffic; If not, determining that the data flow information is not duplicate traffic, and caching the data flow information; When the target session ends, the cached data stream information is stored on disk; Wherein, the method further comprises: When it is determined that the data flow information is duplicate traffic, obtaining address information of a virtual firewall that sends the data flow information; Determine whether the address information of the virtual firewall matches the target hash record; If there is a match, obtaining the data packet information of the data flow information; Update the target hash record according to the data packet information; Wherein, the method further comprises: If the address information of the virtual firewall does not match the target hash record, obtaining the session flag field in the target hash record; Determining a session direction according to the session flag field; The session direction and the tuple information are sent to the distributed firewall corresponding to the target session, so that the distributed firewall marks the traffic of the target session according to the session direction and the tuple information, and does not perform traffic statistics on the marked traffic.

2. The traffic statistics deduplication method according to claim 1, characterized in that: After caching the data stream information, the method further includes: Obtaining the communication address of the virtual firewall that sends the data flow information; A new hash record is generated according to the communication address and the parsed data, and the hash table is updated by the new hash record.

3. The method for deduplication of traffic statistics according to claim 1, characterized in that: The method further comprises: Obtain all data flow information and statistical requirements stored on disk; Performing statistical processing on all the data flow information according to the statistical requirements to obtain flow statistical information; Output the traffic statistics information.

4. A traffic statistics deduplication device, characterized in that: The traffic statistics deduplication device comprises: An acquisition unit, used for acquiring data flow information of a target session; A parsing unit, used for parsing the data stream information to obtain parsed data; A judging unit, configured to judge whether the data flow information is duplicate traffic according to a preset hash table and the parsed data; A cache unit, configured to cache the data flow information when it is determined that the data flow information is not duplicate traffic; A storage unit, configured to store the cached data stream information on a disk when the target session ends; Wherein, the judging unit comprises: An acquisition subunit, used for acquiring tuple information in the parsed data; A judging subunit, used to judge whether a target hash record matching the tuple information can be found in a pre-stored hash table; A determination subunit, configured to determine that the data flow information is duplicate traffic when it is determined that the target hash record can be queried; and to determine that the data flow information is not duplicate traffic when it is determined that the target hash record cannot be queried; The acquisition unit is further used to acquire the address information of the virtual firewall that sends the data flow information when it is determined that the data flow information is duplicate traffic; The judging unit is further used to judge whether the address information of the virtual firewall matches the target hash record; The acquisition unit is further used to acquire data packet information of the data flow information when the address information of the virtual firewall matches the target hash record; The updating unit is further used to update the target hash record according to the data packet information; Wherein, the traffic statistics deduplication device also includes: The acquisition unit is further used to acquire the session flag field in the target hash record when the address information of the virtual firewall does not match the target hash record; A determination unit, used for determining a session direction according to a session flag field; The sending unit is used to send the session direction and tuple information to the distributed firewall corresponding to the target session, so that the distributed firewall marks the traffic of the target session according to the session direction and tuple information, and does not perform traffic statistics on the marked traffic.

5. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform the traffic statistics deduplication method according to any one of claims 1 to 3.

6. A readable storage medium, characterized in that: The readable storage medium stores computer program instructions, and when the computer program instructions are read and executed by a processor, the traffic statistics deduplication method according to any one of claims 1 to 3 is executed.

Images