Method and device for detecting abnormal data flow in a mass data flow scenario

Abstract

The application discloses a kind of detection method and device for abnormal data stream under the scene of mass data stream, belong to data detection field, comprising: generating data fingerprint;In data stream conversion stage, recalculate data fingerprint according to the same way with data report stage;According to data fingerprint, the last hop node before the data stream reaches current gateway node and the next hop node after leaving current gateway node, generate flow conversion trace fingerprint, record the actual flow conversion relationship of "data fingerprint-last node-next node" of the data stream;According to the number of filters actually hit and the total number of filters that should be queried according to the expected flow conversion path, calculate overall hit rate;When overall hit rate is lower than preset threshold, determine that the data stream is abnormal data stream and generate abnormal violation early warning.The application does not depend on frequent database query, realizes the quick detection and real-time early warning of whether data stream is reported, whether it is through specified path, whether there is bypass or abnormal transfer.

Description

Technical Field

[0001] This invention belongs to the field of data stream detection and network security technology, and particularly relates to a method and device for detecting abnormal data streams in scenarios with massive data streams. Background Technology

[0002] Currently, mainstream data flow compliance detection solutions typically rely on real-time queries of a central database, filing rule tables, or whitelist tables. When a data flow reaches a gateway node, the system needs to extract key fields from the data packet and perform matching operations in the central database or large-scale rule tables to determine whether the data flow is compliant. Some solutions have introduced a multi-level gateway linkage mechanism, attempting to trace the complete transmission path of the data flow by performing table lookup operations at each level of the gateway.

[0003] However, in scenarios with massive, high-concurrency, and high-speed data flows, the above solutions have the following drawbacks: First, frequent access to the central database or large-scale rule tables will result in high query latency and system load, making it difficult to meet the real-time detection requirements under high-speed links; Second, traditional table lookup detection can usually only determine whether the data flow hits the rule at a certain node, and it is difficult to record the continuous flow path of data between gateways at low cost, so it is difficult to detect violations such as abnormal transit, path deviation, or detours in a timely manner.

[0004] To address the problems of existing technologies, it is particularly necessary to develop a method and device for detecting abnormal data streams in scenarios involving massive data streams that does not rely on frequent database queries and can achieve rapid detection and real-time early warning of whether data streams have been reported, whether they have passed through a specified path, and whether they have detours or abnormal transfers. Summary of the Invention

[0005] To address the shortcomings and improvement needs of existing technologies, this invention provides a method and apparatus for detecting abnormal data streams in massive data stream scenarios. Its purpose is to achieve rapid detection and real-time early warning of whether data streams are reported, whether they pass through a specified path, and whether there are detours or abnormal transfers in massive, high-concurrency, and high-speed data stream environments without relying on frequent database queries.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for detecting abnormal data streams in massive data stream scenarios is provided, applied to a data flow network including multi-level gateway nodes, wherein each gateway node is equipped with a reporting fingerprint filter and a flow trace filter, including: S1: During the data reporting stage, extract the basic network fields from the reported data flow information, generate data fingerprints based on the basic network fields, and write the data fingerprints into the reporting fingerprint filters of the gateway nodes at all levels on the expected flow path; S2: During the data flow phase, when the data flow arrives at any level of gateway node, extract the basic network fields from the current traffic packet, recalculate the data fingerprint in the same way as in the data reporting phase, and query it in the reporting fingerprint filter of the current gateway node; if it does not match, mark the data flow as unreported traffic; if it matches, execute S3. S3: Generate a flow trace fingerprint based on the data fingerprint, the previous hop node, and the next hop node, and write it into the flow trace filter of the current gateway node; S4: During the anomaly detection phase, along the expected flow path of the data stream, perform a joint query on the reporting fingerprint filter and flow trace filter of each level of gateway node; S5: Calculate the overall hit rate based on the actual number of filters hit and the total number of filters to be queried; S6: When the overall hit rate is lower than the preset threshold, the data stream is determined to be an abnormal data stream and an abnormal violation warning is generated; otherwise, it is determined to be a normal flow.

[0007] According to another aspect of the present invention, an abnormal data stream detection device for massive data stream scenarios is provided, including a fingerprint generation and reporting module, a real-time identification module, a flow tracking module, a joint judgment module, and an early warning module.

[0008] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described detection method.

[0009] According to another aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described detection method.

[0010] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects: (1) This invention replaces real-time table lookup of central database or large-scale rule table with filter query. After the data flow arrives at the gateway, only constant time fingerprint calculation and filter query are needed to complete the reporting and identification, which significantly reduces the query latency and system load in the scenario of massive data flow and meets the real-time detection requirements of high-speed link; (2) This invention is concerned not only with whether the data is reported, but also with whether the data has actually passed through the gateways at all levels according to the expected path: by using the flow fingerprint of "data fingerprint - previous node - next node", a continuous flow trace is formed between the gateways with extremely low storage overhead, making path verification possible. (3) This invention integrates reporting identification, path tracking and abnormal warning into a hierarchical fingerprint filtering structure by combining dual filters of reporting fingerprint filter and flow trace filter and multi-level gateway joint hit rate judgment. It can quantitatively distinguish various abnormal types such as unreported transmission, missing path, abnormal transfer and detour flow, and realize low-overhead, real-time and path-aware abnormal violation detection. Attached Figure Description

[0011] Figure 1 The diagram shows a flowchart of an abnormal data stream detection method for massive data stream scenarios provided by an embodiment of the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0013] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0014] This application discloses a method for detecting abnormal data flows in massive data flow scenarios. This method is based on a hierarchical fingerprint filtering mechanism, deploying a reporting fingerprint filter and a flow tracking filter on multi-level gateways. The reporting fingerprint filter determines whether the current data flow is a reported or monitored data flow, while the flow tracking filter records the actual flow relationship between adjacent gateway nodes. Through this dual-filter structure, the system can quickly identify massive data flows, record flow paths, and provide anomaly warnings without frequent access to the central database or rule tables. The patent explicitly states that the system generates data fingerprints based on source IP, destination IP, source port, destination port, and protocol type, and records the actual flow relationship through a "data fingerprint—previous node—next node" sequence.

[0015] Example 1: Figure 1 The diagram shows a flowchart of the abnormal data stream detection method for massive data stream scenarios provided in this embodiment. This method is based on a hierarchical identifier fingerprint filtering mechanism and specifically includes the following steps.

[0016] Step 1: Construct a hierarchical filtering structure. Two filters are set up on each gateway node: the first filter is a reporting fingerprint filter, used to store the data fingerprints corresponding to reported data flows; the second filter is a flow tracking filter, used to record whether the data flow actually passes through the current gateway node and its adjacent flow relationships. In actual deployment, different gateways correspond to different levels of data flow nodes. When data flows out normally, they should pass through multiple gateways sequentially according to the preset path. By deploying dual filters, subsequent detection no longer relies solely on whether a single gateway is hit, but can determine whether the data flow follows the expected path by the combined hit status of multiple gateways.

[0017] In a preferred embodiment, both types of filters are implemented using Bloom filters: during writing, the fingerprint is mapped to k bits of the bit array using k independent hash functions and set; during querying, if all k bits are set, a match is determined. The length of the bit array and the number of hash functions in the Bloom filter are determined based on the expected number of fingerprints and the tolerable false positive rate. Considering that continuous writing of the flow trace in a massive data stream scenario will gradually saturate the bit array and increase the false positive rate, the flow trace filter is used in a rolling manner according to a preset time window (e.g., 1 hour): when the current window expires, a new bit array is activated, and the oldest bit array that has exceeded the retention period is reset. The bit arrays of the historical windows can still be used for joint queries within the retention period.

[0018] Step Two: Generate Data Fingerprints. During the data reporting phase, basic network fields are extracted from legitimate traffic samples, registered traffic rules, or historical whitelist traffic. These fields include source IP, destination IP, source port, destination port, and protocol type. These fields are basic fields that can be directly read from regular traffic packets, do not depend on complex business attributes, and are easy to calculate quickly in high-speed link environments. Data fingerprints are generated based on these fields: DataFP = Hash(IP_src ∥ IP_dst ∥ Port_src ∥ Port_dst ∥ Protocol) In this context, DataFP represents the data fingerprint, Hash(·) is the hash function (e.g., MurmurHash or SHA series hash functions), ∥ represents field concatenation, IP_src represents the source IP, IP_dst represents the destination IP, Port_src represents the source port, Port_dst represents the destination port, and Protocol represents the protocol type.

[0019] After generating the data fingerprint, it is written into the reporting fingerprint filters of each level of gateway along the expected flow path. At this point, the data fingerprint only indicates that the data flow belongs to an object that has been reported or requires supervision, and does not indicate that the data flow has actually passed through the corresponding gateway node.

[0020] Step 3: Real-time Data Flow Identification. During the data flow phase, when a data flow reaches a certain level of gateway node, the gateway reads the source IP, destination IP, source port, destination port, and protocol type from the current traffic packet and recalculates the data fingerprint in the same way as in the reporting phase. It then inputs this fingerprint into the reporting fingerprint filter of the current gateway node for querying. If no match is found, it means that the data flow has not been reported in the corresponding rules of the current node. The system marks it as unknown traffic or unreported traffic, records its data fingerprint, arrival time, and arrival at the gateway node, and periodically summarizes and reports it for reporting verification and unknown traffic statistics, without interrupting the normal forwarding of the data flow at the current node. If a match is found, it means that the data flow belongs to a reported or monitored data flow, and the flow tracking operation continues.

[0021] Step 4: Write the data flow trace. Once the data fingerprint is matched in the reporting fingerprint filter of the current node, the system generates a data flow trace fingerprint based on the previous hop node before the data flow reaches the current gateway and the next hop node after leaving the current gateway: TraceFP = Hash(DataFP ∥ PrevNode ∥ NextNode) Here, TraceFP represents the flow fingerprint, and PrevNode and NextNode represent the node identifiers of the previous and next hop nodes, respectively. The node identifiers can be obtained from the routing and forwarding information of the current gateway within the encapsulation information of the traffic packet (such as the tunnel encapsulation header or link layer address). The system writes TraceFP into the flow fingerprint filter of the current gateway node to record the actual flow relationship of "data fingerprint—previous node—next node". In this way, the system can form a continuous flow trace between gateways, providing a basis for subsequent path verification.

[0022] Step 5: Jointly query gateway filters. During the anomaly / violation determination phase, along the expected path of the data flow, jointly query the reporting fingerprint filters and flow tracking filters of each level of gateway: First, query the reporting fingerprint filters of each level of gateway to determine whether the data fingerprint belongs to a reported regulatory object; then query the flow tracking filters of each level of gateway to determine whether the data flow forms a continuous and consistent "previous node - next node" relationship. If all reporting fingerprints and flow tracking filters at each level are successfully matched, it indicates that the data flow conforms to the expected path; if some filters are not matched, it indicates that the data flow may have issues such as unreported transmission, missing paths, abnormal transit, or detours.

[0023] Step Six: Calculate the overall hit rate. Calculate the overall hit rate based on the hit rate of each filter: HitRate = N_hit / N_total Where N_hit represents the number of filters actually hit, and N_total represents the total number of filters that should be queried according to the expected path. In this embodiment, each level of gateway is equipped with a reporting fingerprint filter and a flow tracking filter. Therefore, when the expected path includes M level gateways, N_total = 2M.

[0024] Step 7: Generate abnormal violation warning results. When the overall hit rate is not lower than the preset threshold, the data flow is determined to be normal; when the overall hit rate is lower than the preset threshold, an abnormal violation warning is generated. Further, the anomaly type is located based on the type of the missed filter and the gateway: a missed report fingerprint filter corresponds to unreported transmission; a missed flow trace filter on a single gateway corresponds to missing paths or abnormal transfers; a missed flow trace filter on multiple consecutive gateways corresponds to detour transfers. The warning result includes information such as data fingerprint, anomaly occurrence time, gateway node where the anomaly occurred, overall hit rate, anomaly type, and warning level. The warning level is determined according to the threshold range into which the overall hit rate falls (e.g., a hit rate below 50% is a high-level warning, and 50% to 80% is a medium-level warning).

[0025] As a specific example, suppose the expected path includes three levels of gateways (M=3, N_total=6), and the preset threshold is 0.8. A joint query result for a data flow is as follows: if all three levels of reporting fingerprint filters are hit, the first and third level flow trace filters are hit, and the second level flow trace filter is not hit, then N_hit=5, HitRate≈0.83, which is not lower than the threshold, and is judged as normal flow (the missing second-level trace can be recorded for manual review); if neither the second nor the third level flow trace filters are hit, then N_hit=4, HitRate≈0.67, which is lower than the threshold, and two consecutive levels of traces are missing, the data flow is suspected of bypassing the flow, and an abnormal violation warning is generated, including data fingerprints, abnormal gateway nodes (second and third level gateways), a hit rate of 0.67, and the corresponding warning level.

[0026] Through the above methods, a complete implementation process is formed, which includes "data reporting - data fingerprint generation - reporting fingerprint filtering - flow trace recording - joint query - hit rate calculation - anomaly warning", enabling rapid detection of behaviors such as unreported transmission, detours, path deviations and abnormal transfers in massive data streams.

[0027] Example 2: This embodiment provides an abnormal data stream detection device for massive data stream scenarios, including a fingerprint generation and reporting module, a real-time identification module, a data flow tracking module, a joint judgment module, and an early warning module. The fingerprint generation and reporting module is used to execute step two in embodiment 1; the real-time identification module is used to execute step three; the data flow tracking module is used to execute step four; the joint judgment module is used to execute steps five and six; and the early warning module is used to execute step seven. The specific implementation of each module is the same as the corresponding steps in embodiment 1, and will not be repeated here.

[0028] Example 3: This embodiment provides an electronic device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the detection method of Embodiment 1. The electronic device may be a network device, a server, or a cloud computing device deployed at a gateway node. This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the detection method of Embodiment 1.

[0029] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting abnormal data streams in scenarios involving massive data streams, characterized in that, This method is applied to data flow networks comprising multi-level gateway nodes. Each gateway node is equipped with a reporting fingerprint filter and a flow tracking filter. The reporting fingerprint filter stores the data fingerprints corresponding to reported data flows, while the flow tracking filter records the actual flow relationships of data flows as they pass through the corresponding gateway nodes. The method includes: S1: During the data reporting stage, extract the basic network fields from the reported data flow information, generate data fingerprints based on the basic network fields, and write the data fingerprints into the reporting fingerprint filters of the gateway nodes at all levels on the expected flow path; S2: During the data flow phase, when the data flow arrives at any level gateway node, the basic network field is extracted from the current traffic packet, the data fingerprint is recalculated in the same way as in the data reporting phase, and the recalculated data fingerprint is entered into the reporting fingerprint filter of the current gateway node for querying; if no match is found, the data flow is marked as unreported traffic; if a match is found, step S3 is executed. S3: Based on the data fingerprint, the previous hop node before the data flow arrives at the current gateway node, and the next hop node after leaving the current gateway node, generate a flow trace fingerprint and write the flow trace fingerprint into the flow trace filter of the current gateway node to record the actual flow relationship of the data flow from "data fingerprint - previous node - next node". S4: During the anomaly detection phase, along the expected flow path of the data stream, perform a joint query on the reporting fingerprint filter and flow trace filter of each level of gateway node to obtain the hit status of each filter. S5: Calculate the overall hit rate based on the actual number of filters hit and the total number of filters that should be queried according to the expected flow path; S6: When the overall hit rate is lower than the preset threshold, the data stream is determined to be an abnormal data stream and an abnormal violation warning is generated; otherwise, the data stream is determined to be a normal flow.

2. The method for detecting abnormal data streams in massive data stream scenarios according to claim 1, characterized in that, The basic network fields include source IP, destination IP, source port, destination port, and protocol type; the data fingerprint is generated using the following formula: DataFP=Hash(IP_src∥IP_dst∥Port_src∥Port_dst∥Protocol) In this context, DataFP represents data fingerprint, Hash(·) represents hash function, ∥ represents field concatenation, IP_src represents source IP, IP_dst represents destination IP, Port_src represents source port, Port_dst represents destination port, and Protocol represents protocol type.

3. The method for detecting abnormal data streams in massive data stream scenarios according to claim 1, characterized in that, In step S3, the transfer fingerprint is generated using the following formula: TraceFP=Hash(DataFP∥PrevNode∥NextNode) Wherein, TraceFP represents the flow trace fingerprint, DataFP represents the data fingerprint, PrevNode represents the node identifier of the previous hop node before the data flow reaches the current gateway node, and NextNode represents the node identifier of the next hop node after the data flow leaves the current gateway node; the node identifiers of the previous hop node and the next hop node are obtained from the routing and forwarding information of the current gateway node in the encapsulation information of the traffic packet.

4. The method for detecting abnormal data streams in massive data stream scenarios according to claim 1, characterized in that, Both the reporting fingerprint filter and the flow trace filter are Bloom filters. During writing, the fingerprint is mapped to multiple bits of a bit array using multiple independent hash functions and then set. During querying, a hit is determined when all of the multiple bits are set. The flow trace filter is used in a rolling manner according to a preset time window. When the current time window expires, a new bit array is activated and the oldest bit array is reset to avoid the increase in false positive rate caused by filter saturation in massive data flow scenarios.

5. The method for detecting abnormal data streams in massive data stream scenarios according to claim 1, characterized in that, In step S2, for data streams marked as unreported traffic, their data fingerprint, arrival time, and arrival at the gateway node are recorded, without interrupting the forwarding of the data stream at the current gateway node.

6. The method for detecting abnormal data streams in massive data stream scenarios according to claim 1, characterized in that, In step S5, the overall hit rate is calculated using the following formula: HitRate = N_hit / N_total Where HitRate represents the overall hit rate, N_hit represents the number of filters actually hit in the joint query, and N_total represents the total number of filters that should be queried according to the expected flow path; when the expected flow path includes M-level gateway nodes and each level of gateway node queries the reporting fingerprint filter and the flow trace filter, N_total = 2M.

7. The method for detecting abnormal data streams in massive data stream scenarios according to claim 1, characterized in that, In step S6, the anomaly type is also determined based on the type of the missed filter and the gateway node: when the reporting fingerprint filter is missed, it is determined to be unreported transmission; when the flow tracking filter of a single gateway node is missed, it is determined to be a missing path or abnormal transit. When the flow tracking filters of multiple consecutive gateway nodes fail to hit, it is determined to be a bypass flow; the abnormal violation warning includes one or more of the following: data fingerprint, abnormal occurrence time, abnormal occurrence gateway node, overall hit rate, abnormal type and warning level, and the warning level is determined according to the preset threshold range in which the overall hit rate falls.

8. A device for detecting abnormal data streams in scenarios involving massive data streams, characterized in that, include: The fingerprint generation and reporting module is used to extract basic network fields from the reported data flow information, generate data fingerprints, and write the data fingerprints into the reporting fingerprint filters of the gateway nodes at all levels on the expected flow path; The real-time identification module is used to extract the basic network fields from the current traffic packet and recalculate the data fingerprint when the data flow arrives at any level gateway node. It then queries the reporting fingerprint filter of the current gateway node and marks the data flow as unreported traffic if it does not match the fingerprint. The flow tracking module is used to generate a flow tracking fingerprint based on the data fingerprint, the previous hop node, and the next hop node after the data fingerprint is hit, and write it into the flow tracking filter of the current gateway node. The joint judgment module is used to perform joint queries on the reporting fingerprint filters and flow trace filters of gateway nodes at all levels along the expected flow path, and calculate the overall hit rate based on the actual number of filters hit and the total number of filters to be queried. The early warning module is used to determine that the data stream is an abnormal data stream and generate an abnormal violation warning when the overall hit rate is lower than a preset threshold.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for detecting abnormal data streams in a massive data stream scenario as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for detecting abnormal data streams in a massive data stream scenario as described in any one of claims 1 to 7.

Images