Burst traffic monitoring method, device, terminal and storage medium for real-time analysis of high-speed data streams

Abstract

The present application discloses a burst traffic monitoring method, device, terminal, and storage medium for real-time analysis of high-speed data streams, wherein the method comprises: receiving data streams arriving within the current monitoring period, each data stream including multiple arriving data items; assigning the arriving data items to a pre-constructed storage bucket array based on a hash map and a unique identifier of each arriving data item, and determining the data item ID of each arriving data item in the storage bucket array; performing frequency determination on the arriving data items based on the data item ID, and if the data stream corresponding to the arriving data item is a suspicious burst stream, tracking the suspicious burst stream and summarizing it to obtain a suspicious burst stream tracking table; traversing the suspicious burst stream tracking table, determining whether each suspicious burst stream is a burst data stream based on a preset burst data judgment mechanism, summarizing it to obtain a burst data stream list, and outputting it. The method of the present application can effectively reduce computational overhead and memory consumption, and achieve strong real-time and high-precision burst traffic monitoring.

Description

Technical Field

[0001] The present application relates to the field of traffic monitoring of e-commerce platforms, and in particular to a burst traffic monitoring method, device, terminal and storage medium for real-time analysis of high-speed data streams. Background Art

[0002] In recent years, e-commerce platforms' marketing models have become increasingly complex, evolving from traditional static product introductions to a multi-dimensional competitive model encompassing livestreaming, short video ads, and competitive bidding. This shift necessitates fine-grained traffic measurement and analysis of high-speed data streams to evaluate the effectiveness of different promotional methods in real time and select the most appropriate marketing strategy for each product. Traffic analysis plays a crucial role in increasing product awareness and conversion rates, especially within budgetary constraints. Maximizing product traffic generation is a pressing issue.

[0003] Currently, while analyzing massive, high-speed data streams on e-commerce platforms can meet their needs to a certain extent, it exposes significant shortcomings when faced with bursty traffic. For example, existing e-commerce platforms have a large number of products, marketing accounts, and customer groups, and the browsing and transaction data generated in real time is also massive and high-speed. Performing correlation analysis on these data presents challenges such as high computational complexity and excessive runtime memory consumption. Existing technologies, even using dedicated large-scale servers, struggle to achieve real-time monitoring of bursty traffic, resulting in long detection delays and an inability to capture network hotspots and bursty traffic changes in a timely manner. Furthermore, these methods often rely on high-performance servers, resulting in excessive runtime memory consumption and computational complexity, making it difficult to achieve both real-time performance and cost-effectiveness. Therefore, designing a technical solution that can effectively reduce computational overhead and memory consumption while achieving strong real-time, high-precision bursty traffic monitoring has become a key issue that needs to be addressed. Summary of the Invention

[0004] In order to accurately monitor sudden high-speed traffic in real time, the present application provides a sudden traffic monitoring method, device, terminal and storage medium for real-time analysis of high-speed data streams.

[0005] In a first aspect, the present application provides a method for monitoring burst traffic for real-time analysis of high-speed data streams, using the following technical solution:

[0006] A method for monitoring burst traffic for real-time analysis of high-speed data streams includes the following steps:

[0007] receiving a data stream arriving in a current monitoring period, each data stream including a plurality of arriving data items, and all the arriving data items in the same data stream having the same unique identifier;

[0008] Allocating the corresponding arriving data items to a pre-built bucket array according to a hash map and the unique identifier of each arriving data item, and determining a data item ID of each arriving data item in the bucket array;

[0009] Performing a frequency determination on the arrival data item according to the data item ID, and when it is determined that the data flow corresponding to the arrival data item is a suspicious burst flow based on a result of the frequency determination, tracking the suspicious burst flow and summarizing the suspicious burst flow to obtain a suspicious burst flow tracking table;

[0010] At the end of each monitoring cycle, the obtained suspicious burst flow tracking table is traversed, and whether each suspicious burst flow is a burst data flow is determined according to a preset burst data judgment mechanism, and a burst data flow list is obtained and output.

[0011] By adopting the above technical solution, combined with hash mapping and data item ID, real-time analysis of traffic and judgment of suspicious burst flows are realized, efficient data flow processing and distribution are achieved, and the ability to detect abnormal traffic is improved; at the same time, by tracking suspicious burst flows, potential traffic anomalies are fully understood, and accurate judgment and timely response to burst flows are achieved.

[0012] Preferably, before receiving the data stream arriving in the current monitoring period, the method further includes the following steps:

[0013] Constructing an overall monitoring structure, the overall monitoring structure including a bucket array, each bucket in the bucket array including a suspicious traffic identification part, a small flow counting and filtering part, and a suspicious burst flow tracking part;

[0014] The suspicious traffic identification part includes a bit array, the small flow counting and filtering part includes a counter array and a data item ID array, and the suspicious burst flow tracking part includes a hash table;

[0015] The overall monitoring structure is initialized, and all bits in the bit array, each cell in the counter array and the data item ID array, and all fields in the hash table are initialized to 0.

[0016] By adopting the above technical solution, an overall monitoring structure is constructed. By initializing the bit array, counter array, data item ID array and hash table therein, a basis is provided for the subsequent mapping, frequency determination and tracking of data items, avoiding misjudgment caused by unclear initial values.

[0017] Preferably, allocating the corresponding arriving data items to a pre-built bucket array based on a hash map and the unique identifier of each arriving data item, and determining the data item ID of each arriving data item in the bucket array, specifically comprises the following steps:

[0018] Mapping the arriving data item to the bucket array according to a preconfigured first mapping function and a unique identifier of the arriving data item to obtain a first mapping position of the arriving data item;

[0019] The first mapping function is, ;

[0020] in, is an independent and uniform hash function, is the unique identifier of the arrival data item, represents the remainder operator, is the length of the bucket array;

[0021] Calculate the first mapping position of each of the arriving data items in the bucket array, specifically,

[0022] ;

[0023] in, is the calculated first mapping position, that is, the position of the arriving data item in the bucket array.

[0024] By adopting the above technical solution, the first mapping position is determined based on the modulo operator, which simplifies the calculation process and reduces the complexity of the algorithm. It is suitable for real-time processing in high-speed data stream scenarios, and accurately maps the arriving data items to specific positions in the bucket array, ensuring the efficiency and accuracy of data stream processing.

[0025] Preferably, the frequency determination of the arriving data item based on the data item ID specifically includes the following steps:

[0026] Mapping each of the arriving data items arriving at the storage bucket into the bit array according to a preconfigured second mapping function, and calculating a second mapping position of each of the arriving data items in the bit array in combination with the first mapping position;

[0027] The second mapping function is, ;

[0028] in, is an independent and uniform hash function, is the unique identifier of the arrival data item, represents the remainder operator, is the length of the bit array;

[0029] ;

[0030] in, The calculated second mapping position, i.e., the position of each of the arriving data items in the bit array;

[0031] Perform frequency determination on the arrival data item and obtain an evaluation result, specifically,

[0032] Determine whether the data item ID of the second mapping position meets the preset low-frequency judgment rule. If so, the judgment result is that the arriving data item belongs to a low-frequency element; if not, the judgment result is that the arriving data item belongs to a high-frequency element.

[0033] By adopting the above technical solution, the arriving data item is mapped into the bit array through a pre-configured second mapping function, the second mapping position is calculated in combination with the first mapping position, the position of the arriving data item in the bit array is calculated, and the data item ID of the second mapping position is judged according to the low-frequency judgment rule. This can effectively determine the frequency of the arriving data item, thereby providing an accurate basis for subsequent suspicious burst flow identification.

[0034] Preferably, when it is determined that the data flow corresponding to the arriving data item is a suspicious burst flow in combination with the result of the frequency determination, tracking the suspicious burst flow and summarizing it to obtain a suspicious burst flow tracking table specifically includes the following steps:

[0035] If the arriving data item belongs to a high-frequency element, the data flow corresponding to the arriving data item is a suspicious burst flow, and the current suspicious burst flow is marked and tracked, specifically,

[0036] According to a preset third mapping function, the current arrival data item and the corresponding data item frequency are mapped to the hash table item of the hash table to obtain the third mapping position of the arrival data item, each of the hash table items includes a plurality of units, each of the units includes field, the frequency of burst flows in the current monitoring period, and the identifier of suspicious burst flows;

[0037] Determine whether there is a unique identifier that matches the current arrival data item in the unit of the hash table item corresponding to the third mapping position. field, if present, updating the burst flow frequency in the current monitoring period according to the frequency of the data item;

[0038] If it does not exist and there is an empty unit in the hash table entry, then the information of the current arrival data item is updated in the empty unit.

[0039] If it does not exist and there is no empty unit in the hash table entry, select a unit that meets the preset replacement criteria from the hash table entry for replacement, replace the information of the current unit with the information of the current arriving data item, and mark the corresponding data flow as a suspicious burst flow being tracked;

[0040] All tracked suspicious burst flows and the corresponding contents of the hash table entries are summarized to obtain a suspicious burst flow tracking table.

[0041] By adopting the above technical solution, the arriving data items are mapped to the hash table items of the hash table, and their positions are determined by using the third mapping function, thereby improving the efficiency and accuracy of data processing; the units in the hash table items are judged and updated to ensure that the information of high-frequency data items can be recorded in a timely manner and avoid duplicate storage; a replacement mechanism is adopted when there are no empty units in the hash table items to ensure the effective utilization of the hash table; the data streams corresponding to high-frequency elements can be accurately marked and tracked, providing reliable data support for subsequent traffic analysis and decision-making.

[0042] Preferably, if the determination result is that the arriving data item belongs to a low-frequency element, the method specifically includes the following steps:

[0043] Mapping the currently arrived data item to the counter array and the data item ID array according to a preset fourth mapping function to obtain a fourth mapping position of the arrived data item;

[0044] Determine whether the data item ID of the fourth mapping position matches the unique identifier of the currently arrived data item, and if so, increment the counter value of the fourth mapping position by one;

[0045] If there is no match and the fourth mapping position is empty, updating the data item ID of the fourth mapping position according to the information of the currently arrived data item and initializing the counter value to 1;

[0046] If there is no match, and the fourth mapping location is not empty, performing a subtraction operation on the counter value;

[0047] The counter value is monitored, and when the counter value reaches a preset small flow threshold, the arriving data item is marked as a suspicious burst flow and tracked; when the counter value is 0, the data item ID of the fourth mapping position is cleared.

[0048] By adopting the above technical solution, the arriving data items are mapped to the counter array and the data item ID array through the fourth mapping function, and then the counter value is dynamically maintained by judging whether the data item ID matches, which truly reflects the frequency of occurrence of the data item. When the counter value reaches the low-flow threshold, the suspicious burst flow is marked and tracked in time, which effectively identifies potential suspicious burst flows and improves the sensitivity of monitoring.

[0049] Preferably, traversing the obtained suspicious burst flow tracking table at the end of each monitoring cycle, determining whether each suspicious burst flow is a burst data flow according to a preset burst data judgment mechanism, summarizing to obtain a burst data flow list and outputting the list, specifically comprises the following steps:

[0050] After each monitoring cycle ends, the counter array and the data item ID array of each bucket in the bucket array are cleared;

[0051] Traversing each unit of each hash table entry in the suspicious burst flow tracking table, each unit includes the burst flow frequency in the current monitoring period, the burst flow frequency in the previous monitoring period, the timestamp of becoming a suspicious burst flow, and the duration after the burst;

[0052] Obtaining frequency changes in adjacent cycles based on the burst flow frequency in the current monitoring cycle and the burst flow frequency in the previous monitoring cycle, determining whether each of the suspicious burst flows has a burst based on the timestamp of the suspected burst flow and a preset burst threshold, and determining whether each of the suspicious burst flows is a burst data flow based on the duration after the burst;

[0053] All the determined burst data flows are summarized to obtain a burst data flow list and output the list.

[0054] By adopting the above technical solution, the hash table entry units in the suspicious burst flow tracking table are traversed to extract key information for subsequent burst flow judgment; based on the frequency changes of adjacent cycles, combined with the timestamp of the suspicious burst flow and the preset burst threshold, each suspicious burst flow is accurately judged whether a burst occurs, thereby improving the sensitivity and reliability of detection, and further judging whether the suspicious burst flow is a real burst data flow based on the duration after the burst, filtering out false burst flows with short fluctuations, improving the accuracy of the monitoring results, and realizing efficient real-time monitoring of burst traffic in high-speed data streams.

[0055] In a second aspect, the present application provides a burst traffic monitoring device for real-time analysis of high-speed data streams, which adopts the following technical solutions:

[0056] A burst traffic monitoring device for real-time analysis of high-speed data streams, including the following modules:

[0057] A data stream receiving module, configured to receive data streams arriving within a current monitoring period, wherein each data stream includes a plurality of arriving data items, and all arriving data items in the same data stream have the same unique identifier;

[0058] a data item mapping module, configured to allocate the corresponding arriving data items to a pre-built storage bucket array according to a hash map and the unique identifier of each arriving data item, and determine a data item ID of each arriving data item in the storage bucket array;

[0059] a suspicious burst flow tracking module, configured to perform frequency determination on the arriving data item based on the data item ID, and when it is determined that the data flow corresponding to the arriving data item is a suspicious burst flow based on the result of the frequency determination, track the suspicious burst flow and summarize it to obtain a suspicious burst flow tracking table;

[0060] The burst flow detection module is used to traverse the obtained suspicious burst flow tracking table at the end of each monitoring cycle, determine whether each suspicious burst flow is a burst data flow according to a preset burst data judgment mechanism, summarize and output a burst data flow list.

[0061] By adopting the above technical solution, the high-speed computing overhead and cache overhead required for real-time monitoring of burst traffic are reduced. On the basis of matching the high-speed data flow rate, the detection accuracy of bursty large flows is greatly improved, thereby realizing true strong real-time and high-precision bursty large flow monitoring.

[0062] In a third aspect, the present application provides a smart terminal that adopts the following technical solution:

[0063] An intelligent terminal includes a memory and a processor, wherein the memory stores at least one program, and the processor loads and executes the at least one program to implement the burst traffic monitoring method for real-time analysis of high-speed data streams as described above.

[0064] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0065] A computer-readable storage medium stores at least one program, which is loaded and executed by the processor to implement the burst traffic monitoring method for real-time analysis of high-speed data streams as described above.

[0066] In summary, this application has at least the following beneficial effects:

[0067] (1) This application constructs a bucket array monitoring structure including a bit array, a counter array, a data item ID array and a hash table, and combines the hash mapping method to allocate data items, which can significantly reduce the computational complexity and memory consumption, thereby achieving efficient monitoring of burst traffic in high-speed data streams.

[0068] (2) This application combines small flow counting and filtering mechanisms to effectively reduce the computational complexity and memory consumption when performing real-time analysis on massive high-speed data flows, optimize the overall storage space, and improve the operating efficiency of the system; through suspicious traffic identification and suspicious burst flow tracking, unnecessary search operations are reduced, the processing speed of data items is accelerated, and accurate screening and real-time tracking of high-frequency data flows are achieved, which greatly improves the accuracy and real-time performance of burst flow detection.

[0069] (3) This application determines the burst data flow by traversing the frequency changes in the suspicious burst flow tracking table at the end of each monitoring cycle, and further confirms the burst flow in combination with the burst duration, thereby ensuring the reliability and stability of the detection results, realizing the accurate identification and output of the burst traffic, solving the problem of long detection delay in the existing technology, and taking into account cost-effectiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is one of the overall flow charts of the burst traffic monitoring method for real-time analysis of high-speed data streams in this embodiment;

[0071] Figure 2 This is the second overall flow chart of the burst traffic monitoring method for real-time analysis of high-speed data streams in this embodiment;

[0072] Figure 3 This is a flowchart of the suspicious traffic identification part of the burst traffic monitoring method for real-time analysis of high-speed data streams in this embodiment;

[0073] Figure 4 This is a flowchart of the suspicious burst flow tracking portion of the burst flow monitoring method for real-time analysis of high-speed data flows in this embodiment;

[0074] Figure 5 This is a flow chart of the small flow counting and filtering part of the burst traffic monitoring method for real-time analysis of high-speed data flows in this embodiment;

[0075] Figure 6 This is a flow chart of the burst flow detection part of the burst flow monitoring method for real-time analysis of high-speed data flows in this embodiment;

[0076] Figure 7 This is a structural diagram of the burst traffic monitoring device for real-time analysis of high-speed data streams according to this embodiment. DETAILED DESCRIPTION

[0077] The present application provides a burst traffic monitoring method, device, terminal and storage medium for real-time analysis of high-speed data streams. In order to make the purpose, technical solution and advantages of the present application clearer, the implementation method of the present application will be further described in detail below.

[0078] The following is a further detailed description of an embodiment of a burst traffic monitoring method for real-time analysis of high-speed data streams of the present application in conjunction with the accompanying drawings.

[0079] The present invention provides a method for monitoring burst traffic in real time for high-speed data flow analysis. Figure 1 and Figure 2 As shown, the following steps are included:

[0080] S1. Construct an overall monitoring structure and initialize the overall monitoring structure.

[0081] S11. The overall monitoring structure includes a bucket array Each bucket in the bucket array contains a suspicious traffic identification part, a small flow count and a filtering part. and three parts: suspicious burst flow tracking.

[0082] The suspicious traffic identification part includes a bit array , the small flow counting and filtering part includes a counter array and an array of data item IDs ,The suspicious burst flow tracking part includes a hash table .

[0083] S12, for the suspicious traffic identification part, the bit array Each bit in is initialized to 0.

[0084] For the small flow counting and filtering part, the counter array Each counter value in is initialized to 0, and the data item ID array The value of each cell in is initialized to empty.

[0085] For the suspicious burst flow tracking part, the hash table It contains multiple units, each unit contains 6 fields, namely: the ID field used to record the suspicious burst flow identification, the burst flow frequency field used to record the current monitoring period. Field, used to record the frequency of burst flows in the last monitoring cycle Field used to record the timestamp of suspicious burst flow Field, used to record the duration of the burst Fields and used to identify conflicts field. 、 、 、 and All fields are initialized to 0, and the ID field is initialized to empty.

[0086] S2. Receive a data stream arriving in a current monitoring period, where each arriving data stream includes a plurality of arriving data items, wherein all arriving data items in the same data stream have the same unique identifier.

[0087] For each arriving data item, extract the unique identifier that represents the arriving data item. .

[0088] S3. Allocate the corresponding arriving data items to a pre-built bucket array based on the hash map and the unique identifier of each arriving data item, and determine the data item ID of each arriving data item in the bucket array. Specifically, the steps include:

[0089] S31 . Map the arriving data item to a bucket array according to a preconfigured first mapping function and a unique identifier of the arriving data item to obtain a first mapping position of the arriving data item.

[0090] In this embodiment, the first mapping function It is an element-to-bucket mapping function.

[0091] The element-bucket mapping function is,

[0092] ;

[0093] in, is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the remainder operator, The length of the bucket array.

[0094] By using independent and uniform hash functions, the uniform distribution of arriving data items in the bucket array is guaranteed, the collision probability is reduced, and the storage and retrieval efficiency is improved.

[0095] S32. Calculate the first mapping position of each arriving data item in the bucket array, specifically,

[0096] ;

[0097] in, The first mapping position obtained by calculation is the position of the data item in the bucket array.

[0098] S4, according to the data item ID, the frequency of the arrival data item is determined. When the data flow corresponding to the arrival data item is determined to be a suspicious burst flow based on the result of the frequency determination, the suspicious burst flow is tracked and summarized to obtain a suspicious burst flow tracking table, such as Figure 3 As shown, the specific steps include:

[0099] S41, according to the pre-configured second mapping function, the bucket Each arriving data item is mapped to the first bits, in this embodiment, the second mapping function is the element-to-bit mapping function.

[0100] The element-bit mapping function is,

[0101] ;

[0102] in, is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the remainder operator, is the length of the bit array;

[0103] S42, calculate the bit array of each arriving data item in combination with the first mapping position The second mapping position in ;

[0104] ;

[0105] in, is the calculated second mapping position, that is, the position of each arriving data item in the bit array.

[0106] S43. Determine the frequency of the arrival data item and obtain the evaluation result, specifically,

[0107] It is determined whether the data item ID of the second mapping position meets a preset low-frequency determination rule.

[0108] In this embodiment, the low-frequency judgment rule is to judge whether the data item ID of the second mapping position is 0. If it is satisfied, that is, , then the judgment result is that the arriving data item belongs to a low-frequency element; if it does not meet the requirements, that is, , then the judgment result is that the arriving data item belongs to a high-frequency element.

[0109] Through the above steps, it is recorded whether each flow is a suspicious burst flow in a certain monitoring period, so as to reduce unnecessary search operations and speed up the processing of data items.

[0110] S44, if the arriving data item belongs to a high-frequency element, the data flow corresponding to the arriving data item is a suspicious burst flow, and the current suspicious burst flow is marked and tracked, such as Figure 4 As shown, the specific steps include:

[0111] S441, current hash table When the arrival data item currently determined to be a high-frequency element and its data item frequency are inserted into the hash table, the current arrival data item and the corresponding data item frequency are mapped to the hash table according to the preset third mapping function. No. hash table entries.

[0112] In this embodiment, the third mapping function It is the element-hash table entry mapping function.

[0113] The element-hash table item mapping function is,

[0114] ;

[0115] in, is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the remainder operator, For hash tables length.

[0116] S442, obtaining the third mapping position of the data item,

[0117] ;

[0118] in, is the calculated third mapping position, that is, the position of each arriving data item in the hash table.

[0119] As mentioned above, each hash table entry contains multiple units, each of which contains an ID field that records the suspicious burst flow identifier, a field that records the burst flow frequency in the current monitoring period, and a field that records the number of burst flows in the current monitoring period. field, as well as the frequency of burst flows in the current monitoring period and the identification of conflicts field.

[0120] S443: Determine the hash table entry corresponding to the third mapping position Is there a unit in all the units that matches the unique identifier of the currently arriving data item? field.

[0121] S444: If it exists, update the burst flow frequency in the current monitoring period according to the data item frequency.

[0122] In this embodiment, if a matching ID is found, it means that the arrival data item is already being tracked, and its ID is directly updated. The field value is , accumulated according to the frequency carried by the currently arriving data item.

[0123] S445. If it does not exist and there is an empty cell in the hash table entry, update the information of the currently arrived data item in the empty cell.

[0124] In this embodiment, there is an empty unit, that is, an unused unit. In this case, the current arrival data item and its frequency Update to hash table entry In the empty cell of The ID field is set to ,Will of The field is set to .

[0125] S446: If the current cell does not exist and there is no empty cell in the hash table entry, select a cell from the hash table entry that meets the preset replacement criteria for replacement, replace the current cell information with the information of the currently arriving data item, and mark the corresponding data flow as a suspicious burst flow being tracked.

[0126] In this embodiment, the replacement criterion is set to the unit that is not a suspicious burst flow currently being tracked and has the smallest burst flow frequency in the current monitoring period.

[0127] The steps of replacement and update are as follows: S4461, first from the hash table entry Find the fields that meet the requirements is not 1 and has the minimum The unit of the field.

[0128] If it is not 1, it means that it is not a suspicious burst flow currently being tracked. Field Description The minimum burst flow frequency in the current monitoring period.

[0129] S4462, calculate the probability of elimination, elimination probability The calculation formula is,

[0130] ;

[0131] in, Represents a hash table entry Has the smallest The unit of the field.

[0132] By eliminating the probability Will The element stored in is replaced by the currently arrived data item.

[0133] S4463, when the replacement is successful, Has the smallest Field and The ID field of the unit is updated to ,Will The fields are updated to , include others 、 、 Update the fields to 0, and finally The field is updated to 1, marking it as a suspicious burst flow being tracked.

[0134] S4464, using the preset element-candidate table entry mapping function Calculate the data items that are removed from the cell where the replacement occurs Candidate bucket positions ,

[0135] The element-candidate entry mapping function is,

[0136] ;

[0137] The specific calculation method is: .

[0138] in, is an independent and uniform hash function, is the unique identifier of the data item to be removed, represents the remainder operator, For hash tables length.

[0139] S4465. Determine whether the number of times the removed data item has been removed has reached the set maximum removal threshold. If not, repeat the suspicious burst flow tracking operations of steps S441-S446 above until the removed data item is saved in a hash table entry or reaches the maximum removal threshold.

[0140] If the number of times the data item is removed reaches the maximum removal threshold, the bit array The subscript is The bit is set to 0, indicating It is no longer considered part of the suspicious burst flow and the replacement update is completed.

[0141] In the above steps, if there is no available empty cell, then according to the probability of elimination Select a unit with the lowest frequency that is not a suspicious burst flow being tracked to replace it. After the replacement is successful, update the information of the unit to the current information of the arriving data item and mark it as a suspicious burst flow being tracked. , using the element-candidate table entry mapping function Calculate its candidate hash table entries , and try to insert it into another hash table entry. If multiple attempts fail or the maximum number of culling is reached, Remove from tracking of suspicious burst flows.

[0142] The collision problem in the hash table is effectively resolved through an empty cell check and replacement strategy. When an empty cell is available, the new data item is directly inserted; when an empty cell is not available, a probability-based replacement strategy is adopted to balance storage space and the importance of the data item.

[0143] S45: Summarize all the tracked suspicious burst flows and the contents of the corresponding hash table entries to obtain a suspicious burst flow tracking table. In this embodiment, the contents of the hash table entry include the contents of all fields of each unit in the hash table entry.

[0144] S46. If the result of the judgment is that the arriving data item belongs to a low-frequency element, the flow count and filtering part is entered, such as Figure 5 As shown, the specific steps include:

[0145] S461, mapping the currently arriving data item to the counter array according to the preset fourth mapping function and an array of data item IDs The In the unit;

[0146] In this embodiment, the fourth mapping function It is the element-counter mapping function.

[0147] The element-counter mapping function is,

[0148] ;

[0149] in, is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the remainder operator, For the counter array and an array of data item IDs length.

[0150] ;

[0151] in, is the calculated fourth mapping position, that is, the subscript position of each arriving data item in the counter array and the data item ID array.

[0152] S463: Determine whether the data item ID at the fourth mapping position matches the unique identifier of the currently arrived data item, and determine different update operations based on the matching result.

[0153] S4631: If there is a match, increment the counter value at the fourth mapping position by one.

[0154] In this embodiment, if there is a match, then , which means that the hash value of the current data item is the same as the hash value of a previous data item, and they do represent the same data item, at which point the counter is updated The value of .

[0155] S4632: If there is no match and the fourth mapping position is empty, update the data item ID of the fourth mapping position according to the information of the currently arrived data item and initialize the counter value to 1.

[0156] In this embodiment, if there is no match and the fourth mapping position is empty, this means that the unique identifier of any data item has not been stored at the fourth mapping position, that is, the position is empty.

[0157] When a new arrival data item is encountered, if its hash value is mapped to the fourth mapping position, and the fourth mapping position is empty, the unique identifier of the arrival data item is set to Stored in and associate the counter with that position Initialized to 1, indicating that the arriving data item has been encountered once.

[0158] The specific operation in this embodiment is to first determine Is the location empty? If so, update the settings. is 1, and Set to .

[0159] S4633: If there is no match and the fourth mapping position is not empty, perform a subtraction operation on the counter value.

[0160] In this embodiment, if there is no match and the fourth mapping position is not empty, it means that a hash conflict has occurred, that is, two different data items are hashed to the same position.

[0161] The specific operation of this embodiment is as follows: if there is no match, and the judgment If it is not empty, the counter The value of is reduced by one;

[0162] After performing the above operations, the counter is judged Is the value equal to 0? , clear the data item ID of the fourth mapping position, set Is empty, then end this part of the operation; if If it is not 0, the operation ends immediately.

[0163] S464: Monitor the counter value of step S4631 to determine whether the counter value exceeds the preset low flow threshold. .

[0164] When the counter value exceeds the low flow threshold, that is, in this embodiment, When, update The value is 1, the arriving data item is marked as a suspicious burst flow and inserted into the suspicious burst flow part for tracking, and the arriving data item is marked as a suspicious burst flow and inserted into the suspicious burst flow part for tracking. and Send to the suspicious burst flow tracking part;

[0165] When the counter value does not exceed the low flow threshold, the low flow counting and filtering operations are completed.

[0166] The above process uses hash mapping and counter updates to efficiently distinguish low-frequency data flows (small flows) from high-frequency data flows (suspicious burst flows), and filters small flows. When the counter value exceeds the set small flow threshold, the flow is reported as a suspicious burst flow for suspicious burst flow tracking; otherwise, the data flow is blocked from entering the subsequent part to optimize the overall storage space and reduce the computational cost for subsequent burst flow detection.

[0167] S5, according to the suspicious burst flow tracking table, enter the suspicious burst flow detection part, traverse the obtained suspicious burst flow tracking table at the end of each monitoring cycle, determine whether each suspicious burst flow is a burst data flow according to the preset burst data judgment mechanism, summarize and output the burst data flow list, such as Figure 6 As shown, the specific steps include:

[0168] S51 . After each monitoring cycle ends, the counter array and data item ID array of each bucket in the bucket array are cleared.

[0169] In this embodiment, after each monitoring cycle ends, the bucket array is traversed. For each bucket, count the small flows in the bucket and filter the Contains an array of counters and an array of data item IDs Each cell in is reset to 0.

[0170] S52, traverse each unit of each hash table entry in the suspicious burst flow tracking table, each unit includes the burst flow frequency in the current monitoring period , the frequency of burst flows in the last monitoring period , timestamp of the suspicious burst flow and duration after the onset ;

[0171] In this embodiment, each unit in each hash table entry in the suspicious burst flow tracking part in each bucket is traversed, and the ID, 、 、 and field.

[0172] S53, based on the frequency of burst flows in the current monitoring period and the frequency of burst flows in the last monitoring period Obtain the frequency changes of adjacent cycles.

[0173] In this embodiment, the frequency variation of adjacent cycles is: .

[0174] S54, combined with the timestamp of the suspicious burst flow and preset burst thresholds Determine whether each suspicious burst flow has a burst, based on the duration after the burst Determine whether each suspicious burst flow is a burst data flow.

[0175] S541: In this embodiment, first determine the field in the read unit Is it 0?

[0176] S542, when , determine whether the frequency change of adjacent cycles in the read unit is not less than the burst threshold .

[0177] if ,and , it indicates that a burst flow is detected, and the current unit read is updated. For the current monitoring cycle, record the time when the burst starts; and update the current unit read for , save the current frequency as the frequency of the previous monitoring period; the last update To 0, prepare for the counting of the next monitoring cycle;

[0178] if ,and , indicating that the frequency has not increased significantly, then directly update the current unit read for , save the current frequency; and update It is 0, preparing for the counting of the next monitoring cycle.

[0179] S543, when , determine whether the frequency change of adjacent cycles in the read unit is not less than the burst threshold The reciprocal of .

[0180] if ,and , read the current unit The field value increases by one, indicating that the flow has burst for another monitoring period. The fields are , The field is 0;

[0181] if ,and , then directly output the data stream with the ID field in the current unit read as the data item ID as a -Burst data flow and set all fields in the current unit to 0.

[0182] S544: Determine whether there are any table entry units that have not been traversed. If so, repeat the operations of steps S52 to S54. If not, complete the detection of the burst data flow in the current monitoring cycle.

[0183] S55 , summarizing all the burst data flows determined in the above steps to obtain a burst data flow list and outputting the list.

[0184] From the overall structure of this application, this application designs a strong real-time, low-overhead burst flow real-time monitoring technology, which introduces Sketch technology into the real-time processing of high-speed massive data streams: the flow data is mapped to a limited storage space through a hash function, thereby achieving efficient flow measurement and anomaly detection; by comparing and The system uses the ratio of the current and previous monitoring cycles to determine the presence of a large flow burst. It infers overall traffic trends based on limited sample data (i.e., counts from the current and previous monitoring cycles). This demonstrates Sketch's ability to provide approximate calculations of traffic data within a limited space. This reduces the high-speed computing and cache overhead required for real-time monitoring of burst traffic, significantly improving the accuracy of burst detection while matching high-speed data flow rates, thus achieving true real-time, high-precision burst traffic monitoring.

[0185] Based on the same inventive concept as above, the embodiment of the present application also discloses a burst traffic monitoring device for real-time analysis of high-speed data streams, the structure of which is as follows: Figure 7 As shown, the device includes the following modules:

[0186] A burst traffic monitoring device for real-time analysis of high-speed data streams, including the following modules:

[0187] A data stream receiving module is used to receive data streams arriving in the current monitoring period, each data stream includes multiple arriving data items, and all arriving data items in the same data stream have the same unique identifier;

[0188] A data item mapping module is used to allocate the corresponding arriving data items to a pre-built storage bucket array based on the hash map and the unique identifier of each arriving data item, and determine the data item ID of each arriving data item in the storage bucket array;

[0189] A suspicious burst flow tracking module is used to determine the frequency of an arriving data item based on the data item ID. When the data flow corresponding to the arriving data item is determined to be a suspicious burst flow based on the result of the frequency determination, the module tracks the suspicious burst flow and summarizes it to obtain a suspicious burst flow tracking table.

[0190] The burst flow detection module is used to traverse the obtained suspicious burst flow tracking table at the end of each monitoring cycle, determine whether each suspicious burst flow is a burst data flow according to the preset burst data judgment mechanism, summarize and output the burst data flow list.

[0191] In a specific embodiment, the device further includes a monitoring structure initialization module.

[0192] A monitoring structure initialization module is used to build an overall monitoring structure. The overall monitoring structure includes a bucket array. Each bucket in the bucket array contains a suspicious traffic identification part, a small flow counting and filtering part, and a suspicious burst flow tracking part.

[0193] The suspicious traffic identification part includes a bit array, the small flow counting and filtering part includes a counter array and a data item ID array, and the suspicious burst flow tracking part includes a hash table;

[0194] Initialize the entire monitoring structure, initialize all bits in the bit array, each unit in the counter array and data item ID array, and all fields in the hash table to 0.

[0195] In a specific implementation scheme, the data item mapping module includes the following units:

[0196] A first data item mapping unit is configured to map the arriving data item to the bucket array according to a preconfigured first mapping function and a unique identifier of the arriving data item, to obtain a first mapping position of the arriving data item;

[0197] The first mapping function is, ;in, is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the remainder operator, The length of the bucket array.

[0198] The second data item mapping unit is used to calculate the first mapping position of each arriving data item in the bucket array, specifically, ;

[0199] in, The first mapping position obtained by calculation is the position of the data item in the bucket array.

[0200] In a specific implementation scheme, the suspicious burst flow tracking module includes the following units:

[0201] a first suspicious burst flow tracking unit, configured to map each arriving data item arriving at the storage bucket to a bit array according to a preconfigured second mapping function, and calculate a second mapping position of each arriving data item in the bit array in combination with the first mapping position;

[0202] The second mapping function is, ;

[0203] in, is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the remainder operator, is the length of the bit array;

[0204] ;

[0205] in, is the calculated second mapping position, that is, the position of each arriving data item in the bit array.

[0206] The second suspicious burst flow tracking unit is used to determine the frequency of the arriving data items and obtain an evaluation result, specifically,

[0207] Determine whether the data item ID of the second mapping position meets the preset low-frequency judgment rule. If so, the judgment result is that the arriving data item belongs to a low-frequency element; if not, the judgment result is that the arriving data item belongs to a high-frequency element.

[0208] The third suspicious burst flow tracking unit is configured to mark and track the current suspicious burst flow if the arriving data item belongs to a high-frequency element and the data flow corresponding to the arriving data item is a suspicious burst flow.

[0209] In a specific implementation scheme, the third suspicious burst flow tracking unit includes the following subunits:

[0210] The first suspicious burst flow tracking subunit is used to map the current arrival data item and the corresponding data item frequency to the hash table item of the hash table according to the preset third mapping function, and obtain the third mapping position of the arrival data item. Each hash table item includes multiple units, each unit includes field, the frequency of burst flows in the current monitoring period, and the identifier of suspicious burst flows.

[0211] The second suspicious burst flow tracking subunit is used to determine whether there is a unique identifier that matches the current arriving data item in the unit of the hash table item corresponding to the third mapping position. Field, if it exists, updates the burst flow frequency in the current monitoring period based on the frequency of the data item;

[0212] If it does not exist and there is an empty cell in the hash table entry, the information of the currently arrived data item is updated in the empty cell.

[0213] If it does not exist and there is no empty unit in the hash table entry, select a unit that meets the preset replacement criteria from the hash table entry for replacement, replace the information of the current unit with the information of the currently arriving data item, and mark the corresponding data flow as a suspicious burst flow being tracked.

[0214] The third suspicious burst flow tracking subunit is configured to aggregate all tracked suspicious burst flows and the contents of corresponding hash table entries to obtain a suspicious burst flow tracking table.

[0215] In a specific implementation scheme, the suspicious burst flow tracking module further includes the following units:

[0216] The fourth suspicious burst flow tracking unit is configured to map the currently arriving data item to the counter array and the data item ID array according to a preset fourth mapping function, and obtain a fourth mapping position of the arriving data item.

[0217] a fifth suspicious burst flow tracking unit, configured to determine whether the data item ID at the fourth mapping position matches the unique identifier of the currently arriving data item, and if so, increment the counter value at the fourth mapping position by one;

[0218] If there is no match and the fourth mapping position is empty, the data item ID of the fourth mapping position is updated according to the information of the currently arrived data item, and the counter value is initialized to 1;

[0219] If there is no match and the fourth mapping location is not empty, a decrement operation is performed on the counter value.

[0220] The sixth suspicious burst flow tracking unit is used to monitor the counter value. When the counter value reaches a preset low flow threshold, the arriving data item is marked as a suspicious burst flow and tracked. When the counter value is 0, the data item ID at the fourth mapping position is cleared.

[0221] In a specific implementation scheme, the burst flow detection module includes the following units:

[0222] The first burst flow detection unit is configured to clear the counter array and the data item ID array of each bucket in the bucket array after each monitoring period ends.

[0223] The second burst flow detection unit is used to traverse each unit of each hash table entry in the suspicious burst flow tracking table, each unit including the burst flow frequency in the current monitoring cycle, the burst flow frequency in the previous monitoring cycle, the timestamp of becoming a suspicious burst flow, and the duration after the burst.

[0224] The third burst flow detection unit is used to obtain the frequency change of adjacent cycles based on the burst flow frequency in the current monitoring cycle and the burst flow frequency in the previous monitoring cycle, and to determine whether each suspicious burst flow has a burst based on the timestamp of the suspicious burst flow and the preset burst threshold, and to determine whether each suspicious burst flow is a burst data flow based on the duration after the burst.

[0225] The fourth burst flow detection unit is configured to aggregate all the determined burst data flows to obtain a burst data flow list and output the list.

[0226] From the above function introduction, it can be seen that the burst traffic monitoring device in this application for real-time analysis of high-speed data streams has built a burst traffic monitoring system for high-speed data streams, improved the intelligence and automation level of traffic monitoring, improved the detection accuracy of sudden large flows, and promoted the sustained, rapid and healthy development of the economy and society.

[0227] Based on the same inventive concept mentioned above, an embodiment of the present application also discloses a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, at least one program, code set or instruction set can be loaded and executed by a processor to implement the burst traffic monitoring method for real-time analysis of high-speed data streams provided by the above method embodiment.

[0228] Also based on the same inventive concept mentioned above, an embodiment of the present application also discloses a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the burst traffic monitoring method for real-time analysis of high-speed data streams as described above.

[0229] Those skilled in the art will appreciate that all or part of the steps of implementing the above embodiments may be accomplished by hardware, or may be accomplished by a program instructing the relevant hardware. The program may be stored in the computer-readable storage medium, and the computer-readable storage medium may include, for example, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0230] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A burst traffic monitoring method for real-time analysis of high-speed data streams, characterized in that: The steps include: Constructing an overall monitoring structure, the overall monitoring structure including a bucket array, each bucket in the bucket array including a suspicious traffic identification portion, the suspicious traffic identification portion including a bit array; receiving a data stream arriving in a current monitoring period, each data stream including a plurality of arriving data items, and all the arriving data items in the same data stream having the same unique identifier; Mapping the arriving data item to the bucket array according to a preconfigured first mapping function and the unique identifier of the arriving data item, obtaining a first mapping position of the arriving data item, and determining a data item ID of each arriving data item in the bucket array; Mapping each of the arriving data items arriving at the storage bucket into the bit array according to a preconfigured second mapping function, and calculating a second mapping position of each of the arriving data items in the bit array in combination with the first mapping position, wherein the first mapping function and the second mapping function are independent and uniform hash functions; Performing a frequency determination on the arriving data item based on the data item ID and obtaining a determination result, determining whether the data item ID at the second mapping position satisfies a preset low-frequency determination rule; if not, determining that the arriving data item is a high-frequency element and the data flow corresponding to the arriving data item is a suspicious burst flow; When it is determined that the data flow corresponding to the arriving data item is a suspicious burst flow in combination with the result of the frequency determination, the suspicious burst flow is tracked and summarized to obtain a suspicious burst flow tracking table; At the end of each monitoring cycle, the obtained suspicious burst flow tracking table is traversed, and whether each suspicious burst flow is a burst data flow is determined according to a preset burst data judgment mechanism, and a burst data flow list is obtained and output.

2. The method for monitoring burst traffic for real-time analysis of high-speed data streams according to claim 1, characterized in that: Each bucket in the bucket array also includes a small flow counting and filtering part and a suspicious burst flow tracking part; The small flow counting and filtering part includes a counter array and a data item ID array, and the suspicious burst flow tracking part includes a hash table; Before receiving the data stream arriving in the current monitoring period, the overall monitoring structure is initialized, and all bits in the bit array, each unit in the counter array and the data item ID array, and all fields in the hash table are initialized to 0.

3. The method for monitoring burst traffic for real-time analysis of high-speed data streams according to claim 1, characterized in that: The first mapping function is, ; in, is an independent and uniform hash function, is the unique identifier of the arrival data item, represents the remainder operator, is the length of the bucket array; Calculate the first mapping position of each of the arriving data items in the bucket array, specifically, ; in, is the calculated first mapping position, that is, the position of the arriving data item in the bucket array.

4. The method for monitoring burst traffic for real-time analysis of high-speed data streams according to claim 1, characterized in that: The second mapping function is, ； in, is an independent and uniform hash function, is the unique identifier of the arrival data item, represents the remainder operator, is the length of the bit array; ； in, is the calculated second mapping position, that is, the position of each of the arriving data items in the bit array.

5. The method for monitoring burst traffic for real-time analysis of high-speed data streams according to claim 2, characterized in that: When it is determined that the data flow corresponding to the arriving data item is a suspicious burst flow based on the result of the frequency determination, tracking the suspicious burst flow and summarizing it to obtain a suspicious burst flow tracking table specifically includes the following steps: If the arriving data item belongs to a high-frequency element, the data flow corresponding to the arriving data item is a suspicious burst flow, and the current suspicious burst flow is marked and tracked, specifically, According to a preset third mapping function, the current arrival data item and the corresponding data item frequency are mapped to the hash table item of the hash table to obtain the third mapping position of the arrival data item, each of the hash table items includes a plurality of units, each of the units includes field, the frequency of burst flows in the current monitoring period, and the identifier of suspicious burst flows; Determine whether there is a unique identifier that matches the current arrival data item in the unit of the hash table item corresponding to the third mapping position. field, if present, updating the burst flow frequency in the current monitoring period according to the frequency of the data item; If it does not exist and there is an empty unit in the hash table entry, then the information of the current arrival data item is updated in the empty unit. If it does not exist and there is no empty unit in the hash table entry, select a unit that meets the preset replacement criteria from the hash table entry for replacement, replace the information of the current unit with the information of the current arriving data item, and mark the corresponding data flow as a suspicious burst flow being tracked; All tracked suspicious burst flows and the corresponding contents of the hash table entries are summarized to obtain a suspicious burst flow tracking table.

6. The method for monitoring burst traffic for real-time analysis of high-speed data streams according to claim 2, characterized in that: If the determination result is that the arriving data item is a low-frequency element, the following steps are specifically included: Mapping the currently arrived data item to the counter array and the data item ID array according to a preset fourth mapping function to obtain a fourth mapping position of the arrived data item; Determine whether the data item ID of the fourth mapping position matches the unique identifier of the currently arrived data item, and if so, increment the counter value of the fourth mapping position by one; If there is no match and the fourth mapping position is empty, updating the data item ID of the fourth mapping position according to the information of the currently arrived data item and initializing the counter value to 1; If there is no match, and the fourth mapping location is not empty, performing a subtraction operation on the counter value; The counter value is monitored, and when the counter value reaches a preset small flow threshold, the arriving data item is marked as a suspicious burst flow and tracked; when the counter value is 0, the data item ID of the fourth mapping position is cleared.

7. The method for monitoring burst traffic for real-time analysis of high-speed data streams according to claim 2, characterized in that: The method of traversing the obtained suspicious burst flow tracking table at the end of each monitoring cycle, determining whether each suspicious burst flow is a burst data flow according to a preset burst data judgment mechanism, summarizing and outputting a burst data flow list specifically includes the following steps: After each monitoring cycle ends, the counter array and the data item ID array of each bucket in the bucket array are cleared; Traversing each unit of each hash table entry in the suspicious burst flow tracking table, each unit includes the burst flow frequency in the current monitoring period, the burst flow frequency in the previous monitoring period, the timestamp of becoming a suspicious burst flow, and the duration after the burst; Obtaining frequency changes in adjacent cycles based on the burst flow frequency in the current monitoring cycle and the burst flow frequency in the previous monitoring cycle, determining whether each of the suspicious burst flows has a burst based on the timestamp of the suspected burst flow and a preset burst threshold, and determining whether each of the suspicious burst flows is a burst data flow based on the duration after the burst; All the determined burst data flows are summarized to obtain a burst data flow list and output the list.

8. A burst traffic monitoring device for real-time analysis of high-speed data streams, characterized in that: Includes the following modules: A monitoring structure initialization module, configured to construct an overall monitoring structure, wherein the overall monitoring structure includes a bucket array, each bucket in the bucket array includes a suspicious traffic identification portion, and the suspicious traffic identification portion includes a bit array; A data stream receiving module, configured to receive data streams arriving within a current monitoring period, wherein each data stream includes a plurality of arriving data items, and all arriving data items in the same data stream have the same unique identifier; a data item mapping module, configured to map the arriving data item to the bucket array according to a preconfigured first mapping function and the unique identifier of the arriving data item, obtain a first mapping position of the arriving data item, and determine a data item ID of each arriving data item in the bucket array; A suspicious burst flow tracking module is configured to map each of the arriving data items arriving at the storage bucket to the bit array according to a preconfigured second mapping function, and calculate the second mapping position of each of the arriving data items in the bit array in combination with the first mapping position, wherein the first mapping function and the second mapping function are independent and uniform hash functions; perform frequency determination on the arriving data item according to the data item ID and obtain a determination result, and determine whether the data item ID at the second mapping position satisfies a preset low-frequency determination rule; if not, the determination result is that the arriving data item belongs to a high-frequency element, and the data flow corresponding to the arriving data item is a suspicious burst flow; when it is determined that the data flow corresponding to the arriving data item is a suspicious burst flow in combination with the result of the frequency determination, the suspicious burst flow is tracked and summarized to obtain a suspicious burst flow tracking table; The burst flow detection module is used to traverse the obtained suspicious burst flow tracking table at the end of each monitoring cycle, determine whether each suspicious burst flow is a burst data flow according to a preset burst data judgment mechanism, summarize and output a burst data flow list.

9. An intelligent terminal, characterized in that: It includes a memory and a processor, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the burst traffic monitoring method for real-time analysis of high-speed data streams as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the burst traffic monitoring method for real-time analysis of high-speed data streams as described in any one of claims 1 to 7.

Images