Burst traffic monitoring method and device for real-time analysis of high-speed data stream, terminal and storage medium

Abstract

The invention discloses a burst traffic monitoring method and device for real-time analysis of high-speed data streams, a terminal and a storage medium, and the method comprises the steps: receiving data streams arriving in a current monitoring period, and each data stream comprises a plurality of arriving data items; according to the Hash mapping and the unique identifier of each arriving data item, distributing the arriving data item to a pre-constructed storage bucket array, and determining a data item ID of each arriving data item in the storage bucket array; performing frequency judgment on the arriving data item according to the data item ID, and if the data stream corresponding to the arriving data item is a suspicious burst stream, tracking and summarizing the suspicious burst stream to obtain a suspicious burst stream tracking table; and traversing the suspicious burst stream tracking table, determining whether each suspicious burst stream is a burst data stream according to a preset burst data judgment mechanism, summarizing to obtain a burst data stream list, and outputting the burst data stream list. According to the method provided by the invention, the calculation overhead and the memory consumption can be effectively reduced, and strong real-time and high-precision burst traffic monitoring is realized.

Description

Technical Field

[0001] This application relates to the field of traffic monitoring of e-commerce platforms, and particularly to a burst traffic monitoring method, device, terminal, and storage medium for real-time analysis of high-speed data streams. Background Art

[0002] In recent years, the marketing models of e-commerce platforms have become increasingly complex, gradually evolving from traditional static product introductions to a multi-dimensional competition model including various forms such as live streaming with goods, short video advertising placement, and advertising bidding for traffic. This transformation has made it necessary for e-commerce platforms to conduct fine-grained traffic measurement and analysis on high-speed data streams in order to evaluate the effects of different promotion methods in real time, so as to select the most suitable marketing strategy for each product. In this process, traffic analysis plays a crucial role in enhancing product attention and conversion rate. Especially when the budget is limited, how to maximize the drainage of products has become an urgent problem to be solved.

[0003] Currently, for the analysis of massive high-speed data streams on e-commerce platforms, although it can meet the needs of e-commerce platforms for data stream analysis to a certain extent, it exposes obvious deficiencies when facing burst traffic. For example, existing e-commerce platforms have a large number of products, marketing accounts, and customer groups, and the browsing and transaction data generated in real time is also massive and high-speed. There are challenges such as overly high computational complexity and excessive memory consumption during runtime for conducting correlation analysis on it. Even with the use of dedicated large servers in existing technologies, it is difficult to achieve real-time monitoring of burst traffic, resulting in overly long detection delays and the inability to timely capture network hotspots and burst traffic changes. In addition, these methods often rely on high-performance servers, with problems such as excessive memory consumption and overly high computational complexity during runtime, and it is difficult to balance cost-effectiveness while ensuring real-time performance. Therefore, how to design a technical solution that can effectively reduce computational overhead and memory consumption while achieving strong real-time and high-precision burst traffic monitoring has become a key problem that needs to be solved urgently. Summary of the Invention

[0004] In order to accurately and real-time monitor burst high-speed traffic, this application provides a burst traffic monitoring method, device, terminal, and storage medium for real-time analysis of high-speed data streams.

[0005] In a first aspect, this application provides a burst traffic monitoring method for real-time analysis of high-speed data streams, adopting the following technical solution: A burst traffic monitoring method for real-time analysis of high-speed data streams includes the following steps: Receive the data stream arriving within the current monitoring period. Each data stream includes multiple arriving data items, and all the arriving data items in the same data stream have the same unique identifier; Assign the corresponding arriving data item to a pre-constructed array of buckets according to the hash mapping and the unique identifier of each said arriving data item, and determine the data item ID of each said arriving data item in the array of buckets; Determine the frequency of the arriving data item according to the data item ID. When it is determined that the data stream corresponding to the arriving data item is a suspicious burst stream in combination with the result of the frequency determination, track the suspicious burst stream and summarize to obtain a suspicious burst stream tracking table; Traverse the obtained suspicious burst stream tracking table at the end of each said monitoring period, and determine whether each said suspicious burst stream is a burst data stream according to a preset burst data judgment mechanism, summarize to obtain a list of burst data streams and output.

[0006] By adopting the above technical solution, the real-time analysis of traffic and the determination of suspicious burst streams are realized by combining hash mapping and data item ID, the efficient processing and distribution of data streams are realized, and the detection ability of abnormal traffic is improved; at the same time, by tracking the suspicious burst streams, the potential traffic anomalies are comprehensively grasped, and the accurate judgment and timely response of burst streams are realized.

[0007] Preferably, before receiving the data stream arriving in the current monitoring period, the following steps are further included: Construct an overall monitoring structure, the overall monitoring structure includes an array of buckets, and each bucket in the array of buckets contains a suspicious traffic identification part, a small flow count and filtering part, and a suspicious burst stream tracking part; The suspicious traffic identification part includes a bit array, the small flow count and filtering part includes a counter array and a data item ID array, and the suspicious burst stream tracking part includes a hash table; Initialize the overall monitoring structure, and initialize all bits in the bit array, each unit in the counter array and the data item ID array, and all fields in the hash table to 0.

[0008] By adopting the above technical solution, an overall monitoring structure is constructed, and by initializing the bit array, the counter array, the data item ID array and the hash table therein, a basis is provided for the subsequent mapping, frequency determination and tracking of data items, and misjudgment caused by unclear initial values is avoided.

[0009] Preferably, the step of assigning the corresponding arriving data item to a pre-constructed array of buckets according to the hash mapping and the unique identifier of each said arriving data item, and determining the data item ID of each said arriving data item in the array of buckets specifically includes the following steps: Map the arriving data item to the bucket array according to a pre-configured first mapping function and the unique identifier of the arriving data item to obtain the first mapping position of the arriving data item; The first mapping function is ; where is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the modulo operator, is the length of the bucket array; Calculate the first mapping position of each arriving data item in the bucket array. Specifically, ; where is the calculated first mapping position, that is, the position of the arriving data item in the bucket array.

[0010] By adopting the above technical solution, the first mapping position is determined based on the modulo operator, which simplifies the calculation process, reduces the algorithm complexity, is applicable to real-time processing in high-speed data stream scenarios, accurately maps the arriving data item to a specific position in the bucket array, and ensures the efficiency and accuracy of data stream processing.

[0011] Preferably, the frequency determination of the arriving data item according to the data item ID specifically includes the following steps: According to a pre-configured second mapping function, map each arriving data item arriving at the bucket to the bit array, and calculate the second mapping position of each arriving data item in the bit array in combination with the first mapping position; The second mapping function is ; where is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the modulo operator, is the length of the bit array; ; where is the calculated second mapping position, that is, the position of each arriving data item in the bit array; Perform frequency determination on the arriving data item and obtain the evaluation result. Specifically, Determine whether the data item ID at the second mapping position meets a preset low-frequency determination rule. If it meets, the determination result is that the arriving data item belongs to a low-frequency element; if it does not meet, the determination result is that the arriving data item belongs to a high-frequency element.

[0012] By adopting the above technical solution, the arriving data item is mapped to the bit array through a pre-configured second mapping function, the second mapping position is calculated in combination with the first mapping position, the position of the arriving data item in the bit array is calculated, and the data item ID at the second mapping position is judged according to the low-frequency determination rule, so as to effectively judge the frequency of the arriving data item, thereby providing an accurate basis for subsequent identification of suspicious burst streams.

[0013] Preferably, when it is determined that the data stream corresponding to the arriving data item is a suspicious burst stream in combination with the result of the frequency determination, the suspicious burst stream is tracked and summarized to obtain a suspicious burst stream tracking table, which specifically includes the following steps: If the arriving data item belongs to a high-frequency element, the data stream corresponding to the arriving data item is a suspicious burst stream, and the current suspicious burst stream is marked and tracked. Specifically, According to a preset third mapping function, the current arriving data item and the corresponding data item frequency are mapped to the hash table entries of the hash table to obtain the third mapping position of the arriving data item. Each hash table entry contains multiple units, and each unit contains fields, the burst stream frequency within the current monitoring period, and a suspicious burst stream identifier; Judge whether there is a field in the unit of the hash table entry corresponding to the third mapping position that matches the unique identifier of the current arriving data item. If it exists, update the burst stream frequency within the current monitoring period according to the data item frequency. If it does not exist and there is an empty unit in the hash table entry, update the information of the current arriving data item in the empty unit. If it does not exist and there is no empty unit in the hash table entry, select a unit that meets the preset replacement standard from the hash table entry for replacement, replace and update the information of the current unit with the information of the current arriving data item, and mark the corresponding data stream as a suspicious burst stream being tracked. Summarize all the tracked suspicious burst streams and the content of the corresponding hash table entries to obtain a suspicious burst stream tracking table.

[0014] By adopting the above technical solution, the arriving data items are mapped to the hash table entries of the hash table, and the third mapping function is used to determine their positions, which improves the efficiency and accuracy of data processing; the units in the hash table entries are judged and updated, ensuring that the information of high-frequency data items can be recorded in time and avoiding duplicate storage; when there is no empty unit in the hash table entry, a replacement mechanism is adopted to ensure the effective utilization rate of the hash table; it can accurately mark and track the data stream corresponding to high-frequency elements, providing reliable data support for subsequent traffic analysis and decision-making.

[0015] Preferably, if the determination result is that the arriving data item belongs to a low-frequency element, the following specific steps are included: Map the current arriving data item to the counter array and the data item ID array according to a preset fourth mapping function to obtain the fourth mapping position of the arriving data item; Judge whether the data item ID at the fourth mapping position matches the unique identifier of the current arriving data item. If it matches, perform an increment assignment operation on the counter value at the fourth mapping position; If it does not match and the fourth mapping position is empty, update the data item ID at the fourth mapping position according to the information of the current arriving data item and initialize the counter value to 1; If it does not match and the fourth mapping position is not empty, perform a decrement operation on the counter value; Monitor the counter value. When the counter value reaches a preset small flow threshold, mark the arriving data item as a suspicious burst flow and track it. When the counter value is 0, clear the data item ID at the fourth mapping position.

[0016] By adopting the above technical solution, the arriving data items are mapped to the counter array and the data item ID array through the fourth mapping function, and then the counter value is dynamically maintained by judging whether the data item IDs match, truly reflecting the occurrence frequency of the data items. When the counter value reaches the small flow threshold, the suspicious burst flow is marked and tracked in time, effectively identifying potential suspicious burst flows and improving the sensitivity of monitoring.

[0017] Preferably, at the end of each monitoring period, traverse the obtained suspicious burst flow tracking table, determine whether each suspicious burst flow is a burst data stream according to a preset burst data judgment mechanism, summarize to obtain a burst data stream list and output it, and the specific steps are as follows: When each monitoring period ends, empty the counter array and the data item ID array of each storage bucket in the storage bucket array; Traverse each cell of each hash table entry in the suspicious burst flow tracking table. Each cell includes the burst flow frequency in the current monitoring period, the burst flow frequency in the previous monitoring period, the timestamp when it becomes a suspicious burst flow, and the duration after the burst. Based on the burst flow frequency in the current monitoring period and the burst flow frequency in the previous monitoring period, obtain the change in frequency between adjacent periods. Combine the timestamp when it becomes a suspicious burst flow and a preset burst threshold to determine whether each of the suspicious burst flows has a burst. According to the duration after the burst, judge whether each of the suspicious burst flows is a burst data stream. Summarize all the burst data streams obtained from the judgment to obtain a burst data stream list and output it.

[0018] By adopting the above technical solution, traverse the hash table entry cells in the suspicious burst flow tracking table, and extract the key information for subsequent burst flow judgment. Based on the change in frequency between adjacent periods, combine the timestamp when it becomes a suspicious burst flow and a preset burst threshold to accurately judge whether each suspicious burst flow has a burst, improving the sensitivity and reliability of detection. According to the duration after the burst, further judge whether the suspicious burst flow is a real burst data stream, filtering out false burst flows with short-term fluctuations, enhancing the accuracy of the monitoring results, and realizing the efficient real-time monitoring of burst traffic in high-speed data streams.

[0019] In a second aspect, the present application provides a burst traffic monitoring device for real-time analysis of high-speed data streams, adopting the following technical solution: A burst traffic monitoring device for real-time analysis of high-speed data streams includes the following modules: A data stream receiving module, configured to receive the data streams arriving in the current monitoring period. Each of the data streams includes a plurality of arriving data items, and all the arriving data items in the same data stream have the same unique identifier. A data item mapping module, configured to allocate the corresponding arriving data item to a pre-constructed storage bucket array according to hash mapping and the unique identifier of each arriving data item, and determine the data item ID of each arriving data item in the storage bucket array. A suspicious burst flow tracking module, configured to perform frequency determination on the arriving data items according to the data item ID. When it is determined that the data stream corresponding to the arriving data item is a suspicious burst flow in combination with the result of the frequency determination, track the suspicious burst flow and summarize it to obtain a suspicious burst flow tracking table. A burst flow detection module, configured to traverse the obtained suspicious burst flow tracking table at the end of each monitoring period, determine whether each of the suspicious burst flows is a burst data stream according to a preset burst data judgment mechanism, summarize it to obtain a burst data stream list and output it.

[0020] By adopting the above technical solution, the high-speed computing overhead and cache overhead required for real-time monitoring of burst traffic are reduced. On the basis of matching the high-speed data flow rate, the detection accuracy of large burst flows is greatly improved, thus realizing truly strong real-time and high-precision monitoring of large burst flows.

[0021] In a third aspect, the present application provides an intelligent terminal, adopting the following technical solution: An intelligent terminal includes a memory and a processor. At least one program is stored in the memory, and the at least one program is loaded and executed by the processor to implement the burst traffic monitoring method for real-time analysis of high-speed data streams as described above.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium, adopting the following technical solution: A computer-readable storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the burst traffic monitoring method for real-time analysis of high-speed data streams as described above.

[0023] In summary, the present application has at least the following beneficial effects: (1) By constructing a storage bucket array overall monitoring structure including a bit array, a counter array, a data item ID array, and a hash table, and combining a hash mapping method to allocate data items, the present application can significantly reduce the computational complexity and memory consumption, thereby realizing efficient monitoring of burst traffic in high-speed data streams.

[0024] (2) By combining the small flow counting and filtering mechanism, the present application effectively reduces the computational complexity and memory consumption during real-time analysis of massive high-speed data streams, optimizes the overall storage space, and improves the operating efficiency of the system; through suspicious traffic identification and suspicious burst flow tracking, unnecessary search operations are reduced, the processing speed of data items is accelerated, accurate screening and real-time tracking of high-frequency data streams are realized, and the detection accuracy and real-time performance of burst flows are greatly improved.

[0025] (3) By traversing the frequency change of the suspicious burst flow tracking table at the end of each monitoring period to judge the burst data stream, and further confirming the burst stream in combination with the burst duration, the present application ensures the reliability and stability of the detection result, realizes accurate identification and output of burst traffic, solves the problem of too long detection delay in the prior art, and takes into account cost-effectiveness at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is one of the general flowcharts of the burst traffic monitoring method for real-time analysis of high-speed data streams in this embodiment; Figure 2It is the second overall flowchart of the burst traffic monitoring method for real-time analysis of high-speed data streams in this embodiment; Figure 3 It is the flowchart of the suspicious traffic identification part in the burst traffic monitoring method for real-time analysis of high-speed data streams in this embodiment; Figure 4 It is the flowchart of the suspicious burst flow tracking part in the burst traffic monitoring method for real-time analysis of high-speed data streams in this embodiment; Figure 5 It is the flowchart of the small flow counting and filtering part in the burst traffic monitoring method for real-time analysis of high-speed data streams in this embodiment; Figure 6 It is the flowchart of the burst flow detection part in the burst traffic monitoring method for real-time analysis of high-speed data streams in this embodiment; Figure 7 It is the structural diagram of the burst traffic monitoring device for real-time analysis of high-speed data streams in this embodiment. Detailed implementation mode

[0027] The present application provides a burst traffic monitoring method, device, terminal and storage medium for real-time analysis of high-speed data streams. To make the purpose, technical solution and advantages of the present application clearer, the following will further elaborate on the implementation mode of the present application.

[0028] The following further describes in detail an embodiment of a burst traffic monitoring method for real-time analysis of high-speed data streams according to the present application with reference to the accompanying drawings of the specification.

[0029] A burst traffic monitoring method for real-time analysis of high-speed data streams according to the present application, as Figure 1 and Figure 2 shown, includes the following steps: S1. Construct an overall monitoring structure and initialize the overall monitoring structure.

[0030] S11. The overall monitoring structure includes a bucket array , and each bucket in the bucket array contains three parts: a suspicious traffic identification part, a small flow counting and filtering part and a suspicious burst flow tracking part.

[0031] The suspicious traffic identification part includes a bit array , the small flow counting and filtering part includes a counter array and a data item ID array , and the suspicious burst flow tracking part includes a hash table .

[0032] S12. For the suspicious traffic identification part, the bit array Each bit in it is initialized to 0.

[0033] For the small flow counting and filtering part, initialize each counter value in the counter array to 0, and initialize the value of each cell in the data item ID array to empty.

[0034] For the suspicious burst flow tracking part, the hash table contains multiple cells, and each cell contains 6 fields, namely: an ID field used to record the suspicious burst flow identifier, a field used to record the frequency of the burst flow in the current monitoring period, a field used to record the frequency of the burst flow in the previous monitoring period, a field used to record the timestamp when it becomes a suspicious burst flow, a field used to record the duration after the burst, and a field used to identify the conflict. Initialize the , , , and fields of each cell to 0, and initialize the ID field to empty.

[0035] S2. Receive the data stream arriving in the current monitoring period. Each arriving data stream includes multiple arriving data items, and all the arriving data items in the same data stream have the same unique identifier.

[0036] For each arriving data item, extract the unique identifier that represents the arriving data item .

[0037] S3. Allocate the corresponding arriving data item to the pre-constructed storage bucket array according to the hash mapping and the unique identifier of each arriving data item, and determine the data item ID of each arriving data item in the storage bucket array. The specific steps are as follows: S31. Map the arriving data item to the storage bucket array according to the pre-configured first mapping function and the unique identifier of the arriving data item, and obtain the first mapping position of the arriving data item.

[0038] In this embodiment, the first mapping function is the element-bucket mapping function.

[0039] The element-bucket mapping function is ; where is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the remainder operator, is the length of the bucket array.

[0040] By using an independent and uniform hash function, the uniform distribution of arriving data items in the bucket array is ensured, the collision probability is reduced, and the storage and retrieval efficiency are improved.

[0041] S32. Calculate the first mapping position of each arriving data item in the bucket array, specifically, ; where, is the calculated first mapping position, that is, the position of the arriving data item in the bucket array.

[0042] S4. Determine the frequency of the arriving data items based on the data item ID. When it is determined that the data stream corresponding to the arriving data item is a suspicious burst stream in combination with the result of the frequency determination, track the suspicious burst stream and summarize it to obtain a suspicious burst stream tracking table, as Figure 3 shown, specifically including the following steps: S41. Map each arriving data item reaching the bucket to the th bit of the bit array according to the pre-configured second mapping function. In this embodiment, the second mapping function is the element-bit mapping function.

[0043] The element-bit mapping function is, ; where, is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the remainder operator, is the length of the bit array; S42. Calculate the second mapping position of each arriving data item in the bit array in combination with the first mapping position; ; where, is the calculated second mapping position, that is, the position of each arriving data item in the bit array.

[0044] S43. Determine the frequency of the arriving data items and obtain the evaluation result, specifically, judge whether the data item ID at the second mapping position satisfies the preset low-frequency judgment rule.

[0045] In this embodiment, the low-frequency judgment rule is to judge whether the data item ID at the second mapping position is 0. If it is satisfied, that is , the determination result is that the arriving data item belongs to low-frequency elements; if not satisfied, that is , the determination result is that the arriving data item belongs to high-frequency elements.

[0046] By recording whether each flow is a suspicious burst flow in a certain monitoring period through the above steps, unnecessary search operations can be reduced and the processing speed of data items can be accelerated.

[0047] S44. When the arriving data item belongs to high-frequency elements, the data stream corresponding to the arriving data item is a suspicious burst flow, and the current suspicious burst flow is marked and tracked, as Figure 4 shown, specifically including the following steps: S441. When inserting the currently determined high-frequency element arriving data item and its data item frequency into the hash table , according to the preset third mapping function, map the currently arriving data item and the corresponding data item frequency to the th hash table entry of the hash table .

[0048] In this embodiment, the third mapping function is the element-hash table entry mapping function.

[0049] The element-hash table entry mapping function is ; where is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the modulo operator, is the length of the hash table .

[0050] S442. Obtain the third mapping position of the arriving data item, ; where is the calculated third mapping position, that is, the position of each arriving data item in the hash table.

[0051] As mentioned above, each hash table entry contains multiple units, and each unit contains an ID field for recording the suspicious burst flow identifier, a field for recording the burst flow frequency in the current monitoring period, and the burst flow frequency in the current monitoring period, as well as a field for indicating conflict.

[0052] S443. Determine whether there is a in all units of the hash table entry corresponding to the third mapping position that matches the unique identifier of the current arriving data item. Field

[0053] S444. If it exists, update the frequency of the burst stream in the current monitoring period according to the data item frequency.

[0054] In this embodiment, if a matching ID is found, it means that the arriving data item is already being tracked, and its field value is directly updated to , and the accumulation is performed according to the frequency carried by the currently arriving data item.

[0055] S445. If it does not exist and there is an empty cell in the hash table entry, update the information of the currently arriving data item in the empty cell.

[0056] In this embodiment, the existence of an empty cell means that there is an unused cell. At this time, the currently arriving data item and its frequency are updated to the empty cell of the hash table entry . Specifically, the ID field of is set to , and the field of is set to

[0057] S446. If it does not exist and there is no empty cell in the hash table entry, select a cell that meets the preset replacement criteria from the hash table entry for replacement. Replace and update the information of the current cell with the information of the currently arriving data item, and mark the corresponding data stream as a suspicious burst stream being tracked.

[0058] In this embodiment, the replacement criteria are set to a cell that is not the currently tracked suspicious burst stream and has the minimum burst stream frequency in the current monitoring period.

[0059] The steps of replacement and update are as follows. S4461. First, find from the hash table entry a cell that satisfies that the field is not 1 and has the minimum field.

[0060] Not being 1 means it is not the currently tracked suspicious burst stream, and the minimum field means the minimum burst stream frequency in the current monitoring period.

[0061] S4462. Calculate the rejection probability. The calculation formula of the rejection probability is, ; where represents the hash table entry with the minimum Unit of the field.

[0062] Through the rejection probability Replace the elements stored in with the currently arrived data item.

[0063] S4463. When the replacement is successful, then with the smallest field and update the ID field of the unit to , replace field to , and update other fields including , , to 0. Finally, update the field to 1 and mark it as a suspicious burst stream being tracked.

[0064] S4464. Use the preset element-candidate entry mapping function to calculate the candidate bucket position of the rejected data item in the unit where the replacement occurs , The element-candidate entry mapping function is ; The specific calculation method is .

[0065] Among them, is an independent and uniform hash function, is the unique identifier of the rejected data item, represents the modulo operator, is the hash table length.

[0066] S4465. Determine whether the rejection count of the rejected data item has reached the set maximum rejection threshold. If not, repeat the operations of tracking the suspicious burst stream in the above steps S441 - S446 until the rejected data item is saved to a certain hash table entry or reaches the maximum rejection threshold.

[0067] If the rejection count of the rejected data item reaches the maximum rejection threshold, set the bit with the subscript in the bit array to 0, indicating that is no longer regarded as part of the suspicious burst stream, and complete the replacement update.

[0068] In the above steps, in the case where there is no available empty unit, then according to the rejection probability Select a unit with the lowest frequency among the suspicious burst streams that are not being tracked for replacement. After successful replacement, update the information of this unit to the current information of the arriving data item and mark it as a suspicious burst stream being tracked. For the discarded data item to be replaced , use the element-candidate table entry mapping function to calculate its candidate hash table entry , and attempt to insert it into another hash table entry. If multiple attempts fail or the maximum discard count is reached, then remove it from the tracking of the suspicious burst stream.

[0069] Through the empty cell check and replacement strategy, the conflict problem in the hash table is effectively solved. When an empty cell is available, directly insert the new data item; when an empty cell is not available, adopt a probability-based replacement strategy to balance the storage space and the importance of the data item.

[0070] S45. Summarize the content of all the tracked suspicious burst streams and the corresponding hash table entries to obtain a suspicious burst stream tracking table. In this embodiment, the content of the hash table entry includes the content of all fields of each cell in the hash table entry.

[0071] S46. If the determination result is that the arriving data item belongs to a low-frequency element, enter the small flow counting and filtering part, as Figure 5 shown, which specifically includes the following steps: S461. Map the current arriving data item to the counter array and the data item ID array in the th cell according to the preset fourth mapping function; In this embodiment, the fourth mapping function

[0072] is the element-counter mapping function. ; wherein, is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the modulo operator, is the counter array and the data item ID array length.

[0073] ; wherein, is the calculated fourth mapping position, that is, the subscript position of each arriving data item in the counter array and the data item ID array.

[0074] S463. Determine whether the data item ID at the fourth mapping position matches the unique identifier of the currently arriving data item, and decide different update operations according to the matching result.

[0075] S4631. If they match, perform an operation of incrementing the counter value at the fourth mapping position by one. In this embodiment, if they match, , which means that the hash value of the currently arriving data item is the same as the hash value of a previous data item, and they indeed represent the same data item. At this time, the counter is updated to .

[0076] S4632. If they do not match and the fourth mapping position is empty, update the data item ID at the fourth mapping position according to the information of the currently arriving data item and initialize the counter value to 1.

[0077] In this embodiment, if they do not match and the fourth mapping position is empty, it means that no unique identifier of any data item has been stored at the fourth mapping position, that is, this position is empty.

[0078] When encountering a new arriving data item, if its hash value maps to the fourth mapping position and the fourth mapping position is empty, the unique identifier of the arriving data item is stored in , and the counter associated with this position is initialized to 1, indicating that the arriving data item has been encountered once.

[0079] The specific operation in this embodiment is to first determine whether the position of is empty. If it is empty, update and set to 1, and set to .

[0080] S4633. If they do not match and the fourth mapping position is not empty, perform an operation of decrementing the counter value by one.

[0081] In this embodiment, if they do not match and the fourth mapping position is not empty, it means that a hash conflict has occurred, that is, two different data items are hashed to the same position.

[0082] The specific operation in this embodiment is that if they do not match and it is determined that is not empty, then decrement the value of the counter by one; After performing the above operations, determine whether the value of the counter is equal to 0. If , clear the data item ID at the fourth mapping position and set is empty, and then end this part of the operation; if is not 0, directly end this part of the operation.

[0083] S464. Monitor the counter value in step S4631, and determine whether the counter value exceeds the preset small flow threshold .

[0084] When the counter value exceeds the small flow threshold, that is, in this embodiment, when, update to 1, mark the arriving data item as a suspicious burst flow and insert it into the suspicious burst flow part for tracking, and send the arriving data item and to the suspicious burst flow tracking part; When the counter value does not exceed the small flow threshold, complete the small flow counting and filtering operations.

[0085] The above process efficiently distinguishes low-frequency data streams (small flows) and high-frequency data streams (suspicious burst flows) through hash mapping and counter updates, filters the small flows, and when the value of the counter exceeds the set small flow threshold, reports the flow as a suspicious burst flow for suspicious burst flow tracking; otherwise, blocks the data stream from entering the subsequent part to optimize the overall storage space and reduce the computational cost for subsequent burst flow detection.

[0086] S5. According to the suspicious burst flow tracking table, enter the suspicious burst flow detection part, traverse the obtained suspicious burst flow tracking table at the end of each monitoring period, determine whether each suspicious burst flow is a burst data stream according to the preset burst data judgment mechanism, summarize to obtain a burst data stream list and output it, as Figure 6 shown, specifically including the following steps: S51. When each monitoring period ends, clear the counter array and data item ID array of each bucket in the bucket array.

[0087] In this embodiment, after each monitoring period ends, traverse the bucket array and each bucket in it, and reset each unit in the counter array and data item ID array contained in the small flow counting and filtering part of the bucket to 0.

[0088] S52. Traverse each unit of each hash table entry in the suspicious burst flow tracking table. Each unit includes the burst flow frequency in the current monitoring period, the burst flow frequency in the previous monitoring period, the timestamp when it becomes a suspicious burst flow and the duration after the burst ; In this embodiment, each cell in each hash table entry in the suspicious burst flow tracking part of each bucket is traversed, and the ID, , , and field contained in each cell are read.

[0089] S53. Obtain the change in the frequency of burst flows in adjacent cycles based on the frequency of burst flows in the current monitoring cycle and the frequency of burst flows in the previous monitoring cycle .

[0090] In this embodiment, the change in the frequency of burst flows in adjacent cycles is .

[0091] S54. Combine the timestamp of the suspicious burst flow and the preset burst threshold to determine whether each suspicious burst flow has a burst, and judge whether each suspicious burst flow is a burst data stream according to the duration after the burst.

[0092] S541. In this embodiment, first judge whether the field in the read cell is 0; S542. When , judge whether the change in the frequency of burst flows in adjacent cycles in the read cell is not less than the burst threshold .

[0093] If , and , it indicates that a burst flow has been detected. Update the of the currently read cell to the current monitoring cycle, and record the start time of the burst; and update the of the currently read cell to , and save the current frequency as the frequency of the previous monitoring cycle; finally, update to 0 to prepare for the counting of the next monitoring cycle; If , and , it means that the frequency has not increased significantly, so directly update the of the currently read cell to , and save the current frequency; and update to 0 to prepare for the counting of the next monitoring cycle.

[0094] S543. When , judge whether the change in the frequency of burst flows in adjacent cycles in the read cell is not less than the reciprocal of the burst threshold .

[0095] If and , increment by one the field value of the currently read cell, indicating that the stream has had another consecutive burst monitoring period, and update the field to , and set the field to 0; If , then directly output the data stream with the ID field in the currently read cell as the data item ID as a - burst data stream, and set all fields in the current cell to 0.

[0096] S544. Determine whether there are still un-traversed table item cells. If so, repeat the operations in steps S52 - S54 above. If not, complete the detection of burst data streams in the current monitoring period.

[0097] S55. Aggregate all the burst data streams determined in the above steps to obtain a burst data stream list and output it.

[0098] From the overall structure of this application, this application designs a strong real - time and low - overhead burst stream real - time monitoring technology, introducing the Sketch (approximate number) technology into the real - time processing of high - speed massive data streams: mapping traffic data to a limited storage space through a hash function, thereby achieving efficient traffic measurement and anomaly detection; judging whether there is a burst large flow by comparing the and ratio, and inferring the overall traffic trend based on limited sample data (i.e., the counts in the current and previous monitoring periods), which reflects the characteristic of Sketch to provide approximate calculation results of traffic data within a limited space. Overall, it reduces the high - speed computing overhead and high - speed cache overhead required for burst traffic real - time monitoring. On the basis of matching the high - speed data flow rate, it greatly improves the detection accuracy of burst large flows, thus realizing true strong real - time and high - precision burst large flow monitoring.

[0099] Based on the above same inventive concept, the embodiments of this application also disclose a burst traffic monitoring device for real - time analysis of high - speed data streams, the structure of which is as Figure 7 shown. The device includes the following modules: A burst traffic monitoring device for real - time analysis of high - speed data streams includes the following modules: A data stream receiving module, used to receive data streams arriving in the current monitoring period. Each data stream includes multiple arriving data items, and all the arriving data items in the same data stream have the same unique identifier; A data item mapping module, which is used to allocate the corresponding arriving data item to a pre-constructed storage bucket array according to the hash mapping and the unique identifier of each arriving data item, and determine the data item ID of each arriving data item in the storage bucket array; A suspicious burst flow tracking module, which is used to determine the frequency of the arriving data item according to the data item ID. When it is determined that the data stream corresponding to the arriving data item is a suspicious burst flow in combination with the result of the frequency determination, track the suspicious burst flow and summarize it to obtain a suspicious burst flow tracking table; A burst flow detection module, which is used to traverse the obtained suspicious burst flow tracking table at the end of each monitoring period, determine whether each suspicious burst flow is a burst data stream according to a preset burst data judgment mechanism, summarize it to obtain a burst data stream list and output it.

[0100] In a specific feasible implementation, the device further includes a monitoring structure initialization module, The monitoring structure initialization module is used to construct an overall monitoring structure. The overall monitoring structure includes a storage bucket array, and each storage bucket in the storage bucket array contains a suspicious traffic identification part, a small flow counting and filtering part, and a suspicious burst flow tracking part; The suspicious traffic identification part includes a bit array, the small flow counting and filtering part includes a counter array and a data item ID array, and the suspicious burst flow tracking part includes a hash table; Initialize the overall monitoring structure, and initialize all bits in the bit array, each cell in the counter array and the data item ID array, and all fields in the hash table to 0.

[0101] In a specific feasible implementation, the data item mapping module includes the following units: The first data item mapping unit is used to map the arriving data item to the storage bucket array according to a pre-configured first mapping function and the unique identifier of the arriving data item, and obtain the first mapping position of the arriving data item; The first mapping function is ; where is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the modulo operator, is the length of the storage bucket array.

[0102] The second data item mapping unit is used to calculate the first mapping position of each arriving data item in the storage bucket array, specifically ; where is the calculated first mapping position, that is, the position of the arriving data item in the storage bucket array.

[0103] In a specific implementable embodiment, the suspicious burst flow tracking module includes the following units: The first suspicious burst flow tracking unit is configured to map each arrival data item reaching the bucket to a bit array according to a pre-configured second mapping function, and calculate a second mapping position of each arrival data item in the bit array in combination with a first mapping position; The second mapping function is ; Wherein, is an independent and uniform hash function, is the unique identifier of the arrival data item, represents the modulo operator, is the length of the bit array; ; Wherein, is the calculated second mapping position, that is, the position of each arrival data item in the bit array.

[0104] The second suspicious burst flow tracking unit is configured to perform a frequency determination on the arrival data item and obtain an evaluation result. Specifically, judge whether the data item ID at the second mapping position meets a preset low-frequency judgment rule. If it meets, the judgment result is that the arrival data item belongs to a low-frequency element; if it does not meet, the judgment result is that the arrival data item belongs to a high-frequency element.

[0105] The third suspicious burst flow tracking unit is configured to, when the arrival data item belongs to a high-frequency element, mark and track the data stream corresponding to the arrival data item as a suspicious burst flow.

[0106] In a specific implementable embodiment, the third suspicious burst flow tracking unit includes the following subunits: The first suspicious burst flow tracking subunit is configured to map the current arrival data item and the corresponding data item frequency to a hash table entry of a hash table according to a preset third mapping function, and obtain a third mapping position of the arrival data item. Each hash table entry contains multiple units, and each unit contains fields, the burst flow frequency within the current monitoring period, and a suspicious burst flow identifier.

[0107] The second suspicious burst flow tracking subunit is configured to judge whether there is a field in the unit of the hash table entry corresponding to the third mapping position that matches the unique identifier of the current arrival data item. If it exists, update the burst flow frequency within the current monitoring period according to the data item frequency; If it does not exist and there is an empty unit in the hash table entry, update the information of the current arrival data item in the empty unit, If it does not exist and there is no empty cell in the hash table entry, select a cell that meets the preset replacement criteria from the hash table entry for replacement, replace and update the information of the current cell with the information of the currently arrived data item, and mark the corresponding data stream as a suspicious burst stream being tracked.

[0108] The third suspicious burst stream tracking subunit is used to summarize the content of all the suspicious burst streams being tracked and the corresponding hash table entries to obtain a suspicious burst stream tracking table.

[0109] In a specific feasible implementation, the suspicious burst stream tracking module further includes the following units: The fourth suspicious burst stream tracking unit is used to map the currently arrived data item into the counter array and the data item ID array according to a preset fourth mapping function to obtain the fourth mapping position of the arrived data item.

[0110] The fifth suspicious burst stream tracking unit is used to determine whether the data item ID at the fourth mapping position matches the unique identifier of the currently arrived data item. If it matches, perform an increment assignment operation on the counter value at the fourth mapping position; If it does not match and the fourth mapping position is empty, update the data item ID at the fourth mapping position according to the information of the currently arrived data item and initialize the counter value to 1; If it does not match and the fourth mapping position is not empty, perform a decrement operation on the counter value.

[0111] The sixth suspicious burst stream tracking unit is used to monitor the counter value. When the counter value reaches the preset small flow threshold, mark the arrived data item as a suspicious burst stream and track it. When the counter value is 0, clear the data item ID at the fourth mapping position.

[0112] In a specific feasible implementation, the burst stream detection module includes the following units: The first burst stream detection unit is used to empty the counter array and the data item ID array of each bucket in the bucket array after each monitoring period ends.

[0113] The second burst stream detection unit is used to traverse each cell of each hash table entry in the suspicious burst stream tracking table. Each cell includes the burst stream frequency in the current monitoring period, the burst stream frequency in the previous monitoring period, the timestamp when it becomes a suspicious burst stream, and the duration after the burst.

[0114] The third burst flow detection unit is used to obtain the change in the frequency of burst flows in adjacent cycles based on the frequency of burst flows in the current monitoring cycle and the frequency of burst flows in the previous monitoring cycle, and determine whether each suspected burst flow has a burst in combination with the timestamp of the suspected burst flow and a preset burst threshold, and determine whether each suspected burst flow is a burst data stream according to the duration after the burst.

[0115] The fourth burst flow detection unit is used to summarize all the determined burst data streams to obtain a burst data stream list and output it.

[0116] As can be seen from the above function introduction, a burst traffic monitoring device for real-time analysis of high-speed data streams in this application builds a set of burst traffic monitoring systems for high-speed data streams, improves the intelligent and automated level of traffic monitoring, improves the detection accuracy of burst large flows, and promotes the sustainable, rapid and healthy development of the economic society.

[0117] Based on the same inventive concept, an embodiment of the present application also discloses a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored, and at least one instruction, at least one program, a code set or an instruction set can be loaded and executed by a processor to implement the burst traffic monitoring method for real-time analysis of high-speed data streams provided in the above method embodiment.

[0118] Also based on the same inventive concept, an embodiment of the present application also discloses a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored, and at least one instruction, at least one program, a code set or an instruction set is loaded and executed by a processor to implement the burst traffic monitoring method for real-time analysis of high-speed data streams as described above.

[0119] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above embodiments can be completed by hardware, or can be completed by a program instructing related hardware. The program can be stored in the computer-readable storage medium. The computer-readable storage medium includes, for example: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks or optical discs that can store program codes.

[0120] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A burst traffic monitoring method for real-time analysis of high-speed data streams, characterized in that, It includes the following steps: Receive the data streams arriving in the current monitoring period. Each of the data streams includes multiple arriving data items, and all the arriving data items in the same data stream have the same unique identifier; According to the hash mapping and the unique identifier of each arriving data item, allocate the corresponding arriving data item to the pre-constructed bucket array, and determine the data item ID of each arriving data item in the bucket array; Perform frequency determination on the arriving data items according to the data item ID. When it is determined that the data stream corresponding to the arriving data item is a suspicious burst stream in combination with the result of the frequency determination, track the suspicious burst stream and summarize it to obtain a suspicious burst stream tracking table; At the end of each monitoring period, traverse the obtained suspicious burst stream tracking table, and determine whether each suspicious burst stream is a burst data stream according to the preset burst data judgment mechanism, summarize it to obtain a burst data stream list and output it.

2. The burst traffic monitoring method for real-time analysis of high-speed data streams according to claim 1, characterized in that Before receiving the data streams arriving in the current monitoring period, it further includes the following steps: Construct an overall monitoring structure. The overall monitoring structure includes a bucket array, and each bucket in the bucket array includes a suspicious traffic identification part, a small flow count and filtering part, and a suspicious burst stream tracking part; The suspicious traffic identification part includes a bit array, the small flow count and filtering part includes a counter array and a data item ID array, and the suspicious burst stream tracking part includes a hash table; Initialize the overall monitoring structure, and initialize all bits in the bit array, each unit in the counter array and the data item ID array, and all fields in the hash table to 0.

3. The burst traffic monitoring method for real-time analysis of high-speed data streams according to claim 2, characterized in that, The step of allocating the corresponding arriving data item to the pre-constructed bucket array according to the hash mapping and the unique identifier of each arriving data item and determining the data item ID of each arriving data item in the bucket array specifically includes the following steps: Map the arriving data item to the bucket array according to the pre-configured first mapping function and the unique identifier of the arriving data item to obtain the first mapping position of the arriving data item; The first mapping function is ; Among them, is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the modulo operator, is the length of the bucket array; Calculate the first mapping position of each arriving data item in the bucket array. Specifically, ； Among them, is the calculated first mapping position, that is, the position of the arrival data item in the bucket array.

4. The burst traffic monitoring method for real-time analysis of high-speed data streams according to claim 3, characterized in that, The step of performing frequency determination on the arriving data item according to the data item ID specifically includes the following steps: According to the pre-configured second mapping function, map each arriving data item arriving at the bucket to the bit array, and calculate the second mapping position of each arriving data item in the bit array in combination with the first mapping position; The second mapping function is ; Among them, is an independent and uniform hash function, is the unique identifier of the arriving data item, represents the modulo operator, is the length of the bit array; ； Among them, is the calculated second mapping position, that is, the position of each of the arriving data items in the bit array; Perform frequency determination on the arriving data item and obtain the evaluation result. Specifically, Judge whether the data item ID at the second mapping position meets the preset low-frequency judgment rule. If it meets, the judgment result is that the arriving data item belongs to a low-frequency element; if it does not meet, the judgment result is that the arriving data item belongs to a high-frequency element.

5. The burst traffic monitoring method for real-time analysis of high-speed data streams according to claim 4, characterized in that When it is determined that the data stream corresponding to the arrived data item is a suspicious burst stream based on the result of the frequency determination, track and summarize the suspicious burst stream to obtain a suspicious burst stream tracking table, which specifically includes the following steps: If the arrived data item belongs to a high-frequency element, the data stream corresponding to the arrived data item is a suspicious burst stream, and the current suspicious burst stream is marked and tracked. Specifically, Mapping the currently mentioned arrival data item and the corresponding data item frequency to the hash table entries of the hash table according to a preset third mapping function to obtain the third mapping position of the arrival data item, where each hash table entry contains multiple units, and each unit contains a field, the burst stream frequency within the current monitoring period, and a suspicious burst stream identifier; Determine whether there is a field in the unit of the hash table entry corresponding to the third mapping position that matches the unique identifier of the currently arrived data item. If there is, update the burst flow frequency in the current monitoring period according to the data item frequency; If there is, update the burst flow frequency in the current monitoring period according to the data item frequency; If it does not exist and there is an empty cell in the hash table entry, update the information of the current arrived data item in the empty cell. If it does not exist and there is no empty cell in the hash table entry, select a cell that meets the preset replacement criteria from the hash table entry for replacement, replace and update the information of the current cell with the information of the current arrived data item, and mark the corresponding data stream as a suspicious burst stream being tracked; Summarize all the tracked suspicious burst streams and the contents of the corresponding hash table entries to obtain a suspicious burst stream tracking table.

6. The burst traffic monitoring method for real-time analysis of high-speed data streams according to claim 4, characterized in that If the determination result is that the arrived data item belongs to a low-frequency element, it specifically includes the following steps: Map the current arrived data item to the counter array and the data item ID array according to a preset fourth mapping function to obtain the fourth mapping position of the arrived data item; Judge whether the data item ID at the fourth mapping position matches the unique identifier of the current arrived data item. If it matches, perform an increment assignment operation on the counter value at the fourth mapping position; If it does not match and the fourth mapping position is empty, update the data item ID at the fourth mapping position according to the information of the current arrived data item and initialize the counter value to 1; If it does not match and the fourth mapping position is not empty, perform a decrement operation on the counter value; Monitor the counter value. When the counter value reaches a preset small flow threshold, mark the arrived data item as a suspicious burst stream and track it. When the counter value is 0, clear the data item ID at the fourth mapping position.

7. The burst traffic monitoring method for real-time analysis of high-speed data streams according to claim 2, characterized in that At the end of each monitoring period, traverse the obtained suspicious burst stream tracking table, determine whether each suspicious burst stream is a burst data stream according to a preset burst data judgment mechanism, summarize to obtain a burst data stream list and output it, which specifically includes the following steps: After the end of each monitoring period, empty the counter array and the data item ID array of each bucket in the bucket array; Traverse each cell of each hash table entry in the suspicious burst stream tracking table. Each cell includes the burst stream frequency in the current monitoring period, the burst stream frequency in the previous monitoring period, the timestamp when it becomes a suspicious burst stream, and the duration after the burst; Obtain the change in the frequency between adjacent periods based on the burst stream frequency in the current monitoring period and the burst stream frequency in the previous monitoring period, determine whether each suspicious burst stream has a burst in combination with the timestamp when it becomes a suspicious burst stream and a preset burst threshold, and judge whether each suspicious burst stream is a burst data stream according to the duration after the burst; Summarize all the determined burst data streams to obtain a burst data stream list and output it.

8. A burst traffic monitoring device for real-time analysis of high-speed data streams, characterized in that, It includes the following modules: A data stream receiving module, configured to receive the data streams arriving within the current monitoring period, each of the data streams includes a plurality of arriving data items, and all the arriving data items in the same data stream have the same unique identifier; A data item mapping module, configured to allocate the corresponding arriving data item to a pre-constructed storage bucket array according to the hash mapping and the unique identifier of each arriving data item, and determine the data item ID of each arriving data item in the storage bucket array; A suspicious burst stream tracking module, configured to perform frequency determination on the arriving data items according to the data item ID, and when it is determined that the data stream corresponding to the arriving data item is a suspicious burst stream in combination with the result of the frequency determination, track the suspicious burst stream and summarize it to obtain a suspicious burst stream tracking table; A burst stream detection module, configured to traverse the obtained suspicious burst stream tracking table at the end of each monitoring period, determine whether each suspicious burst stream is a burst data stream according to a preset burst data judgment mechanism, summarize it to obtain a burst data stream list and output it.

9. An intelligent terminal, characterized in that, It includes a memory and a processor, and at least one segment of program is stored in the memory, and the at least one segment of program is loaded and executed by the processor to implement the burst traffic monitoring method for real-time analysis of high-speed data streams according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, At least one segment of program is stored in the readable storage medium, and the at least one segment of program is loaded and executed by the processor to implement the burst traffic monitoring method for real-time analysis of high-speed data streams according to any one of claims 1 to 7.

Images