Stabilized flow detection method for network security monitoring

Abstract

A steady flow detection method for network security monitoring screens out suspected steady substreams through a steady substream filter, and filters out non-steady flows in data flows in combination with a '0 / 1' stability pre-judgment method, so that not only can efficient utilization and high throughput rate of a memory be realized, but also the precision of a steady substream monitor is improved; an efficiency and precision foundation is laid for steady flow detection; meanwhile, a stable substream monitor is used for screening out stable substream in a few time windows, the problem of false positive caused by variance estimation deviation is solved in combination with a relative regeneration judgment algorithm, and the recognition efficiency and precision of the stable substream are improved while memory use is saved; and finally, a stable substream combiner is used for combining and restoring the multiple sections of stable substream to obtain the original stable substream, so that the memory for storing the multiple sections of stable substream can be reduced, and the accuracy of calculating the variance of the stable substream is improved. The method has the advantages of less memory usage, high throughput rate and high detection precision.

Description

Technical Field

[0001] The present invention relates to the technical field of network security monitoring, and in particular to a stable flow detection method for network security monitoring. Background Art

[0002] As a data flow pattern that combines temporal continuity and frequency stability, stable flow is a core detection target in network measurement. In high-speed network scenarios, stable flow detection technology has been deeply integrated into resource scheduling and security protection. Its detection efficiency directly determines the network service quality and security defense capabilities.

[0003] In resource scheduling, when using cache data prefetching technology, if a cache line is considered a data item, stable cache lines form a stable flow. Therefore, stable flow detection can be used to prefetch upcoming cache lines, thereby improving cache hit rate and efficiency. When allocating transmission bandwidth, it is necessary to ensure that network traffic has sufficient bandwidth. Stable flow detection can accurately pre-allocate bandwidth for stable network traffic. In the field of security protection, stable flow detection technology can effectively identify abnormal traffic characteristics and block potential attacks. In the face of distributed denial of service (DDoS) attacks, attack traffic often persists for a long time and exhibits stable abnormal characteristics. Stable flow detection can quickly identify stable abnormal traffic, allowing for prompt implementation of defensive measures such as traffic filtering and restriction to protect the network from paralysis. For data leakage attempts, stable flow detection can monitor and block abnormal and continuous data transmissions, which often exhibit clear regularity and continuity. This prevents the leakage of sensitive data by promptly detecting abnormal behavior.

[0004] Currently, relevant research focuses on persistence detection, with less research on frequency stability. Existing persistence flow detection methods can be divided into the following three types: sampling-based methods, coding-based methods, and sketch-based methods.

[0005] The core idea of sampling-based methods is to select data items with a certain sampling rate and mark the sampled data items in the filter. For example, by sampling and tracking the persistence of data items stored in a hash table to detect continuous flows, this method can reduce time and space overhead, but has the disadvantage of recording many non-persistent data items, affecting memory efficiency. Encoding-based methods improve space utilization by storing the encoding of data items instead of the data item ID. For example, by quickly encoding the data items appearing in each window for efficient data stream storage, and by calculating the stability of the data items during decoding. However, encoding and decoding require a lot of computational overhead, which reduces the update speed and makes PIE unable to handle high-speed data streams.

[0006] To address the accuracy and efficiency issues of the two aforementioned methods, the Sketch-based approach hashes items into buckets and stores cumulative information for all data flows, achieving fast updates and minimal memory usage at the expense of bounded errors. For example, on / off flags are set to prevent repeated flows within the same time window from being counted multiple times, while a replacement algorithm is used to probabilistically replace non-persistent flows. However, because each bucket only records cumulative information for data items, multiple data items are hashed into the same counter, resulting in many non-persistent data items being mistakenly identified as persistent. Data flow persistence is detected based on the popularity of consecutively arriving data items. However, because this approach does not consider the impact of data item arrival order, if a large number of non-persistent data items arrive in a bucket before persistent items, persistent items that enter later within the same time window will be incorrectly removed, resulting in reduced detection accuracy. For example, bucket stability metrics are introduced to detect persistent flows. However, due to the highly skewed data distribution in real-world scenarios, most data items have low persistence. The large number of non-persistent data items will expel potentially persistent data from the bucket, resulting in low detection accuracy.

[0007] Zhuochen Fan (ZCFan. et al., “SteadySketch: A High-Performance Algorithm for Finding Steady Flows in Data Streams,” IEE E / ACM Transaction Networking, 2024.) and others proposed a real-time detection method for steady flows. By combining a multi-level sketch structure with a regeneration technique, they achieved improved accuracy, increased throughput, and reduced memory requirements for real-time detection of steady flows. However, the regeneration technique has a certain false positive rate, which may lead to false alarms. In addition, the multi-level sketch structure uses linear averaging to process adjacent time windows, which can lead to deviations in variance estimation and reduce detection accuracy. Summary of the Invention

[0008] In order to overcome the defects of the above-mentioned prior art, the purpose of the present invention is to provide a stable flow detection method for network security monitoring. Suspected stable sub-flows are screened out through a stable sub-flow filter, and the unstable flow in the data flow is filtered out in combination with a "0 / 1" stability pre-judgment method. This not only achieves efficient memory utilization and high throughput, but also improves the accuracy of the stable sub-flow monitor, laying a foundation for efficiency and accuracy in stable flow detection; at the same time, the stable sub-flow monitor is used to screen out stable sub-flows within a smaller time window, and combined with the relative rebirth judgment algorithm, the false positive problem caused by variance estimation bias is solved, while saving memory usage and improving the recognition efficiency and accuracy of stable sub-flows; finally, a stable sub-flow merger is used to merge multiple stable sub-flows to restore the original stable flow, which not only reduces the memory for storing multiple stable sub-flows, but also improves the accuracy of calculating the variance of the stable flow; the present invention has the advantages of low memory usage, high throughput and high detection accuracy.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] A stable flow detection method for network security monitoring, comprising:

[0011] Inserting the data flow into the stable sub-flow filter to filter out suspected stable sub-flows, and transmitting the suspected stable sub-flows to the stable sub-flow monitor;

[0012] The stable sub-flow monitor judges the suspected stable sub-flow and transmits the successfully judged suspected stable sub-flow as a stable sub-flow to the stable sub-flow merger;

[0013] The stable sub-stream merger merges the stable sub-streams to obtain a stable stream.

[0014] Furthermore, the stable sub-stream filter specifically includes:

[0015] A two-dimensional array of size d×m, each row uses a hash function h1,...,h with a range of [0,m-1] d ; Let A ij is the bucket at row i and column j, where 1≤i≤d, 0≤j≤m-1; each bucket consists of the following three fields:

[0016] Continuity field: A 2-bit counter used to store the number of consecutive arrival windows of the data stream and to filter out suspected stable sub-streams that have arrived at least three consecutive windows. ij.c express;

[0017] Arrival field: A 1-bit sequence number used to store the arrival sequence number of the data stream, determine whether the arrival window of the data stream is continuous, and filter out unstable flows in the data stream through the "0 / 1" stability pre-judgment method.ij.a express;

[0018] Jump field: A 1-bit flag used to determine whether the suspected stable sub-flow in the bucket has been transmitted to the stable sub-flow monitor. ij.jp express.

[0019] Furthermore, the inserting of the data stream into the stable sub-stream filter to filter out suspected stable sub-streams, and transmitting the suspected stable sub-streams to the stable sub-stream monitor specifically includes:

[0020] Stable sub-stream filter initialization phase: Set the arrival field A of each bucket ij.a NULL, the continuity field A of each bucket ij.c and jump field A ij.jp All are 0;

[0021] Insertion phase: When the data stream reaches the stable sub-stream filter, the hash function h1,...,h d Find the bucket where the data stream is located, recorded as A ip ; Assume that the timestamp of the data stream arriving at the stable sub-stream filter is t, the window size is ws, and the window sequence number of the data stream is Then the arrival sequence number a of the data stream e =t n mod; where p = h i (e)(1≤i≤d); there are four cases:

[0022] Case 1: If the jump field A of the bucket where the data flow is located ip.jp All are 1, indicating that the data flow is filtered as a suspected stable sub-flow, and the arrival field A of the bucket where the data flow is located is set. ip.a =a e , the continuity field A of the bucket where the data flow is located ip.c and the jump field A of the bucket where the data flow is located ip.jp No update is made, and the suspected stable sub-flow is transmitted to the stable sub-flow monitor; where 1≤i≤d;

[0023] Case 2: If the data stream is located in bucket A ip If it is empty, the continuity field A of the bucket where the data flow is located is set to ip.c Add 1 and set the arrival field A of the bucket where the data stream is located ip.a =a e , the jump field A of the bucket where the data flow is located ip.jp remain unchanged;

[0024] Case 3: If the data stream is located in bucket A ipThe arrival field A of the bucket where the data flow is located is not empty ip.a =a e , indicating that the bucket in the current time window is accessed and no update operation is performed;

[0025] Case 4: If the data stream is located in bucket A ip The arrival field A of the bucket where the data flow is located is not empty ip.a ≠a e , indicating the bucket A where the data flow in the window is located ip If not accessed, the continuity field A of the bucket where the data stream is located is set to ip.c Add 1 and update the arrival field A of the bucket where the data stream is located ip.a =a e , the jump field A of the bucket where the data flow is located ip.jp Remain unchanged; if the continuity field A of the bucket where the data flow is located ip.c If it is 3, the continuity field A of the bucket where the data flow is located is maintained. ip.c unchanged; when all buckets in the window are updated, if the continuity field A of the bucket where the data stream is located is ip.c If all are 3, it means that the data flow has arrived in at least three consecutive windows, and it is filtered into a suspected stable sub-flow. The jump field A of the bucket where the data flow is located is set to ip.jp Set to 1 and pass suspected stable sub-flows to the stable sub-flow monitor.

[0026] Furthermore, filtering the unstable flow in the data flow by the "0 / 1" stability pre-judgment method specifically includes:

[0027] At the end of the insertion phase of each time window, the arrival sequence number of the last data flow in the given end window is a e ;

[0028] Traverse the arrival field A of d×m buckets ij.a ;For arrival field A ij.a and arrival number a e The same bucket indicates that the bucket in the current window has been accessed by the data stream, keeping the bucket status unchanged; for the arrival field A ij.a and arrival number a e Different buckets indicate that the data flow in the bucket within the current window is interrupted. The unstable flow in the data flow is cleared by performing the operations of the stable sub-flow filter initialization phase on the bucket.

[0029] Furthermore, the stable sub-flow monitor specifically includes:

[0030] A hash table consisting of r buckets uses a hash function g(.) with a range of [0, r-1]; let B f is the fth bucket, each bucket has z cells, bucket B f The kth cell in is B fk ; where 0≤f≤r-1, 1≤k≤z;

[0031] Each cell contains the following three fields:

[0032] ID field: used to store the ID of the stable sub-stream, which is stored by B fk .ID means;

[0033] Count field: It consists of s+1 counters of size N bits. Let B fk .c x is the xth counter; s counters are used to record the frequency of the flow in s time windows, and one counter is used to determine the start of a new window, 1≤x≤s+1;

[0034] Check field: consists of two reset bits B of size N. fk .ck1 and reset B fk .ck2, which is used to record the difference in the number of counter resets in adjacent time windows and to determine suspected stable sub-flows in combination with the count field.

[0035] Furthermore, the stable sub-flow monitor determines the suspected stable sub-flow and transmits the successfully determined suspected stable sub-flow as a stable sub-flow to the stable sub-flow merger, specifically including:

[0036] Initialization phase of stable sub-flow monitor: Set the s+1 counters of the count field to NULL, and the first reset B of the check field fk .ck1 and second reset B fk .ck2 are all 1;

[0037] Update and judgment phase: Let the ID of the suspected stable sub-flow transmitted by the stable sub-flow filter be ID e , the window number of the suspected stable sub-flow is t n , the counter number corresponding to the count field is y=t n mod(s+1); use the hash function g(.) to find the bucket B mapped to the suspected stable sub-flow g(e) There are three situations:

[0038] Case 1: If there is no bucket B mapped by the suspected stable sub-flow g(e)The ID of the cell is the same as the ID of the suspected stable subflow, but there is an empty cell. The empty cell is recorded as the kth cell, and the cell B where the suspected stable subflow is located is recorded as g(e)k The ID is set to ID e , the yth counter B of the count field corresponding to the current time window of the suspected stable sub-flow g(e)k .c y Set to 1;

[0039] Case 2: If there is a bucket B mapped by a suspected stable sub-flow g(e) The cell whose ID is the same as the ID of the suspected stable sub-flow, and the counter B of the count field corresponding to the current time window of the suspected stable sub-flow g(e)k .c y If it is not empty, the counter B of the count field corresponding to the current time window of the suspected stable sub-flow is g(e)k .c y Add 1 and update the check field; if the counter B of the count field corresponding to the current time window of the suspected stable sub-flow g(e)k .c y If it is empty, it indicates that the suspected stable sub-flow comes from a new window. The suspected stable sub-flow needs to be judged according to the relative rebirth judgment algorithm. After the judgment, there are two situations:

[0040] i) Relative stability is determined to be successful; if s counters have been used, the suspected stable sub-flow that has been determined to be successful <ID e , w, var>, that is, the stable sub-flow is transmitted to the stable sub-flow merger, and then the cell B where the suspected stable sub-flow is located is reset g(e)k The count field and check field of the suspected stable sub-flow are updated to update the counter B of the count field corresponding to the current time window. g(e) k .c y , that is, the operation of the initialization phase of the stable sub-flow monitor is performed; if s counters are not used, it means that all windows meet the relative stability requirements, and only the counter B of the count field corresponding to the current time window of the suspected stable sub-flow needs to be updated g(e) k .c y ; where w = t n -s, var is the minimum value of the direct frequency variance and the offset frequency variance of s counters;

[0041] ii) Relative stability determination fails, reset cell B where the suspected stable subflow is located g(e)k , i.e., perform the operations of the initialization phase of the stable sub-flow monitor and update it according to the steps of case 1;

[0042] Case 3: Bucket B mapped by the suspected stable sub-flow does not existg(e) The ID of the cell is the same as the ID of the suspected stable sub-flow, and there is no empty cell. Query the counter B of the check field of all buckets g(e)k .c y1 , if there is a counter B g(e)k .c y1 If the bucket is NULL, reset the bucket and store the suspected stable sub-flow, that is, perform the operation of the stable sub-flow monitor initialization phase and update it according to the steps of case 1; if there is no counter B g(e)k .c y1 If the bucket is NULL, find the bucket with the least number of consecutive time windows, that is, the bucket with the most empty counters, and replace it. The replacement probability is 1-(S min / S), where S min is the number of continuous time windows; if the replacement is successful, the suspected stable sub-flow is stored in the bucket, that is, the operation of the stable sub-flow monitor initialization phase is performed and updated according to the steps of case 1; if the replacement fails, the suspected stable sub-flow is ignored;

[0043] In other cases, no update operation will be performed.

[0044] Furthermore, the determining of the suspected stable sub-flow according to the relative rebirth determination algorithm specifically includes:

[0045] Assume that three suspected stable sub-flows with the same ID appear in sequence number t n , t n +1, t n In the +2 time window, use bucket B f The kth cell in B fk The count field in the counter B fk .c x1 , Counter B fk .c x2 , Counter B fk .c x3 ; where x1 = t n mod(s+1),x2=(t n +1)mod(s+1),x3=(t n +2)mod(s+1);

[0046] When the suspected stable subflow appears at sequence number t n When the time window is fk .c x1 Record the frequency of suspected stable sub-flows and enable the resetter B in the check field fk .ck1, whenever reset B fk .ck1 reset once, reset device B fk .ck1 corresponds to plus 1;

[0047] When the suspected stable subflow appears at sequence number t n +1 time window, use counter B fk .c x2 and resetter B fk .ck2, reset device B fk .ck1 stops increasing; resets B fk .ck2 resets each time, resets B fk .ck1 corresponds to minus 1, reset device B fk .ck2 corresponds to plus 1, through reset device B fk .ck1 reflects counter B fk .c x1 and counter B fk .c x2 The relative reset times of the counter in adjacent time windows is the difference between the reset times of the counter in adjacent time windows; if the reset device B fk .ck1 is 0, then the first reset B is no longer fk .ck1 performs a decrement operation and is directly set to NULL. n +1 No longer affects Resetter B in the window fk .ck1 update;

[0048] When the suspected stable subflow appears at sequence number t n +2 time window, use counter B fk .c x3 , reset device B fk .ck1 is reset to 1 and re-enabled, reset device B fk .ck2 is disabled; whenever counter B fk .c x3 Reset once, reset device B fk .ck2 minus 1, reset B fk .ck1 plus 1;

[0049] The same operation as the above three suspected stable sub-flows is performed on the suspected stable sub-flows arriving in the subsequent window, and the reset device B is used alternately. fk .ck1 and reset B fk .ck2;

[0050] When the sequence number is t n At the end of the window of +1, the suspected stable sub-flow arriving in the subsequent window is n and t n Relative stability is determined within the +1 window; there are five situations:

[0051] Case 1: If reset device B fk .ck1 is greater than 2, indicating that the sequence number is t nThe frequency of the window with sequence number t n The frequency of the window with +1 is at least 2 greater N , relative stability determination failed;

[0052] Case 2: If reset device B fk .ck1 is equal to 2, then calculate B fk .c x1 +2 N -B fk .c x2 Is it less than the judgment threshold α? If B fk .c x1 +2 N -B fk .c x2< α, relative stability is determined successfully; if B fk .c x1 +2 N -B fk .c x2≥ α, relative stability determination failed;

[0053] Case 3: If reset device B fk .ck1 is equal to 1, which means that counter B fk .c x1 and counter B fk .c x2 The difference in the number of reset times of the counter in adjacent time windows is consistent, and |B fk .c x1 -B fk .c x2 | is less than the threshold α, if | B fk .c x1 -B fk .c x2 | < α, relative stability determination success; |B fk .c x1 -B fk .c x2 | ≥ α, relative stability determination failed;

[0054] Case 4: If reset device B fk .ck1 is equal to 0, then calculate B fk .c x2 +2 N -B fk .c x1 Is it less than the threshold α? If B fk .c x2 +2 N -B fk .c x1< α, relative stability is determined successfully; if Bfk .c x2 +2 N -B fk .c x1≥ α, relative stability determination failed;

[0055] Case 5: If the first reset device B fk .ck1 is equal to NULL, indicating that the sequence number is t n The frequency of the window with the number t is greater than that of the window with the number t n The frequency of the window is at least 2 N , the relative stability determination failed.

[0056] Furthermore, the stable sub-stream merger specifically includes:

[0057] A hash table consisting of l buckets uses a hash function q(.) with a range of [0, l-1]; let D u is the u-th bucket, where 0≤u≤l-1; there are b cells in each bucket, let bucket D u The ath cell in is D ua , where 1≤a≤b;

[0058] Each cell contains the following four fields:

[0059] Identify field: used to store the ID of the stable sub-flow, which is determined by D ua .id means;

[0060] Start field: The size is N bits, used to store the starting window sequence number of the stable sub-stream, used to determine the stability of each stable sub-stream, and is determined by D ua .s means;

[0061] Variance field: a one-dimensional array of length P, used to record the variance of each merged stable sub-stream, represented by D ua .v c Indicates; where 1≤c≤P;

[0062] Number field: A counter of size N bits, used to record the number of segments of the stable sub-streams of the merged stable stream, combined with the variance field to merge the stable sub-streams, and is determined by D ua .n means.

[0063] Furthermore, the stable sub-stream merger merges the stable sub-streams to obtain the stable stream, specifically comprising:

[0064] Stable substream merger initialization phase: set the identify field D of all cells ua .id, start field D ua.s, variance field D ua .v c 、number field D ua .n are all 0;

[0065] Update and report phase: For the stable sub-flow transmitted by the stable sub-flow monitor, use the hash function q(.) to find the bucket D where the stable sub-flow is located. y(e) ; There are four update situations:

[0066] Case 1: If there is no cell with the same ID as the stable sub-flow's ID in the identify field of the cell where the stable sub-flow is located, but there is an empty cell, then select an empty cell and insert the stable sub-flow information into it, which is recorded as the vth cell. At the same time, the identify field D of the cell where the stable sub-flow is located is set to y(e)v. The id is set to IDe, and the start field D of the cell where the subflow is located is stabilized y(e)v. s is set to w, the counter D of the variance field of the cell where the sub-flow is located is stabilized y(e)v. v1 is set to var, which stabilizes the number field D of the cell where the sub-flow is located. y(e)v. n plus 1;

[0067] Case 2: If there is a cell where the ID of the identify field of the cell where the stable sub-flow is located is the same as the ID of the stable sub-flow, and D y(e)v. s+D y(e)v. n*s+1≠w, which means the cell D where the stable subflow is located y(e)v If there is a break between the stable sub-flow stored in the time window and the stable sub-flow entered, the report <IDe,D y(e)v. s, D y(e)v. s+D y(e)v. n*s> is a stable flow, reset the cell D where the stable sub-flow is located y(e)v , that is, to perform the operation of the initialization phase of the stable sub-stream merger and update it according to the situation 1; among them, D y(e)v. s is the start field of the cell where the stable sub-flow is located; D y(e)v. n is the number field of the cell where the stable sub-flow is located; s+1 is the number of counters in the count field of the stable sub-flow monitor; w is the window number of the starting stable sub-flow;

[0068] Case 3: If there is a cell where the ID of the identify field of the stable sub-flow is the same as the ID of the stable sub-flow, and D y(e)v. s+D y(e)v. n*s+1=w, which means the cell D where the stable subflow is located y(e)v The stable sub-flow stored in is continuous with the time window of the incoming stable sub-flow, and the cell D where the stable sub-flow is located is accessed.y(e)v The variance field D ua .v c and number field D ua .n, and uses multi-segment sub-stream merging technology to determine whether the stable sub-stream is successfully merged;

[0069] If the merge is successful, the number field D of the cell where the sub-flow is located will be stabilized. y(e)v. n plus 1, the variance field D of the cell where the sub-flow is located is stabilized y(e)v. v Dy(e)v.n Set to var; when the number field D of the cell where the stable sub-flow is located y(e)v. When n=P, report <ID e , D y(e)v. s, D y(e)v. s+P*s> is a stable flow, reset the cell D where the stable sub-flow is located y(e)v , that is, to perform the initialization phase of the stable sub-stream merger and update it according to the situation; if the merger fails, report the stable stream <IDe,D y(e)v. s, D y(e)v. s+D y(e)v. n*s> and reset the cell D where the stable subflow is located y(e)v , i.e., perform the operations of the initialization phase of the stable sub-stream combiner and update it according to situation 1;

[0070] Case 4: If there is no cell where the ID of the identify field of the cell where the stable sub-flow is located is the same as the ID of the stable sub-flow, and there is no empty cell, check the nearest window of each cell; if there is D in the nearest window y(e)v. s+D y(e) v. n*s+1≠w, indicating that the stable flow stored in the cell has a window interruption. After reporting the stable flow information in the cell, reset the cell to store the stable sub-flow, that is, perform the operation of the stable sub-flow merger initialization phase and update it according to the situation 1; if the latest window of all cells has D y(e)v. s+D y(e)v. n*s+1=w, then find the minimum number of stable windows W min The cell, calculate the replacement probability T = 1 / (W min -s+1); if the replacement is successful, the stable flow information in the cell is reported and the cell is reset, and the stable sub-flow is stored, that is, the operation of the stable sub-flow merger initialization phase is performed and updated according to situation 1; if the replacement fails, the stable sub-flow is ignored.

[0071] Furthermore, the use of the multi-segment sub-stream merging technology to determine whether the stable sub-stream is successfully merged specifically includes:

[0072] Given multiple stable sub-streams with the same ID, the duration window of each stable sub-stream is continuous and of length s, and the frequency variances are var1, var2, ...var n , calculate the variance of the merged stable flow according to the merge variance formula; the merge variance formula is specifically:

[0073]

[0074] If the merged variance U≤Φ, the merge is successful; if the merged variance U>Φ, the merge fails; where Φ is the defined variance threshold.

[0075] Compared with the prior art, the present invention has the following beneficial effects:

[0076] 1. The present invention designs a stable sub-stream filter to screen out suspected stable sub-streams, and combines it with the "0 / 1" stability pre-judgment method to filter out unstable streams in the data stream. With the unique design of windowless sequence number storage, it realizes rapid pre-screening of the data stream. At the same time, it cooperates with the specific field jump judgment mechanism to reduce memory consumption while reducing a large number of unnecessary update operations, achieving efficient utilization and high throughput.

[0077] 2. The present invention designs a stable sub-flow monitor to perform frequency statistics in a continuous window on the suspected stable flow transmitted by the stable sub-flow filter, and combines it with a relative rebirth judgment algorithm to quickly screen out stable sub-flows from the suspected stable flow, thereby saving memory usage and improving the recognition efficiency and accuracy of stable sub-flows.

[0078] 3. The present invention merges the stable sub-stream monitors by designing a stable sub-stream merger, and combines the multi-segment sub-stream merging technology to determine the merging status of the stable stream to ensure that the original stable stream is accurately merged. This not only reduces the memory for storing multiple segments of stable sub-streams, but also improves the accuracy of calculating the stable stream variance.

[0079] In summary, the present invention uses a stable sub-flow filter to filter out suspected stable sub-flows, and combines the "0 / 1" stability pre-judgment method to filter out unstable flows in the data flow, which not only can achieve efficient memory utilization and high throughput, but also improves the accuracy of the stable sub-flow monitor, laying a foundation for efficiency and accuracy in the detection of stable flows; at the same time, the stable sub-flow monitor is used to filter out stable sub-flows within a smaller time window, and combined with the relative rebirth judgment algorithm, the false positive problem caused by variance estimation deviation is solved, while saving memory usage and improving the recognition efficiency and accuracy of stable sub-flows; finally, a stable sub-flow merger is used to merge multiple stable sub-flows to restore the original stable flow, which not only reduces the memory for storing multiple stable sub-flows, but also improves the accuracy of calculating the variance of the stable flow; the present invention has the advantages of low memory usage, high throughput and high detection accuracy, and can be widely used in resource scheduling and security protection and other fields. Brief Description of the Drawings

[0080] Figure 1 It is a schematic flowchart of a stable flow detection method for network security monitoring.

[0081] Figure 2 It is a data structure diagram of a stable sub - flow filter.

[0082] Figure 3 It is a data structure diagram of a stable sub - flow monitor.

[0083] Figure 4 It is a data structure diagram of a stable sub - flow combiner. Detailed Implementation Manner

[0084] The present invention will be further described in detail below with reference to the drawings and embodiments.

[0085] Stable flow detection needs to take into account both the persistence and frequency stability of the data flow. However, most current methods can only measure the persistence or frequency of the data flow from a single dimension, resulting in low detection accuracy of stable flows; during the process of detecting stable flows, it is necessary to continuously perform information statistics on non - stable flows and stable flows in consecutive windows, resulting in the detection time complexity increasing linearly with the number of detection windows, significantly reducing the detection efficiency; traditional single - dimension data flow detection needs to maintain both the persistence state in the time domain and the stability index in the frequency domain, resulting in the algorithm space complexity increasing exponentially with the scale of the data volume, increasing the memory occupancy and the overhead of computing resources;

[0086] Aiming at the problems of low detection accuracy, poor efficiency, and high overhead in the above - mentioned stable flow detection, the present invention proposes a stable flow detection method for network security monitoring, which transforms the process of identifying stable flows into the process of identifying their stable sub - flows. Given n stable flows e1, e2,... e with the same ID n , where the number of duration windows for each stable flow is s, and the stable durations are [t1, t2], [t2 + 1, t3],..., [tn - 1+1, tn]; if the variances of the n stable flows are close, they can be combined into a stable flow F with a stable duration of [t1, tn], then the n stable flows e1, e2,... e with the same IDs n are called stable sub - flows of the stable flow F, and the number of duration windows W of the stable flow F satisfies s << W; by detecting stable sub - flows that are stable within fewer time windows and combining multiple stable sub - flows into the required stable flow, the purpose of saving memory and quickly filtering non - stable flows is achieved. See Figure 1 The specific implementation steps are as follows:

[0087] Phase 1: Insert the data stream into the stable sub-flow filter to filter out suspected stable sub-flows, and transmit the suspected stable sub-flows to the stable sub-flow monitor;

[0088] like Figure 2 As shown, the stable sub-stream filter specifically includes:

[0089] A two-dimensional array of size d×m, each row uses a hash function h1,...,h with a range of [0,m-1] d ; Let A ij is the bucket at row i and column j, where 1≤i≤d, 0≤j≤m-1; each bucket consists of the following three fields:

[0090] Continuity field: A 2-bit counter used to store the number of consecutive arrival windows of the data stream and to filter out suspected stable sub-streams that have arrived at least three consecutive windows. ij.c express;

[0091] Arrival field: A 1-bit sequence number used to store the arrival sequence number of the data stream, determine whether the arrival window of the data stream is continuous, and filter out unstable flows in the data stream through the "0 / 1" stability pre-judgment method. ij.a express;

[0092] Jump field: A 1-bit flag used to determine whether the suspected stable sub-flow in the bucket has been transmitted to the stable sub-flow monitor. ij.jp express.

[0093] The stable sub-flow filter in this embodiment is combined with the "0 / 1" stability pre-judgment method to quickly judge the stability of the flow in advance without storing the time window sequence number, which greatly reduces the memory consumption required for stability judgment and achieves efficient memory utilization and high throughput.

[0094] Furthermore, the inserting of the data stream into the stable sub-stream filter to filter out suspected stable sub-streams, and transmitting the suspected stable sub-streams to the stable sub-stream monitor specifically includes:

[0095] Stable sub-stream filter initialization phase: Set the arrival field A of each bucket ij.a NULL, the continuity field A of each bucket ij.c and jump field A ij.jp All are 0;

[0096] Insertion phase: When the data stream reaches the stable sub-stream filter, the hash function h1,...,h d Find the bucket where the data stream is located, recorded as A ip; Assume that the timestamp of the data stream arriving at the stable sub-stream filter is t, the window size is ws, and the window sequence number of the data stream is Then the arrival sequence number a of the data stream e =t n mod; where p = h i (e)(1≤i≤d); there are four cases:

[0097] Case 1: If the jump field A of the bucket where the data flow is located ip.jp All are 1, indicating that the data flow is filtered as a suspected stable sub-flow, and the arrival field A of the bucket where the data flow is located is set. ip.a =a e , the continuity field A of the bucket where the data flow is located ip.c and the jump field A of the bucket where the data flow is located ip.jp No update is performed, and the suspected stable sub-flow is sent to the stable sub-flow monitor; where 1≤i≤d; Case 1 is the priority judgment when the data flow is mapped to the bucket. If the judgment is successful, it does not need to enter the bucket of the first stage and is directly sent to the stable sub-flow monitor of the second stage;

[0098] Case 2: If the data stream is located in bucket A ip If it is empty, the continuity field A of the bucket where the data flow is located is set to ip.c Add 1 and set the arrival field A of the bucket where the data stream is located ip.a =a e , the jump field A of the bucket where the data flow is located ip.jp remains unchanged; Case 2 is the operation of data flow entering the empty bucket, which is equivalent to the initial value assignment operation of the bucket;

[0099] Case 3: If the data stream is located in bucket A ip The arrival field A of the bucket where the data flow is located is not empty ip.a =a e , indicating that the bucket in the current time window is accessed and no update operation is performed; Case three is the case where the data stream enters the bucket without accumulation;

[0100] Case 4: If the data stream is located in bucket A ip The arrival field A of the bucket where the data flow is located is not empty ip.a ≠a e , indicating the bucket A where the data flow in the window is located ip If not accessed, the continuity field A of the bucket where the data stream is located is set to ip.c Add 1 and update the arrival field A of the bucket where the data stream is located ip.a =a e, the jump field A of the bucket where the data flow is located ip.jp Remain unchanged; if the continuity field A of the bucket where the data flow is located ip.c If it is 3, the continuity field A of the bucket where the data flow is located is maintained. ip.c unchanged; when all buckets in the window are updated, if the continuity field A of the bucket where the data stream is located is ip.c If all are 3, it means that the data flow has arrived in at least three consecutive windows, and it is filtered into a suspected stable sub-flow. The jump field A of the bucket where the data flow is located is set to ip.jp Set to 1, and transmit the suspected stable sub-flow to the stable sub-flow monitor. Case 4 is the accumulation operation of the bucket. When the accumulation reaches the threshold 3, it is reported and transmitted to the stable sub-flow monitor in the second stage.

[0101] Through the jump judgment in the above-mentioned initialization stage and insertion stage, the suspected stable sub-flow that has been screened out can directly enter the stable sub-flow monitor, reducing the number of unnecessary updates, improving the screening efficiency of suspected stable sub-flows, avoiding the sacrifice of frequency information during the screening process, and laying a foundation for efficiency and accuracy for subsequent detection.

[0102] The specific methods of filtering unstable data streams using the "0 / 1" stability pre-judgment method include:

[0103] At the end of the insertion phase of each time window, the arrival sequence number of the last data flow in the given end window is a e ;

[0104] Traverse the arrival field A of d×m buckets ij.a ;For arrival field A ij.a and arrival number a e The same bucket indicates that the bucket in the current window has been accessed by the data stream, keeping the bucket status unchanged; for the arrival field A ij.a and arrival number a e Different buckets indicate that the data flow in the bucket within the current window is interrupted. By performing the operation of the stable sub-stream filter initialization phase on the bucket, the unstable flow in the data flow is cleared, which significantly improves the accuracy of the subsequent arriving data flow information; although the arrival window of the data flow is interrupted, it will also cause the arrival sequence number a e The same or different results, but at the end of each window insertion phase, the arrival sequence number a is used e Resetting the unaccessed buckets can ensure the window continuity of the data flow information recorded in each bucket; therefore, by comparing the arrival sequence number a of the data flow e, we can quickly determine whether the data stream arrives continuously or repeatedly, and only need to consider the arrival of adjacent windows. Compared with the method of checking whether the data stream has arrived in the previous s windows each time, it saves a lot of time and computational overhead.

[0105] Phase 2: The stable sub-flow monitor determines the suspected stable sub-flows and transmits the successfully determined suspected stable sub-flows as stable sub-flows to the stable sub-flow merger;

[0106] like Figure 3 As shown, the stable sub-flow monitor specifically includes:

[0107] A hash table consisting of r buckets uses a hash function g(.) with a range of [0, r-1]; let B f is the fth bucket, each bucket has z cells, bucket B f The kth cell in is B fk ; where 0≤f≤r-1, 1≤k≤z;

[0108] Each cell contains the following three fields:

[0109] ID field: used to store the ID of the stable sub-stream, which is stored by B fk .ID means;

[0110] Count field: It consists of s+1 counters of size N bits. Let B fk .c x is the xth counter; s counters are used to record the frequency of the flow in s time windows, and one counter is used to determine the start of a new window, 1≤x≤s+1. Since the stability detection of suspected stable subflows only needs to analyze the frequency changes of suspected stable subflows in consecutive windows, there is no need to record the specific frequency of suspected stable subflows. Therefore, in order to save memory overhead, a relative rebirth determination algorithm is proposed to process each counter.

[0111] Check field: consists of two reset bits B of size N. fk .ck1 and reset B fk .ck2 is composed of the reset times of the counter in adjacent time windows, and is used to record the difference between the reset times of the counter in adjacent time windows, and to judge the suspected stable sub-flow in combination with the count field. fk .ck1 and reset B fk .ck2 are all 3 bits;

[0112] Furthermore, the stable sub-flow monitor determines the suspected stable sub-flow and transmits the successfully determined suspected stable sub-flow as a stable sub-flow to the stable sub-flow merger, specifically including:

[0113] Initialization phase of stable sub-flow monitor: Set the s+1 counters of the count field to NULL, and the first reset B of the check field fk .ck1 and second reset B fk .ck2 are all 1;

[0114] Update and judgment phase: Let the ID of the suspected stable sub-flow transmitted by the stable sub-flow filter be ID e , the window number of the suspected stable sub-flow is t n , the counter number corresponding to the count field is y=t n mod(s+1); use the hash function g(.) to find the bucket B mapped to the suspected stable sub-flow g(e) There are three situations:

[0115] Case 1: If there is no bucket B mapped by the suspected stable sub-flow g(e) The ID of the cell is the same as the ID of the suspected stable subflow, but there is an empty cell. The empty cell is recorded as the kth cell, and the cell B where the suspected stable subflow is located is recorded as g(e)k The ID is set to ID e , the yth counter B of the count field corresponding to the current time window of the suspected stable sub-flow g(e)k .c y Set to 1; Case 1 is the operation of a suspected stable subflow entering an empty cell, which is equivalent to the operation of assigning an initial value to the cell;

[0116] Case 2: If there is a bucket B mapped by a suspected stable sub-flow g(e) The cell whose ID is the same as the ID of the suspected stable sub-flow, and the counter B of the count field corresponding to the current time window of the suspected stable sub-flow g(e)k .c y Not empty, that is, the stable sub-flow does not belong to the new time window, directly the counter B of the count field corresponding to the current time window g(e)k .c y Accumulate the counter B of the count field corresponding to the current time window of the suspected stable sub-flow g(e)k .c y Add 1 and update the check field; if the counter B of the count field corresponding to the current time window of the suspected stable sub-flow g(e)k .c y If it is empty, it indicates that the suspected stable sub-flow comes from a new window. It is necessary to judge the suspected stable sub-flow according to the relative rebirth judgment algorithm, that is, to judge the counter B of the count field corresponding to the current time window of the suspected stable sub-flow. g(e)k .c y1 Counter B of the count field corresponding to the current time window of the suspected stable sub-flowg(e)k .c y2 ; Among them, y1=(t n -1)mod(s+1),y2=(t n -2)mod(s+1);

[0117] Furthermore, the determining of the suspected stable sub-flow according to the relative rebirth determination algorithm specifically includes:

[0118] Assume that three suspected stable sub-flows with the same ID appear in sequence number t n , t n +1, t n In the +2 time window, use bucket B f The kth cell in B fk The count field in the counter B fk .c x1 , Counter B fk .c x2 , Counter B fk .c x3 ; where x1 = t n mod(s+1),x2=(t n +1)mod(s+1),x3=(t n +2)mod(s+1);

[0119] When the suspected stable subflow appears at sequence number t n When the time window is fk .c x1 Record the frequency of suspected stable sub-flows and enable the resetter B in the check field fk .ck1, whenever reset B fk .ck1 reset once, reset device B fk .ck1 corresponds to plus 1;

[0120] When the suspected stable subflow appears at sequence number t n +1 time window, use counter B fk .c x2 and resetter B fk .ck2, reset device B fk .ck1 stops increasing; resets B fk .ck2 resets each time, resets B fk .ck1 corresponds to minus 1, reset device B fk .ck2 corresponds to plus 1, through reset device B fk .ck1 reflects counter B fk .c x1 and counter B fk .c x2The relative reset times of the counter in adjacent time windows is the difference between the reset times of the counter in adjacent time windows; if the reset device B fk .ck1 is 0, then the first reset B is no longer fk .ck1 performs a decrement operation and is directly set to NULL. n +1 No longer affects Resetter B in the window fk .ck1 update;

[0121] When the suspected stable subflow appears at sequence number t n +2 time window, use counter B fk .c x3 , reset device B fk .ck1 is reset to 1 and re-enabled, reset device B fk .ck2 is disabled; whenever counter B fk .c x3 Reset once, reset device B fk .ck2 minus 1, reset B fk .ck1 plus 1;

[0122] The same operation as the above three suspected stable sub-flows is performed on the suspected stable sub-flows arriving in the subsequent window, and the reset device B is used alternately. fk .ck1 and reset B fk .ck2;

[0123] When the sequence number is t n At the end of the window of +1, the suspected stable sub-flow arriving in the subsequent window is n and t n The relative stability is determined within a window of +1. An 8-bit counter is used as an example. There are five cases:

[0124] Case 1: If reset device B fk .ck1 is greater than 2, indicating that the sequence number is t n The frequency of the window with sequence number t n The frequency of the window with +1 is at least 2 greater N , the relative stability determination fails; for example, when the frequencies of the suspected stable sub-flow e1 and the suspected stable sub-flow e2 are 522 and 12 respectively, the reset device B fk .ck1 is 3, the suspected stable sub-flow e1 and the suspected stable sub-flow e1 do not meet the relative stability condition;

[0125] Case 2: If reset device B fk .ck1 is equal to 2, then calculate B fk .c x1 +2 N -B fk .c x2 Is it less than the judgment threshold α? If Bfk .c x1 +2 N -B fk .c x2< α, relative stability is determined successfully; if B fk .c x1 +2 N -B fk .c x2≥ α, relative stability determination fails; for example, set the threshold α to 5, when the frequencies of the suspected stable sub-flow e1 and the suspected stable sub-flow e2 are 257 and 254 respectively, reset B fk .ck1 is 2, counter B fk .c x1 , Counter B fk .c x2 When the frequencies of the suspected stable sub-flow e1 and the suspected stable sub-flow e2 are 258 and 8 respectively, the reset device B is reset. fk .ck1 is 2, counter B fk .c x1 , Counter B fk .c x2 2 and 8 respectively, and the relative stability determination failed;

[0126] Case 3: If reset device B fk .ck1 is equal to 1, which means that counter B fk .c x1 and counter B fk .c x2 The difference in the number of reset times of the counter in adjacent time windows is consistent, and |B fk .c x1 -B fk .c x2 | is less than the threshold α, if | B fk .c x1 -B fk .c x2 | < α, relative stability determination success; |B fk .c x1 -B fk .c x2 | ≥ α, relative stability determination fails; for example, set the threshold α to 5, when the frequencies of the suspected stable sub-flow e1 and the suspected stable sub-flow e2 are 258 and 259 respectively, reset B fk .ck1 is 1, counter B fk .c x1 , Counter B fk .c x2When the frequencies of the suspected stable sub-flow e1 and the suspected stable sub-flow e2 are 500 and 259 respectively, the reset device B is reset. fk .ck1 is 1, counter B fk .c x1 , Counter B fk .c x2 They are 244 and 3 respectively, and the relative stability determination fails;

[0127] Case 4: If reset device B fk .ck1 is equal to 0, then calculate B fk .c x2 +2 N -B fk .c x1 Is it less than the threshold α? If B fk .c x2 +2 N -B fk .c x1< α, relative stability is determined successfully; if B fk .c x2 +2 N -B fk .c x1≥ α, relative stability determination fails; for example, set the threshold α to 5, when the frequencies of the suspected stable sub-flow e1 and the suspected stable sub-flow e2 are 254 and 258 respectively, reset B fk .ck1 is 0, counter B fk .c x1 , Counter B fk .c x2 When the frequencies of the suspected stable sub-flow e1 and the suspected stable sub-flow e2 are 8 and 258 respectively, the reset device B is reset. fk .ck1 is 0, counter B fk .c x1 , Counter B fk .c x2 They are 8 and 2 respectively, and the relative stability determination fails;

[0128] Case 5: If the first reset device B fk .ck1 is equal to NULL, indicating that the sequence number is t n The frequency of the window with the number t is greater than that of the window with the number t n The frequency of the window is at least 2 N , the relative stability determination fails; for example, when the frequencies of the suspected stable sub-flow e1 and the suspected stable sub-flow e2 are 12 and 513 respectively, the reset device B fk .ck1 is NULL, counter B fk .c x22, the relative stability determination fails;

[0129] There are two situations after the judgment:

[0130] i) Relative stability is determined to be successful; if s counters have been used, it means that the stored stable sub-flow time window has reached the threshold, and the suspected stable sub-flow will be determined to be successful. <ID e , w, var>, that is, the stable sub-flow is transmitted to the stable sub-flow merger, and then the cell B where the suspected stable sub-flow is located is reset g(e)k The count field and check field of the suspected stable sub-flow are updated to update the counter B of the count field corresponding to the current time window. g(e)k .c y , that is, the operation of the stable sub-flow monitor initialization phase; if s counters are not used, it means that all windows meet the relative stability requirements, but the upper limit of the number of stable sub-flow windows has not been reached. Only the counter B of the count field corresponding to the current time window of the suspected stable sub-flow needs to be updated g(e)k .c y ; where w = t n -s, var is the minimum value of the direct frequency variance and the offset frequency variance of s counters;

[0131] ii) Relative stability determination fails, reset cell B where the suspected stable subflow is located g(e)k , i.e., perform the operations of the initialization phase of the stable sub-flow monitor and update it according to the steps of case 1;

[0132] Case 3: Bucket B mapped by the suspected stable sub-flow does not exist g(e) The ID of the cell is the same as the ID of the suspected stable sub-flow, and there is no empty cell. Query the counter B of the check field of all buckets g(e)k .c y1 , if there is a counter B g(e)k .c y1 If the bucket is NULL, it means that the stable sub-flow stored in the cell is interrupted and the condition is no longer met, then the bucket is reset and the suspected stable sub-flow is stored, that is, the operation of the stable sub-flow monitor initialization phase is performed and updated according to the steps of case 1; if there is no counter B g(e)k .c y1 If the bucket is NULL, find the bucket with the least number of consecutive time windows, that is, the bucket with the most empty counters, and replace it. The replacement probability is 1-(S min / S), where S min is the number of continuous time windows; if the replacement is successful, the suspected stable sub-flow is stored in the bucket, that is, the operation of the stable sub-flow monitor initialization phase is performed and updated according to the steps of case 1; if the replacement fails, the suspected stable sub-flow is ignored;

[0133] In other cases, no update operation will be performed.

[0134] By using the stable sub-flow monitor to screen out stable sub-flows within a shorter time window and combining it with the relative rebirth determination algorithm, the false positive problem caused by variance estimation bias is solved. This improves the efficiency and accuracy of stable sub-flow identification while saving memory usage.

[0135] The third stage: The stable sub-stream merger merges the stable sub-streams to obtain a stable stream.

[0136] like Figure 4 As shown, the stable sub-stream merger specifically includes:

[0137] A hash table consisting of l buckets uses a hash function q(.) with a range of [0, l-1]; let D u is the u-th bucket, where 0≤u≤l-1; there are b cells in each bucket, let bucket D u The ath cell in is D ua , where 1≤a≤b;

[0138] Each cell contains the following 4 fields:

[0139] Identify field: used to store the ID of the stable sub-flow, which is determined by D ua .id means;

[0140] Start field: The size is N bits, used to store the starting window sequence number of the stable sub-stream, used to determine the stability of each stable sub-stream, and is determined by D ua .s means;

[0141] Variance field: a one-dimensional array of length P, used to record the variance of each merged stable sub-stream, represented by D ua .v c Indicates; where 1≤c≤P;

[0142] Number field: A counter of size N bits, used to record the number of segments of the stable sub-streams of the merged stable stream, combined with the variance field to merge the stable sub-streams, and is determined by D ua .n means.

[0143] Furthermore, the stable sub-stream merger merges the stable sub-streams to obtain the stable stream, specifically comprising:

[0144] Stable substream merger initialization phase: set the identify field D of all cells ua .id, start field D ua .s, variance field D ua .v c、number field D ua .n are all 0;

[0145] Update and report phase: For the stable sub-flow transmitted by the stable sub-flow monitor, use the hash function q(.) to find the bucket D where the stable sub-flow is located. y(e) ; There are four update situations:

[0146] Case 1: If there is no cell with the same ID as the stable sub-flow's ID in the identify field of the cell where the stable sub-flow is located, but there is an empty cell, then select an empty cell and insert the stable sub-flow information into it, which is recorded as the vth cell. At the same time, the identify field D of the cell where the stable sub-flow is located is set to y(e)v. The id is set to IDe, and the start field D of the cell where the subflow is located is stabilized y(e)v. s is set to w, the counter D of the variance field of the cell where the sub-flow is located is stabilized y(e)v. v1 is set to var, which stabilizes the number field D of the cell where the sub-flow is located. y(e)v. n plus 1; Case 1 is the operation of the stable subflow entering the empty cell, which is equivalent to the operation of assigning the initial value to the cell;

[0147] Case 2: If there is a cell where the ID of the identify field of the cell where the stable sub-flow is located is the same as the ID of the stable sub-flow, and D y(e)v. s+D y(e)v. n*s+1≠w, which means the cell D where the stable subflow is located y(e)v If there is a break between the stable sub-flow stored in the time window and the stable sub-flow entered, the report <IDe,D y(e)v. s, D y(e)v. s+D y(e)v. n*s> is a stable flow, reset the cell D where the stable sub-flow is located y(e)v , that is, to perform the operation of the initialization phase of the stable sub-stream merger and update it according to the situation 1; among them, D y(e)v. s is the start field of the cell where the stable sub-flow is located; D y(e)v. n is the number field of the cell where the stable subflow is located; s+1 is the number of the counter in the count field of the stable subflow monitor; w is the window number of the starting window of the stable subflow; Case 2 is the accumulation of stable subflows based on Case 1, that is, the case when the time window is interrupted when the ID is the same;

[0148] Case 3: If there is a cell where the ID of the identify field of the stable sub-flow is the same as the ID of the stable sub-flow, and D y(e)v. s+D y(e)v. n*s+1=w, which means the cell D where the stable subflow is located y(e)vThe stable sub-flow stored in is continuous with the time window of the incoming stable sub-flow, and the cell D where the stable sub-flow is located is accessed. y(e)v The variance field D ua .v c and number field D ua .n, and uses multi-segment sub-stream merging technology to determine whether the stable sub-stream is successfully merged; Case 3 is the accumulation of stable sub-streams based on Case 2, that is, the time windows are continuous when the ID is the same;

[0149] Furthermore, the use of the multi-segment sub-stream merging technology to determine whether the stable sub-stream is successfully merged specifically includes:

[0150] Given multiple stable sub-streams with the same ID, the duration window of each stable sub-stream is continuous and of length s, and the frequency variances are var1, var2, ...var n , calculate the variance of the merged stable flow according to the merge variance formula; the merge variance formula is specifically:

[0151]

[0152] If the merged variance U≤Φ, the merge is successful; if the merged variance U>Φ, the merge fails; where Φ is the defined variance threshold.

[0153] If the merge is successful, the number field D of the cell where the sub-flow is located will be stabilized. y(e)v. n plus 1, the variance field D of the cell where the sub-flow is located is stabilized y(e)v. v Dy(e)v.n Set to var; when the number field D of the cell where the stable sub-flow is located y(e)v. When n=P, it means that the stored stable flow reaches the storage threshold, and the report <ID e , D y(e)v. s, D y(e)v. s+P*s> is a stable flow, reset the cell D where the stable sub-flow is located y(e)v , that is, to perform the initialization phase of the stable sub-stream merger and update it according to the situation; if the merger fails, it means that the merged stream is not stable, so the stable stream is reported <IDe,D y(e)v. s, D y(e)v. s+D y(e)v. n*s> and reset the cell D where the stable subflow is located y(e)v , i.e., perform the operations of the initialization phase of the stable sub-stream combiner and update it according to situation 1;

[0154] Case 4: If there is no cell where the ID of the identify field of the cell where the stable sub-flow is located is the same as the ID of the stable sub-flow, and there is no empty cell, check the nearest window of each cell; if there is D in the nearest windowy(e)v. s+D y(e) v. n*s+1≠w, indicating that the stable flow stored in the cell has a window interruption. After reporting the stable flow information in the cell, reset the cell to store the stable sub-flow, that is, perform the operation of the stable sub-flow merger initialization phase and update it according to the situation 1; if the latest window of all cells has D y(e)v. s+D y(e)v. n*s+1=w, indicating that the stable flows stored in all cells are continuous, and the minimum number of stable windows is W. min The cell, calculate the replacement probability T = 1 / (W min -s+1); if the replacement is successful, the stable flow information in the cell is reported and the cell is reset, and the stable sub-flow is stored, that is, the initialization phase operation is performed and updated according to the situation 1; if the replacement fails, the stable sub-flow is ignored.

[0155] The multiple stable sub-streams transmitted by the stable sub-stream monitor are merged into one stable stream one by one through the stable sub-stream merger and multi-segment sub-stream merging technology, which not only ensures the accuracy of stable stream detection, but also can report the stable stream in real time.

[0156] In summary, compared with the existing technology:

[0157] In terms of economy, the stable flow detection method proposed in the present invention can realize the detection of stable flow in scenarios with limited memory resources. It has the advantages of low memory usage, high throughput, and high detection accuracy. It is easy to deploy and has high commercial value.

[0158] In terms of real-time performance, the stable flow detection method proposed in the present invention can greatly reduce the amount of data that needs to be analyzed during the detection process, ensure the efficiency of massive data collection and analysis, and have less time consumption.

[0159] In terms of resource scheduling, it can not only improve the hit rate and efficiency of cache data prefetching technology, but also dynamically allocate bandwidth resources to links that need to continuously ensure data transmission quality.

[0160] In terms of security protection, by identifying traffic with stable characteristics, it is possible to detect link flooding DoS attacks, prevent continuous scanning attacks, and block data leaks.

[0161] In terms of versatility, the stable flow detection method proposed in the present invention can not only be used to detect the continuity of data flow, but also can be used for frequency and cardinality analysis of data flow, and has certain scalability.

[0162] In terms of accuracy: the stable flow detection method proposed in the present invention can solve the false positive problem of the current method and improve the accuracy of stable flow detection.

[0163] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A stable flow detection method for network security monitoring, characterized by: include: Inserting the data flow into the stable sub-flow filter to filter out suspected stable sub-flows, and transmitting the suspected stable sub-flows to the stable sub-flow monitor; The stable sub-flow monitor judges the suspected stable sub-flow and transmits the successfully judged suspected stable sub-flow as a stable sub-flow to the stable sub-flow merger; The stable sub-stream merger merges the stable sub-streams to obtain a stable stream.

2. The method for detecting a stable flow for network security monitoring according to claim 1, characterized in that: The stable sub-flow filter specifically includes: A two-dimensional array of size d×m, each row uses a hash function h1,...,h with a range of [0,m-1] d ; Let A ij is the bucket at row i and column j, where 1≤i≤d, 0≤j≤m-1; each bucket consists of the following three fields: Continuity field: A 2-bit counter used to store the number of consecutive arrival windows of the data stream and to filter out suspected stable sub-streams that have arrived at least three consecutive windows. ij.c express; Arrival field: A 1-bit sequence number used to store the arrival sequence number of the data stream, determine whether the arrival window of the data stream is continuous, and filter out unstable flows in the data stream through the "0 / 1" stability pre-judgment method. ij.a express; Jump field: A 1-bit flag used to determine whether the suspected stable sub-flow in the bucket has been transmitted to the stable sub-flow monitor. ij.jp express.

3. The method for detecting a stable flow for network security monitoring according to claim 1, wherein: The step of inserting the data stream into the stable sub-stream filter to filter out suspected stable sub-streams, and transmitting the suspected stable sub-streams to the stable sub-stream monitor specifically includes: Stable sub-stream filter initialization phase: Set the arrival field A of each bucket ij.a NULL, the continuity field A of each bucket ij.c and jump field A ij.jp All are 0; Insertion phase: When the data stream reaches the stable sub-stream filter, the hash function h1,...,h d Find the bucket where the data flow is located, recorded as A ip ; Assume that the timestamp of the data stream arriving at the stable sub-stream filter is t, the window size is ws, and the window sequence number of the data stream is Then the arrival sequence number a of the data stream e =t n mod; where p = h i (e)(1≤i≤d); there are four cases: Case 1: If the jump field A of the bucket where the data flow is located ip.jp All are 1, indicating that the data flow is filtered as a suspected stable sub-flow, and the arrival field A of the bucket where the data flow is located is set. ip.a =a e , the continuity field A of the bucket where the data flow is located ip.c and the jump field A of the bucket where the data flow is located ip.jp No update is made, and the suspected stable sub-flow is transmitted to the stable sub-flow monitor; where 1≤i≤d; Case 2: If the data stream is located in bucket A ip If it is empty, the continuity field A of the bucket where the data flow is located is set to ip.c Add 1 and set the arrival field A of the bucket where the data stream is located ip.a =a e , the jump field A of the bucket where the data flow is located ip.jp remain unchanged; Case 3: If the data stream is located in bucket A ip The arrival field A of the bucket where the data flow is located is not empty ip.a =a e , indicating that the bucket in the current time window is accessed and no update operation is performed; Case 4: If the data stream is located in bucket A ip The arrival field A of the bucket where the data flow is located is not empty ip.a ≠a e , indicating the bucket A where the data flow in the window is located ip If not accessed, the continuity field A of the bucket where the data stream is located is set to ip.c Add 1 and update the arrival field A of the bucket where the data stream is located ip.a =a e , the jump field A of the bucket where the data flow is located ip.jp Remain unchanged; if the continuity field A of the bucket where the data flow is located ip.c If it is 3, the continuity field A of the bucket where the data flow is located is maintained. ip.c unchanged; when all buckets in the window are updated, if the continuity field A of the bucket where the data stream is located is ip.c If all are 3, it means that the data flow has arrived in at least three consecutive windows, and it is filtered into a suspected stable sub-flow. The jump field A of the bucket where the data flow is located is set to ip.jp Set to 1 and pass suspected stable sub-flows to the stable sub-flow monitor.

4. A stable flow detection method for network security monitoring according to claim 2 or 3, characterized in that: The filtering of unstable flows in the data flow by the "0 / 1" stability pre-judgment method specifically includes: At the end of the insertion phase of each time window, the arrival sequence number of the last data flow in the given end window is a e ; Traverse the arrival field A of d×m buckets ij.a ; For arrival field A ij.a and arrival number a e The same bucket indicates that the bucket in the current window has been accessed by the data stream, keeping the bucket status unchanged; for the arrival field A ij.a and arrival number a e Different buckets indicate that the data flow in the bucket within the current window is interrupted. The unstable flow in the data flow is cleared by performing the operations of the stable sub-flow filter initialization phase on the bucket.

5. The method for detecting a stable flow for network security monitoring according to claim 1, characterized in that: The stable sub-flow monitor specifically includes: A hash table consisting of r buckets uses a hash function g(.) with a range of [0, r-1]; let B f is the fth bucket, each bucket has z cells, bucket B f The kth cell in is B fk ; where 0≤f≤r-1, 1≤k≤z; Each cell contains the following three fields: ID field: used to store the ID of the stable sub-stream, which is stored by B fk .ID means; Count field: It consists of s+1 counters of size N bits. Let B fk .c x is the xth counter; s counters are used to record the frequency of the flow in s time windows, and one counter is used to determine the start of a new window, 1≤x≤s+1; Check field: consists of two reset bits B of size N. fk .ck1 and reset B fk .ck2, which is used to record the difference in the number of counter resets in adjacent time windows and to determine suspected stable sub-flows in combination with the count field.

6. A stable flow detection method for network security monitoring according to claim 1 or 5, characterized in that: The stable sub-flow monitor determines the suspected stable sub-flow and transmits the successfully determined suspected stable sub-flow as a stable sub-flow to the stable sub-flow merger, which specifically includes: Initialization phase of stable sub-flow monitor: Set the s+1 counters of the count field to NULL, and the first reset B of the check field fk .ck1 and second reset B fk .ck2 are all 1; Update and judgment phase: Let the ID of the suspected stable sub-flow transmitted by the stable sub-flow filter be ID e , the window number of the suspected stable sub-flow is t n , the counter number corresponding to the count field is y=t n mod(s+1); use the hash function g(.) to find the bucket B mapped to the suspected stable sub-flow g(e) There are three situations: Case 1: If there is no bucket B mapped by the suspected stable sub-flow g(e) The ID of the cell is the same as the ID of the suspected stable subflow, but there is an empty cell. The empty cell is recorded as the kth cell, and the cell B where the suspected stable subflow is located is recorded as g(e)k The ID is set to ID e , the yth counter B of the count field corresponding to the current time window of the suspected stable sub-flow g(e)k .c y Set to 1; Case 2: If there is a bucket B mapped by a suspected stable sub-flow g(e) The cell whose ID is the same as the ID of the suspected stable sub-flow, and the counter B of the count field corresponding to the current time window of the suspected stable sub-flow g(e)k .c y If it is not empty, the counter B of the count field corresponding to the current time window of the suspected stable sub-flow is g(e)k .c y Add 1 and update the check field; if the counter B of the count field corresponding to the current time window of the suspected stable sub-flow g(e)k .c y If it is empty, it indicates that the suspected stable sub-flow comes from a new window. The suspected stable sub-flow needs to be judged according to the relative rebirth judgment algorithm. After the judgment, there are two situations: i) Relative stability is determined to be successful; if s counters have been used, the suspected stable sub-flow that has been determined to be successful <ID e , w, var>, that is, the stable sub-flow is transmitted to the stable sub-flow merger, and then the cell B where the suspected stable sub-flow is located is reset g(e)k The count field and check field of the suspected stable sub-flow are updated to update the counter B of the count field corresponding to the current time window. g(e)k .c y , that is, the operation of the initialization phase of the stable sub-flow monitor is performed; if s counters are not used, it means that all windows meet the relative stability requirements, and only the counter B of the count field corresponding to the current time window of the suspected stable sub-flow needs to be updated g(e)k .c y ; where w = t n -s, var is the minimum value of the direct frequency variance and the offset frequency variance of s counters; ii) Relative stability determination fails, reset cell B where the suspected stable subflow is located g(e)k , i.e., perform the operations of the initialization phase of the stable sub-flow monitor and update it according to the steps of case 1; Case 3: Bucket B mapped by the suspected stable sub-flow does not exist g(e) The ID of the cell is the same as the ID of the suspected stable sub-flow, and there is no empty cell. Query the counter B of the check field of all buckets g(e)k .c y1 , if there is a counter B g(e)k .c y1 If the bucket is NULL, reset the bucket and store the suspected stable sub-flow, that is, perform the operation of the stable sub-flow monitor initialization phase and update it according to the steps of case 1; if there is no counter B g(e)k .c y1 If the bucket is NULL, find the bucket with the least number of consecutive time windows, that is, the bucket with the most empty counters, and replace it. The replacement probability is 1-(S min / S), where S min is the number of continuous time windows; if the replacement is successful, the suspected stable sub-flow is stored in the bucket, that is, the operation of the stable sub-flow monitor initialization phase is performed and updated according to the steps of case 1; if the replacement fails, the suspected stable sub-flow is ignored; In other cases, no update operation will be performed.

7. The method for detecting a stable flow for network security monitoring according to claim 6, characterized in that: The determining of the suspected stable sub-flow according to the relative rebirth determination algorithm specifically includes: Assume that three suspected stable sub-flows with the same ID appear in sequence number t n , t n +1, t n In the +2 time window, use bucket B f The kth cell in B fk The count field in the counter B fk .c x1 , Counter B fk .c x2 , Counter B fk .c x3 ; where x1 = t n mod(s+1),x2=(t n +1)mod(s+1),x3=(t n +2)mod(s+1); When the suspected stable subflow appears at sequence number t n When the time window is fk .c x1 Record the frequency of suspected stable sub-flows and enable the resetter B in the check field fk .ck1, whenever reset B fk .ck1 reset once, reset device B fk .ck1 corresponds to plus 1; When the suspected stable subflow appears at sequence number t n +1 time window, use counter B fk .c x2 and resetter B fk .ck2, reset device B fk .ck1 stops increasing; resets B fk .ck2 resets each time, resets B fk .ck1 corresponds to minus 1, reset device B fk .ck2 corresponds to plus 1, through reset device B fk .ck1 reflects counter B fk .c x1 and counter B fk .c x2 The relative reset times of the counter in adjacent time windows is the difference between the reset times of the counter in adjacent time windows; if the reset device B fk .ck1 is 0, then the first reset B is no longer fk .ck1 performs a decrement operation and is directly set to NULL. n +1 No longer affects Resetter B in the window fk .ck1 update; When the suspected stable subflow appears at sequence number t n +2 time window, use counter B fk .c x3 , reset device B fk .ck1 is reset to 1 and re-enabled, reset device B fk .ck2 is disabled; whenever counter B fk .c x3 Reset once, reset device B fk .ck2 minus 1, reset B fk .ck1 plus 1; The same operation as the above three suspected stable sub-flows is performed on the suspected stable sub-flows arriving in the subsequent window, and the reset device B is used alternately. fk .ck1 and reset B fk .ck2; When the sequence number is t n At the end of the window of +1, the suspected stable sub-flow arriving in the subsequent window is n and t n Relative stability is determined within the +1 window; there are five situations: Case 1: If reset device B fk .ck1 is greater than 2, indicating that the sequence number is t n The frequency of the window with sequence number t n The frequency of the window with +1 is at least 2 greater N , relative stability determination failed; Case 2: If reset device B fk .ck1 is equal to 2, then calculate B fk .c x1 +2 N -B fk .c x2 Is it less than the judgment threshold α? If B fk .c x1 +2 N -B fk .c x2< α, relative stability is determined successfully; if B fk .c x1 +2 N -B fk .c x2≥ α, relative stability determination failed; Case 3: If reset device B fk .ck1 is equal to 1, which means that counter B fk .c x1 and counter B fk .c x2 The difference in the number of reset times of the counter in adjacent time windows is consistent, and |B fk .c x1 -B fk .c x2 | is less than the threshold α, if | B fk .c x1 -B fk .c x2 | < α, relative stability determination success; |B fk .c x1 -B fk .c x2 | ≥ α, relative stability determination failed; Case 4: If reset device B fk .ck1 is equal to 0, then calculate B fk .c x2 +2 N -B fk .c x1 Is it less than the threshold α? If B fk .c x2 +2 N -B fk .c x1< α, relative stability is determined successfully; if B fk .c x2 +2 N -B fk .c x1≥ α, relative stability determination failed; Case 5: If the first reset device B fk .ck1 is equal to NULL, indicating that the sequence number is t n The frequency of the window with the number t is greater than that of the window with the number t n The frequency of the window is at least 2 N , the relative stability determination failed.

8. The method for detecting a stable flow for network security monitoring according to claim 1, characterized in that: The stable sub-stream merger specifically includes: A hash table consisting of l buckets uses a hash function q(.) with a range of [0, l-1]; let D u is the u-th bucket, where 0≤u≤l-1; there are b cells in each bucket, let bucket D u The ath cell in is D ua , where 1≤a≤b; Each cell contains the following four fields: Identify field: used to store the ID of the stable sub-flow, which is determined by D ua .id means; Start field: The size is N bits, used to store the starting window sequence number of the stable sub-flow, used to determine the stability of each stable sub-flow, and is determined by D ua .s means; Variance field: a one-dimensional array of length P, used to record the variance of each merged stable sub-stream, represented by D ua .v c Indicates; where 1≤c≤P; Number field: A counter of size N bits, used to record the number of segments of the stable sub-streams of the merged stable stream, combined with the variance field to merge the stable sub-streams, and is determined by D ua .n means.

9. A stable flow detection method for network security monitoring according to claim 1 or 8, characterized in that: The stable sub-stream merger merges the stable sub-streams to obtain a stable stream, specifically comprising: Stable substream merger initialization phase: set the identify field D of all cells ua .id, start field D ua .s, variance field D ua .v c 、number field D ua .n are all 0; Update and report phase: For the stable sub-flow transmitted by the stable sub-flow monitor, use the hash function q(.) to find the bucket D where the stable sub-flow is located. y(e) There are four situations: Case 1: If there is no cell with the same ID as the stable sub-flow's ID in the identify field of the cell where the stable sub-flow is located, but there is an empty cell, then select an empty cell and insert the stable sub-flow information into it, which is recorded as the vth cell. At the same time, the identify field D of the cell where the stable sub-flow is located is set to y(e)v. The id is set to IDe, and the start field D of the cell where the subflow is located is stabilized y(e)v. s is set to w, the counter D of the variance field of the cell where the sub-flow is located is stabilized y(e)v. v1 is set to var, which stabilizes the number field D of the cell where the sub-flow is located. y(e)v. n plus 1; Case 2: If there is a cell where the ID of the identify field of the cell where the stable sub-flow is located is the same as the ID of the stable sub-flow, and D y(e)v. s+D y(e)v. n*s+1≠w, which means the cell D where the stable subflow is located y(e)v If there is a break between the stable sub-flow stored in the time window and the stable sub-flow entered, the report <IDe,D y(e)v. s, D y(e)v. s+D y(e)v. n*s> is a stable flow, reset the cell D where the stable sub-flow is located y(e)v , that is, to perform the operation of the initialization phase of the stable sub-stream merger and update it according to the situation 1; among them, D y(e)v. s is the start field of the cell where the stable sub-flow is located; D y(e)v. n is the number field of the cell where the stable sub-flow is located; s+1 is the number of counters in the count field of the stable sub-flow monitor; w is the window number of the starting stable sub-flow; Case 3: If there is a cell where the ID of the identify field of the stable sub-flow is the same as the ID of the stable sub-flow, and D y(e)v. s+D y(e)v. n*s+1=w, which means the cell D where the stable subflow is located y(e)v The stable sub-flow stored in is continuous with the time window of the incoming stable sub-flow, and the cell D where the stable sub-flow is located is accessed. y(e)v The variance field D ua .v c and number field D ua .n, and uses multi-segment sub-stream merging technology to determine whether the stable sub-stream is successfully merged; If the merge is successful, the number field D of the cell where the sub-flow is located will be stabilized. y(e)v. n plus 1, the variance field D of the cell where the sub-flow is located is stabilized y(e)v. v Dy(e)v.n Set to var; when the number field D of the cell where the stable sub-flow is located y(e)v. When n=P, report <ID e , D y(e)v. s, D y(e)v. s+P*s> is a stable flow, reset the cell D where the stable sub-flow is located y(e)v , that is, to perform the initialization phase of the stable sub-stream merger and update it according to the situation; if the merger fails, report the stable stream <IDe,D y(e)v. s, D y(e)v. s+D y(e)v. n*s> and reset the cell D where the stable subflow is located y(e)v , i.e., perform the operations of the initialization phase of the stable sub-stream combiner and update it according to situation 1; Case 4: If there is no cell where the ID of the identify field of the cell where the stable sub-flow is located is the same as the ID of the stable sub-flow, and there is no empty cell, check the nearest window of each cell; if there is D in the nearest window y(e)v. s+D y(e)v. n*s+1≠w, indicating that the stable flow stored in the cell has a window interruption. After reporting the stable flow information in the cell, reset the cell to store the stable sub-flow, that is, perform the operation of the stable sub-flow merger initialization phase and update it according to the situation 1; if the latest window of all cells has D y(e)v. s+D y(e)v. n*s+1=w, then find the minimum number of stable windows W min The cell, calculate the replacement probability T = 1 / (W min -s+1); if the replacement is successful, the stable flow information in the cell is reported and the cell is reset, and the stable sub-flow is stored, that is, the initialization phase operation is performed and updated according to the situation 1; if the replacement fails, the stable sub-flow is ignored.

10. The method for detecting a stable flow for network security monitoring according to claim 9, characterized in that: The use of the multi-segment sub-stream merging technology to determine whether the stable sub-stream is successfully merged specifically includes: Given multiple stable sub-streams with the same ID, the duration window of each stable sub-stream is continuous and of length s, and the frequency variances are var1, var2, ...var n , calculate the variance of the merged stable flow according to the merge variance formula; the merge variance formula is specifically: If the merged variance U≤Φ, the merge is successful; if the merged variance U>Φ, the merge fails; where Φ is the defined variance threshold.

Images