Data stream frequency estimation method based on RNN double model and application

Abstract

The application discloses a data stream frequency estimation method based on an RNN double model. The method combines a popular recurrent neural network model, combines frequency estimation and a neural network, and realizes better error control and more accurate frequency estimation based on an RNN and a Conservative-Update Sketch. The method estimates the frequency of data items in a data stream, classifies the data stream through two RNN neural network models, stores high-frequency data and low-frequency data separately, reduces the hash collision between the high-frequency data and the low-frequency data, thereby improving the accuracy of data stream frequency estimation. Since the high-frequency data is stored separately, the number of hash calculation times of the high-frequency data can be reduced, and the processing efficiency of the data stream is improved to a certain extent. The estimation method is applied to network traffic monitoring, can monitor network burst conditions, and improves the monitoring accuracy and efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of stream data processing, and relates to a data stream frequency estimation method based on RNN double models and application. BACKGROUND

[0002] The frequency estimation of data stream is one of the most important methods in the field of stream data processing, and is widely applied, such as feature selection in machine learning, network traffic monitoring, detection of Dos attack and search engine hot ranking. Conservative-Update Sketch [1] is one of the methods that are widely applied, and the basic idea of the method is to use a small space to store a large amount of frequency by sacrificing part of the estimation accuracy, which belongs to a relatively classic method in the sketch method. However, the existing method based on sketch has the phenomenon that the estimation accuracy is reduced due to hash collision when processing a large amount of data. SUMMARY

[0003] The application provides a data stream frequency estimation method based on RNN double models and application, which combines the popular recurrent neural network model at present, combines frequency estimation and neural network together, and realizes better error control and more accurate frequency estimation based on RNN (recurrent neural network) and Conservative-Update Sketch [1] . The method estimates the frequency of data items in the data stream, classifies the data stream through two RNN neural network models, stores high-frequency data and low-frequency data separately, reduces the hash collision between high-frequency data and low-frequency data, and improves the accuracy of data stream frequency estimation. Since the high-frequency data is stored separately, the number of hash calculations of the high-frequency data can be reduced, and the processing efficiency of the data stream is improved to a certain extent. The estimation method is applied in network traffic monitoring, can monitor the burst situation of the network, and improves the accuracy and efficiency of monitoring.

[0004] To achieve the above purpose, the technical scheme of the application is:

[0005] In the first aspect, the application provides a data stream frequency estimation method based on RNN double models, and the flow of the estimation method is:

[0006] 1) Data acquisition unit: obtain historical data stream, determine the target of historical data stream frequency estimation, mark the data in the historical data stream, the labels of the marking are low-frequency data, global high-frequency data and local high-frequency data, and establish a data set;

[0007] 2) Construct RNN double model

[0008] The RNN dual model includes an RNN-Global model and an RNN-Local model, which are cascaded together. Both RNN models adopt an n-to-one structure. The RNN-Global model is used to determine whether the data is globally high-frequency data, while the RNN-Local model is used to continue predicting data that is not globally high-frequency data as determined by the RNN-Global model, and to determine whether it is locally high-frequency data. If the data is determined to be globally high-frequency data by the RNN-Global model, the RNN dual model is exited; if the data is determined to be low-frequency data by the RNN-Global model, it will then be further determined by the RNN-Local model to determine whether it is locally high-frequency data.

[0009] Using the dataset from step 1), train an RNN dual model to obtain a trained RNN dual model. The data stream to be estimated is classified into three types of data: low frequency, global high frequency, and local high frequency after passing through the trained RNN dual model.

[0010] 3) Data storage update

[0011] The RNN dual-model determines that a piece of data is low-frequency data, and this data is stored in the low-frequency data area in a conservative update manner.

[0012] The RNN dual model determines that a piece of data is a local high-frequency data. This data will be placed into the local bucket of the high-frequency data area. If the data is the same as the data in the local bucket, the frequency in the bucket is incremented by 1. If the local bucket is empty or the data is different from the data in the local bucket, the data is stored in the local bucket with a frequency value of 1, and the timer t is restarted.

[0013] If the RNN dual model determines that the data is of high frequency globally, it traverses all global buckets in the high frequency data region, finds the bucket corresponding to the data, and records the bucket b with the smallest frequency. If a corresponding bucket is found, the frequency is incremented by 1. If no corresponding bucket is found, the data is conservatively updated to the low frequency data region to estimate the frequency g of the data. If there is an empty global bucket, a global bucket is requested to store the data and the estimated frequency value g in the bucket. If there is no empty global bucket, the frequency in bucket b is compared with g to determine whether the trigger condition for the final round of elimination is met. If not, the previous conservative update belongs to the next round of elimination.

[0014] 4) When it is necessary to estimate the frequency of a certain data, first traverse all buckets in the high-frequency data area. If the matching bucket is a global bucket, return the frequency in the bucket. If it is a local bucket, enter the low-frequency data area and use hash calculation to return the minimum value in the mapped bucket and the sum of the values ​​in the local bucket. If it is not in the high-frequency data area, obtain the minimum value in the bucket from the low-frequency data area through calculation, and thus realize the data stream frequency estimation.

[0015] The data types for the final round of elimination are of two kinds: local high-frequency data and global high-frequency data.

[0016] The global high-frequency data must meet the following conditions to be eliminated in the final round:

[0017] ① The global bucket has no empty buckets;

[0018] ②The global high-frequency data that needs to be stored in the high-frequency data area is not in the global bucket;

[0019] ③ The frequency of this global high-frequency data in ② is greater than the minimum frequency in the global bucket.

[0020] Under certain conditions, the final round of elimination is triggered: the global high-frequency data corresponding to the bucket with the lowest frequency in the global bucket will be eliminated from the high-frequency data area and stored in the low-frequency data area. The global bucket vacated after elimination will be allocated to the required global high-frequency data.

[0021] The global high-frequency data eliminated in the last round is first hashed and mapped to certain buckets of Conservative-UpdateSketch in the low-frequency data area. Then, the frequency value stored in each bucket is compared with the frequency of the eliminated data in the last round, and the maximum value of the two is taken as the final result for the bucket. If the global high-frequency data eliminated in the last round is hashed and mapped to multiple buckets, and the frequency value of the data in two or more buckets is less than the frequency value of the global high-frequency data eliminated in the last round, then the frequency value of the data in these two or more buckets is replaced with the frequency value of the global high-frequency data eliminated in the last round.

[0022] The trigger condition for the final elimination of the local high-frequency data is: the final elimination of the local high-frequency data is triggered when the timer in the local bucket reaches zero.

[0023] When the timer of the local bucket reaches zero, the existing local high-frequency data is discarded and stored in the low-frequency data area;

[0024] In the final round of elimination, the local high-frequency data is assigned to buckets in the low-frequency data area via hash mapping. Then, the estimated value d of data in the low-frequency data area is estimated. Finally, the value of d+f is compared with the value in each bucket after hash mapping. If the value in the bucket is less than d+f, the value in the bucket is replaced with d+f; otherwise, no update is performed.

[0025] The trigger condition for the second round of elimination is that although the data is determined to be global high-frequency data and enters the high-frequency data area, the number of global buckets is limited, and the frequency of the data is lower than the frequency of all global high-frequency data in the high-frequency data area. Compared with the elimination conditions of the last round of global high-frequency data, the second round of elimination needs to meet ①② but not condition ③. The data eliminated in the second round will be stored in the low-frequency data area in a conservative update manner.

[0026] Secondly, the present invention provides an application of the above-mentioned data flow frequency estimation method based on the RNN dual model, which is used for network traffic monitoring to monitor network bursts.

[0027] Thirdly, the present invention provides a data flow frequency estimation method for network traffic monitoring, characterized in that the method comprises the following steps:

[0028] 1) Dataset Preprocessing: The format of data tuples in the dataset is set to <target host address, target port number, packet size (unit: bytes), label (0 indicates global high frequency, 1 indicates local high frequency, 2 indicates low frequency data)>. First, the frequency of packets of different sizes in historical network traffic monitoring data is counted. Then, the packets are sorted from largest to smallest according to their frequency, and the top 10% of packets after sorting are taken as global high frequency data. Finally, the matching tuples are labeled with a value of 0 according to the packet size of these high frequency data. At this point, the labeling of global high frequency data is completed.

[0029] Next, labels are added to the local high-frequency data in the dataset. First, a sliding window of size 10 is taken on the data stream and slides sequentially along the data stream. When a data packet of a certain size appears in the window at a frequency greater than or equal to 7, if the matching tuple is not labeled with a 0, then the tuple is labeled with a value of 1. At this point, the labeling of the local high-frequency data is completed.

[0030] Finally, the unlabeled tuples are labeled with a value of 2, indicating that the tuple belongs to low-frequency data. The final label values ​​of 0, 1, and 2 represent global high frequency, local high frequency, and low frequency, respectively. This completes the data set labeling work.

[0031] 2) Construct a dual RNN model

[0032] The RNN dual model includes an RNN-Global model and an RNN-Local model, which are cascaded together. Both RNN models adopt an n-to-one structure. The RNN-Global model is used to determine whether the data is globally high-frequency data, and the RNN-Local model is used to further determine whether the data that is not globally high-frequency data determined by the RNN-Global model is locally high-frequency data. If the data is determined to be globally high-frequency data by the RNN-Global model, it exits the RNN dual model; if it is determined to be low-frequency data by the RNN-Global model, it will then be further determined by the RNN-Local model to determine whether it is locally high-frequency data.

[0033] After the dataset is labeled, the RNN dual model is trained. The first two terms of the tuple are taken as the basic unit of training input. The output of the RNN dual model represents the prediction score for the label values ​​0, 1 and 2. The larger the value, the greater the probability of the corresponding prediction result. After multiple rounds of training and evaluation, the trained RNN dual model is obtained and saved.

[0034] 3) Data storage area partitioning: The data storage area is divided into a high-frequency data area and a low-frequency data area. There are two types of buckets in the high-frequency data area: a global bucket that stores global high-frequency data and a local bucket that stores local high-frequency data. The global buckets are arranged in an array and contain multiple buckets. The buckets in the global bucket are traversed sequentially. The global bucket stores a data and its corresponding frequency. The local bucket has a quantity of 1 and stores the data, the time value t, and the frequency. The low-frequency data area stores data that is determined to be low-frequency by the RNN dual model and "high-frequency data" that has been eliminated from the high-frequency data area. The storage structure of the low-frequency data area is based on Conservative-UpdateSketch—multiple hash functions correspond to multiple bucket sequences, and the value range of the hash function is the length of the bucket sequence.

[0035] 4) Next, the trained RNN dual model is used to process the specific data stream. The host address and port number of the data packet are extracted as the input of the RNN dual model, and the size of the data packet is used as the data in the frame. The RNN dual model determines that the data packet is low frequency, performs hash calculation on the data, and stores it in the low frequency data area in a conservative update manner.

[0036] The RNN dual model identifies data as locally high-frequency data. First, it checks if the data is in the local bucket. If it exists, the frequency of the local bucket is incremented by 1. If it does not exist and the bucket is empty, the data value and the frequency value of 1 are stored in the local bucket, and a new timer is started. When a new locally high-frequency data that is different from the existing locally high-frequency data in the local bucket comes in, the previous locally high-frequency data will be eliminated from the high-frequency data area. At this time, the timer is passively reset to zero, and the final round of elimination of locally high-frequency data is performed. The new locally high-frequency data is put into the local bucket with a frequency value of 1, and a new timer is started.

[0037] If the data is identified as high-frequency data using the RNN dual model, then all global buckets within the high-frequency data region are traversed to find the corresponding bucket and record the bucket b with the lowest frequency. If a corresponding bucket is found, the frequency is incremented by 1. If no corresponding bucket is found, the data is conservatively updated to the low-frequency data region to estimate the frequency g. If an empty global bucket exists, a global bucket is allocated to store the data and the frequency value f. If no empty global bucket exists, the frequency in bucket b is compared with g to determine whether the triggering condition for the final round of elimination is met. If not, the previous conservative update belongs to the next round of elimination.

[0038] 5) After a period of data processing, the statistical information of all data packets is stored in the high-frequency data area and the low-frequency data area. When querying the frequency of data packets of the target size, first traverse all buckets in the high-frequency data area. If the bucket corresponding to the data packet of the target size is found to be a global bucket, the frequency in the bucket is returned. If it is a local bucket, then enter the low-frequency data area and use hash calculation to return the minimum value in the mapped bucket and the sum of the values ​​in the local bucket. Otherwise, enter the low-frequency data area and use hash calculation to return the minimum value in the mapped bucket. This minimum value is the estimated frequency of the data packet of the target size.

[0039] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when loaded by a computer, executes the aforementioned data flow frequency estimation method based on an RNN dual model or the aforementioned data flow frequency estimation method for network traffic monitoring.

[0040] Fifthly, the present invention provides an apparatus comprising:

[0041] The data acquisition unit is used to acquire data streams labeled with low frequency, global high frequency, and local high frequency.

[0042] Classification models are used to classify data streams into low-frequency, global high-frequency, and local high-frequency categories.

[0043] The storage unit is used to store high-frequency data area and low-frequency data separately. The high-frequency data area stores global high-frequency data and local high-frequency data, while the low-frequency data area is used to store low-frequency data and obsolete data.

[0044] The update unit uses a first-round, second-round, and final-round elimination mechanism to update the high-frequency data area and the low-frequency data area;

[0045] The query unit is used to estimate the frequency of the target data.

[0046] Compared with the prior art, the beneficial effects of the present invention are:

[0047] This invention combines the RNN dual model and Conservative-UpdateSketch. [1] By combining the two RNN models, the data is classified and then high-frequency and low-frequency data are stored separately to reduce conflicts between high and low frequency data, thereby improving the accuracy of frequency estimation.

[0048] Using a dual RNN model to predict data streams can accurately classify the data, and storing high-frequency and low-frequency data separately can largely avoid hash collisions between them. Conservative-UpdateSketch... [1] The conservative update strategy in the method further reduces the impact of hash collisions on low-frequency data. The most frequent data in the data stream is almost entirely stored in the high-frequency data area, reducing the number of hash calculations when processing high-frequency data and accelerating the data stream processing efficiency. The final-round elimination mechanism for high-frequency data is one of the core features of this invention, further controlling errors and making frequency estimation more accurate.

[0049] The goal of this invention is to minimize the impact of frequency overestimation caused by hash collisions between data. Dividing data into high-frequency and low-frequency types aims to reduce collisions between them. Conservative update and elimination mechanisms mitigate the impact of collisions between low-frequency data. By suppressing and controlling collisions through these methods, the accuracy of frequency estimation is improved. When this invention is applied to network traffic monitoring, the estimated frequency is more accurate and faster, thus improving the accuracy of network traffic monitoring. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the basic framework of the data flow frequency estimation method based on the RNN dual model of the present invention.

[0051] Figure 2 This is a schematic diagram of the RNN model.

[0052] Figure 3 This is a schematic diagram of the data structure for the high-frequency data area.

[0053] Figure 4 This is a schematic diagram of the data structure for the low-frequency data area.

[0054] Figure 5 This is a schematic diagram of the final round of elimination process for global high-frequency data.

[0055] Figure 6 This is a schematic diagram illustrating the process of placing the final round of elimination data (data3) into the low-frequency data area.

[0056] Figure 7 This is a schematic diagram of the final round of elimination process for local high-frequency data.

[0057] Figure 8 This is a diagram illustrating how the final round of elimination data (data4) is placed in the low-frequency data area.

[0058] Figure 9 This is a schematic diagram of the conservative update process. Detailed Implementation

[0059] The present invention will be further explained below with reference to the embodiments and accompanying drawings, but this is not intended to limit the scope of protection of this application.

[0060] The key points of this invention's technical solution are mainly threefold: First, it classifies data streams into low-frequency data, global high-frequency data, and local high-frequency data using an RNN dual model; second, it stores high-frequency and low-frequency data separately, dividing the data storage area into a high-frequency data area for storing high-frequency data and a low-frequency data area for storing low-frequency data; and third, it employs a data eviction mechanism. These three key points will be elaborated upon below.

[0061] 1. RNN Dual Model:

[0062] Recurrent Neural Networks (RNNs) are widely used in data stream processing because they are effective at predicting data streams with time-series attributes. This application employs an RNN model for data stream prediction. The scheme uses two RNN models to predict globally high-frequency data and locally high-frequency data in the data stream, respectively, classifying the data stream into low-frequency data, globally high-frequency data, and locally high-frequency data.

[0063] The RNN dual model includes an RNN-Global model and an RNN-Local model, which are connected in series. If the data is determined to be high-frequency data globally by the RNN-Global model, it will exit the RNN dual model; if it is determined to be low-frequency data by the RNN-Global model, it will then be determined by the RNN-Local model to determine whether it is local high-frequency data.

[0064] The RNN-Global model is used to predict high-frequency data in a data stream—data that appears frequently in a certain segment of the data stream and remains frequent in the overall data stream.

[0065] The RNN-Local model is used to predict local high-frequency data in a data stream—data that appears frequently in a certain segment of the data stream, but appears infrequently in the overall data stream.

[0066] Both the RNN-Global and RNN-Local models are used to classify data streams. Both RNN models employ an n-to-one structure, as shown below. Figure 2 As shown.

[0067] 2. High-frequency data area and low-frequency data area:

[0068] Data identified as high-frequency by the RNN dual model is stored in the high-frequency data area. Each high-frequency data point is assigned a separate bucket. There are two types of buckets in the high-frequency data area: a global bucket storing all high-frequency data globally, and a local bucket storing local high-frequency data. Although both types of buckets store the frequency of the data, the global bucket stores the actual frequency of all high-frequency data globally, while the local bucket stores only the frequency within the time period required for the timer to reset. The global buckets are arranged in an array. The number of global buckets should not be too large because the bucket traversal is sequential, and too many buckets will impact performance. Therefore, the number is generally around 20-30. Each global bucket stores one data point and its corresponding frequency. There is one local bucket, which stores the data, the timer value 't', and the frequency. The high-frequency data area has the following format: Figure 3 As shown.

[0069] The low-frequency data area stores data that is identified as low-frequency by the RNN dual model and "high-frequency data" that has been discarded from the high-frequency data area. The storage structure of the low-frequency data area is based on Conservative-UpdateSketch. [1] —Multiple hash functions correspond to multiple bucket sequences, and the range of the hash function is the length of the bucket sequence. The structure of the low-frequency data area is as follows: Figure 4 As shown, h is the name of the hash function.

[0070] 3. Elimination Mechanism:

[0071] The process by which the RNN dual model classifies data as low-frequency data is called the first round of elimination; the process by which data that is classified as high-frequency data by the RNN dual model is eliminated because it is not assigned to a bucket after entering the high-frequency data area is called the second round of elimination (the second round of elimination targets global high-frequency data); the process by which data that has been assigned to a bucket after entering the high-frequency data area is replaced after a period of time is called the final round of elimination (the final round of elimination targets both global high-frequency data and local high-frequency data).

[0072] Final round elimination mechanism:

[0073] There are two types of data that are eliminated in the final round: local high-frequency data and global high-frequency data. Global high-frequency data must meet the following conditions to be eliminated:

[0074] ① The global bucket has no empty buckets;

[0075] ②The global high-frequency data that needs to be stored in the high-frequency data area is not in the global bucket;

[0076] ③ The frequency of this global high-frequency data in ② is greater than the minimum value of all frequencies in the global bucket.

[0077] Under the condition that the above three conditions are met, the final round of elimination is triggered—the global high-frequency data corresponding to the bucket with the lowest frequency in the global bucket will be eliminated from the high-frequency data area and stored in the low-frequency data area. The global bucket vacated after elimination will be allocated to the required global high-frequency data. As shown in the example. Figure 5 As shown, the high-frequency global data 'data' is not in the global bucket and there is no empty global bucket. Because the frequency of 'data' is 9, which is greater than the minimum frequency of all frequencies in the global bucket, 'data3', 'data3' is discarded.

[0078] The high-frequency data that is eliminated in the final round is first hashed and mapped to Conservative-UpdateSketch in the low-frequency data area. [1] The algorithm iterates through certain buckets, then sequentially compares the frequency value stored in each bucket with the frequency of the data eliminated in the last round. The maximum value of these two values ​​is taken as the final result for that bucket. A specific example is shown below. Figure 6 As shown, data3 is hashed and mapped to three buckets, with frequency values ​​stored in the three buckets being <5, 8, 7>. Only the value 5 is less than the frequency value 7 in data3. Therefore, the bucket with the final value of 5 is changed to 7, while the values ​​in the other buckets remain unchanged. If the globally high-frequency data removed in the final round is hashed and mapped to multiple buckets, and the frequency values ​​of data in two or more buckets are less than the frequency values ​​of the globally high-frequency data removed in the final round, then the frequency values ​​of the data in these two or more buckets are replaced with the frequency values ​​of the globally high-frequency data removed in the final round.

[0079] The triggering condition for eliminating local high-frequency data is relatively simple. The final round of elimination of local high-frequency data is triggered when the timer in the local bucket reaches zero. There are two ways the timer can reach zero: first, the timer resets to zero when it finishes counting down; second, the timer has not finished counting down (indicating the local bucket is not empty). When new local high-frequency data, different from the data in the local bucket, enters, the timer is passively reset to zero. For example... Figure 7 As shown, when the timer t of the local bucket returns to zero, data4 is evicted and stored in the low-frequency data area.

[0080] In the final round of data elimination, the locally high-frequency data (assuming the eliminated local bucket data is 'data', and the frequency is 'f') is first hashed and mapped to certain buckets in the low-frequency data region using the Conservative-UpdateSketch algorithm. Then, the estimated value 'd' of 'data' in the low-frequency data region is estimated. Finally, the value 'd+f' is compared with the value in each hashed bucket. If the value in the bucket is less than 'd+f', the value in the bucket is replaced with 'd+f'; otherwise, no update is performed. A specific example is shown below. Figure 8 As shown, firstly, the estimated frequency of data4 is estimated to be 1 from the low-frequency data area, that is, d=1. Then, 4 (1+3, where 3 is the frequency f of the local data data4 in the bucket when it is eliminated) is compared with the value on each bucket. If it is less than 4, the value on the bucket is replaced with 4.

[0081] First round of elimination and second round of elimination:

[0082] The first round of elimination is actively triggered by the RNN dual model. The trigger condition for the second round of elimination is that data is determined to be globally high-frequency data and enters the high-frequency data area. However, due to the limited number of global buckets and the fact that the frequency of this data is lower than the frequency of all globally high-frequency data in the high-frequency data area (ensuring that the highest frequency data in the data stream is stored in the high-frequency data area), compared to the final elimination condition for globally high-frequency data, the second round of elimination requires that conditions ① and ② are not met, while condition ③ is not met. Data eliminated in the first and second rounds will be stored in the low-frequency data area using a conservative update method, such as... Figure 9 As shown.

[0083] Conservative updates are based on Conservative-UpdateSketch [1] When data is hashed and mapped to Conservative-UpdateSketch [1] The algorithm maps to some buckets, and then increments the value by 1 for the buckets with the smallest value among all the mapped buckets. These buckets can be a single bucket or multiple buckets; when there are multiple buckets, the values ​​are all equal. For example... Figure 9The data is hashed and mapped to three buckets with values ​​of <4, 7, 2>. Then, the bucket with the minimum value of 2 is incremented by 1, and the value of the bucket containing the minimum value of 2 is updated to 3.

[0084] In summary, the update methods for low-frequency data areas differ between global high-frequency data and local high-frequency data under the final round of data eviction mechanism. The main reason is that the global bucket stores the "true frequency" of the data, while the local bucket only stores a portion of the data's frequency (the frequency within the time period required for the timer to be cleared). This is done because the number of global buckets is much larger than the number of local buckets. Therefore, the final round of data eviction for global high-frequency data is not as frequent as that for local high-frequency data. If the local bucket stored the "true frequency," obtaining the estimated frequency from the low-frequency data area every time new local high-frequency data enters the high-frequency data area would have a significant impact on data stream processing performance.

[0085] The flow of the data flow frequency estimation method based on the RNN dual model of this invention is as follows:

[0086] 1) Data acquisition unit: Obtain historical data streams, determine the target for frequency estimation of historical data streams, label the data in the historical data streams, and label them as low-frequency data, global high-frequency data, and local high-frequency data, and establish a dataset;

[0087] Historical data refers to data that has already been generated. For example, if this method is to process the data stream for October, then the data streams generated in August and September are historical data.

[0088] 2) Constructing a dual RNN model

[0089] The RNN dual model includes an RNN-Global model and an RNN-Local model, which are cascaded together. Both RNN models adopt an n-to-one structure. The RNN-Global model is used to determine whether the data is globally high-frequency data, while the RNN-Local model is used to continue predicting data that is not globally high-frequency data as determined by the RNN-Global model, and to determine whether it is locally high-frequency data. If the data is determined to be globally high-frequency data by the RNN-Global model, the RNN dual model is exited. If the data is determined to be low-frequency data by the RNN-Global model, it will then be processed by the RNN-Local model to determine whether it is locally high-frequency data.

[0090] Using the dataset from step 1), train an RNN dual model to obtain a trained RNN dual model. The data stream to be estimated is classified into three types of data: low frequency, global high frequency, and local high frequency after passing through the trained RNN dual model.

[0091] In this embodiment, IP address and port number are used as input feature vectors for the RNN dual-model, and the output is a prediction result of three types: low frequency, global high frequency, and local high frequency. If the data is determined to be global high frequency by the RNN-Global model, it exits the model; if it is determined to be low frequency by the RNN-Global model, it will be further evaluated by the RNN-Local model. The two models are processed together, and only one prediction result is obtained for the current data. That is, if a data point is determined to be high frequency by the RNN-Global model, then the data is global high frequency data; if it is determined to be low frequency data, then the data may also be determined to be local high frequency data by the RNN-Local model; otherwise, it is low frequency data.

[0092] 6) Data storage update

[0093] The RNN dual-model determines that a piece of data is low-frequency data, and this data is stored in the low-frequency data area in a conservative update manner.

[0094] The RNN dual-model determines that a piece of data is a local high-frequency data. This data will be placed into the local bucket of the high-frequency data area. If the data is the same as the data in the local bucket, the frequency in the bucket will be incremented by 1. If the local bucket is empty or the data is different from the data in the local bucket (in the case that it is not empty, the timer will be passively set to 0 and the final round of elimination will be triggered), the data will be stored in the local bucket with a frequency value of 1, and the timer t will start counting again (the timer t is relatively short, and is generally set according to the data stream processing speed, with 10 data processing times as the basis. If 1000 data are processed per second, then t is set to 10ms).

[0095] If the RNN dual model determines that the data is of high frequency globally, it traverses all global buckets in the high frequency data region, finds the bucket corresponding to the data, and records the bucket b with the smallest frequency. If a corresponding bucket is found, the frequency is incremented by 1. If no corresponding bucket is found, the data is conservatively updated to the low frequency data region to estimate the frequency g of the data. If there is an empty global bucket, a global bucket is requested to store the data and the estimated frequency value g in the bucket. If there is no empty global bucket, the frequency in bucket b is compared with g to determine whether the trigger condition for the final round of elimination is met. If not, the previous conservative update belongs to the next round of elimination.

[0096] The conservative update mentioned here refers to the +1 operation.

[0097] 4) When it is necessary to estimate the frequency of a certain data, first traverse all buckets in the high-frequency data area. If the matching bucket is a global bucket, return the frequency in the bucket. If it is a local bucket, enter the low-frequency data area and use hash calculation to return the minimum value in the mapped bucket and the sum of the values ​​in the local bucket. If it is not in the high-frequency data area, obtain the minimum value in the bucket from the low-frequency data area through calculation, and thus realize the data stream frequency estimation.

[0098] Example:

[0099] This embodiment uses a data flow frequency estimation method for network traffic monitoring, specifically to monitor network bursts and the number of data packets of various sizes being forwarded. Taking the data packet forwarding between two links A and B by a specific ISP (Internet Service Provider) as an example, the goal is to statistically determine the frequency of different sized data packets forwarded from link A to link B within a single day. Then, the frequency of different sized data packets can be estimated from the statistics, such as querying the number of 1MB data packets forwarded within a day.

[0100] Based on historical packet forwarding data, each packet includes the source IP, destination IP (IP includes host address and port number), and packet size, etc., to train an RNN dual model. Generally, the data flow characteristics of the two links of an ISP will not change much in the short term, so historical data can be used to train the RNN dual model. In the example, the destination IP (such as the host address of a certain packet: 127.33.46.88, port number: 8080) is used as the feature vector for training.

[0101] Specific implementation steps:

[0102] 1) Dataset Preprocessing: The format of data tuples in the dataset is set to <target host address, target port number, packet size (unit: bytes), label (0 indicates global high frequency, 1 indicates local high frequency, 2 indicates low frequency data)>, such as <127.33.46.88, 8080, 512, 0>. First, the frequency of packets of different sizes in historical network traffic monitoring data is counted. For example, (512, 9931) means that there are 9931 packets of 512 bytes in the dataset. Then, the packets are sorted from largest to smallest according to their frequency, and the top 10% of packets after sorting are taken as global high frequency data. Finally, the matching tuples are labeled with a value of 0 according to the packet size of these high frequency data. At this point, the labeling of global high frequency data is completed.

[0103] Next, labels are added to the local high-frequency data in the dataset. First, a sliding window of size 10 is taken on the data stream and slides sequentially along the data stream. When a data packet of a certain size appears in the window at a frequency greater than or equal to 7, if the matching tuple is not labeled with a 0, then the tuple is labeled with a value of 1. At this point, the labeling of the local high-frequency data is completed.

[0104] Finally, the unlabeled tuples are labeled with a value of 2, indicating that the tuple belongs to low-frequency data. The final label values ​​of 0, 1, and 2 represent global high frequency, local high frequency, and low frequency, respectively. This completes the labeling of the dataset.

[0105] 2) Construct a dual RNN model

[0106] The RNN dual model includes an RNN-Global model and an RNN-Local model, which are cascaded together. Both RNN models adopt an n-to-one structure. The RNN-Global model is used to determine whether the data is globally high-frequency data, and the RNN-Local model is used to further determine whether the data that is not globally high-frequency data determined by the RNN-Global model is locally high-frequency data. If the data is determined to be globally high-frequency data by the RNN-Global model, it exits the RNN dual model; if the data is determined to be low-frequency data by the RNN-Global model, it will then be further determined by the RNN-Local model to determine whether it is locally high-frequency data.

[0107] After the dataset is labeled, the RNN dual model is trained. The first two terms of each tuple are taken as the basic units of training input. For example, the input form of the tuple <127.33.46.88, 8080, 512, 0> is [[127, 33, 46, 88, 8080]]. Assuming the corresponding output form is [0.7, 0.3, 0.5], where 0.7, 0.3, and 0.5 represent the prediction scores for label values ​​0, 1, and 2, respectively, with larger values ​​indicating a higher probability of the corresponding prediction result. After multiple rounds of training and evaluation, the trained RNN dual model is obtained and saved.

[0108] 3) Data Storage Area Partitioning: The data storage area is divided into a high-frequency data area and a low-frequency data area. There are two types of buckets in the high-frequency data area: a global bucket storing all high-frequency data globally, and a local bucket storing local high-frequency data locally. The global buckets are arranged in an array, containing multiple buckets. Each bucket stores one data point and its corresponding frequency. The local bucket has only one bucket, storing the data, the timer value t, and the frequency. The low-frequency data area stores "high-frequency data" that is determined to be low-frequency by the RNN dual model and has been removed from the high-frequency data area. The storage structure of the low-frequency data area is based on Conservative-UpdateSketch. [1] — Multiple hash functions correspond to multiple bucket sequences, and the range of the hash function is the length of the bucket sequence.

[0109] 4) Next, the trained RNN dual model is used to process the specific data stream. The host address and port number of the data packet are extracted as the input of the RNN dual model, and the size of the data packet is used as the data in the framework. Data packets identified as low-frequency by the RNN dual model are hashed and stored in the low-frequency data area in a conservative update manner.

[0110] The RNN dual model identifies data as locally high-frequency data. First, it checks if the data is in the local bucket. If it exists, the frequency of the local bucket is incremented by 1. If it does not exist and the bucket is empty, the data value and the frequency value of 1 are stored in the local bucket, and a new timer is started. When a new locally high-frequency data that is different from the existing locally high-frequency data in the local bucket comes in, the previous locally high-frequency data will be eliminated from the high-frequency data area. At this time, the timer is passively reset to zero, and the final round of elimination of locally high-frequency data is performed. The new locally high-frequency data is put into the local bucket with a frequency value of 1, and a new timer is started.

[0111] If the data is identified as high-frequency data using the RNN dual model, then all global buckets within the high-frequency data region are traversed to find the corresponding bucket and record the bucket b with the lowest frequency. If a corresponding bucket is found, the frequency is incremented by 1. If no corresponding bucket is found, the data is conservatively updated to the low-frequency data region to estimate the frequency g. If an empty global bucket exists, a global bucket is allocated to store the data and the frequency value f. If no empty global bucket exists, the frequency in bucket b is compared with g to determine whether the triggering condition for the final round of elimination is met. If not, the previous conservative update belongs to the next round of elimination.

[0112] 5) After one day of data processing, the statistical information of all data packets is stored in the high-frequency data area and the low-frequency data area. When querying the frequency of a 1M data packet, first traverse all buckets in the high-frequency data area. If the bucket corresponding to 1M (1M = 1048576 bytes) is found to be a global bucket, the frequency in the bucket is returned. If it is a local bucket, then enter the low-frequency data area and use hash calculation to return the minimum value in the mapped bucket and the sum of the values ​​in the local bucket. Otherwise, enter the low-frequency data area and use hash calculation to return the minimum value in the mapped bucket. This minimum value is the estimated frequency of the 1M data packet.

[0113] In this invention, both RNN and Conservative-UpdateSketch are existing concepts and methods. The conservative update method is a mature update method in Conservative-UpdateSketch, and the RNN model is a type of neural network.

[0114] Any aspects not covered in this invention are applicable to existing technologies.

[0115] [1]EstanC,VargheseG.New directionsintrafficmeasurementandaccounting[J].ACM SIGCOMM Computer Communication Review, 2002, 32(1):75.

Claims

1. A data flow frequency estimation method based on an RNN dual model, characterized in that, The process of this estimation method is as follows: 1) Data acquisition unit: Obtain historical data streams, determine the target for frequency estimation of historical data streams, label the data in the historical data streams, and label them as low-frequency data, global high-frequency data, and local high-frequency data, and establish a dataset; 2) Constructing a dual RNN model The RNN dual model includes an RNN-Global model and an RNN-Local model, which are cascaded together. Both RNN models adopt an n-to-one structure. The RNN-Global model is used to determine whether the data is globally high-frequency data, while the RNN-Local model is used to continue predicting data that is not globally high-frequency data as determined by the RNN-Global model, and to determine whether it is locally high-frequency data. If the data is determined to be globally high-frequency data by the RNN-Global model, the RNN dual model is exited; if the data is determined to be low-frequency data by the RNN-Global model, it will then be further determined by the RNN-Local model to determine whether it is locally high-frequency data. Using the dataset from step 1), train an RNN dual model to obtain a trained RNN dual model. The data stream to be estimated is classified into three types of data: low frequency, global high frequency, and local high frequency after passing through the trained RNN dual model. 3) Data storage update The RNN dual-model determines that a piece of data is low-frequency data, and this data is stored in the low-frequency data area in a conservative update manner. The RNN dual model determines that a piece of data is a local high-frequency data. This data will be placed into the local bucket of the high-frequency data area. If the data is the same as the data in the local bucket, the frequency in the bucket is incremented by 1. If the local bucket is empty or the data is different from the data in the local bucket, the data is stored in the local bucket with a frequency value of 1, and the timer t is restarted. If the RNN dual model determines that the data is of high frequency globally, it traverses all global buckets in the high frequency data region, finds the bucket corresponding to the data, and records the bucket b with the smallest frequency. If a corresponding bucket is found, the frequency is incremented by 1. If no corresponding bucket is found, the data is conservatively updated to the low frequency data region to estimate the frequency g of the data. If there is an empty global bucket, a global bucket is requested to store the data and the estimated frequency value g in the bucket. If there is no empty global bucket, the frequency in bucket b is compared with g to determine whether the trigger condition for the final round of elimination is met. If not, the previous conservative update belongs to the next round of elimination. 4) When it is necessary to estimate the frequency of a certain data, first traverse all buckets in the high-frequency data area. If the matching bucket is a global bucket, return the frequency in the bucket. If it is a local bucket, enter the low-frequency data area and use hash calculation to return the minimum value in the mapped bucket and the sum of the values ​​in the local bucket. If it is not in the high-frequency data area, obtain the minimum value in the bucket from the low-frequency data area through calculation, and thus realize the data stream frequency estimation.

2. The data flow frequency estimation method based on an RNN dual model according to claim 1, characterized in that, The data types for the final round of elimination are of two kinds: local high-frequency data and global high-frequency data. The global high-frequency data must meet the following conditions to be eliminated in the final round: ① The global bucket has no empty buckets; ②The global high-frequency data that needs to be stored in the high-frequency data area is not in the global bucket; ③ The frequency of this global high-frequency data in ② is greater than the minimum frequency in the global bucket; Under certain conditions, the final round of elimination is triggered: the global high-frequency data corresponding to the bucket with the lowest frequency in the global bucket will be eliminated from the high-frequency data area and stored in the low-frequency data area. The global bucket vacated after elimination will be allocated to the required global high-frequency data. The global high-frequency data eliminated in the final round is first hashed and mapped to certain buckets in the Conservative-Update Sketch in the low-frequency data area. Then, the frequency value stored in each bucket is compared with the frequency of the eliminated data in the final round, and the maximum value of the two is taken as the final result for the bucket. If the global high-frequency data eliminated in the final round is hashed and mapped to multiple buckets, and the frequency value of the data in two or more buckets is less than the frequency value of the global high-frequency data eliminated in the final round, then the frequency value of the data in these two or more buckets is replaced with the frequency value of the global high-frequency data eliminated in the final round. The trigger condition for the final elimination of the local high-frequency data is: the final elimination of the local high-frequency data is triggered when the timer in the local bucket reaches zero. When the timer of the local bucket reaches zero, the existing local high-frequency data is discarded and stored in the low-frequency data area; In the final round of elimination, the local high-frequency data is assigned to buckets in the Conservative-Update Sketch via hash mapping in the low-frequency data area. Then, the estimated value d of data in the low-frequency data area is estimated. Finally, the value of d+f is compared with the value in each bucket after hash mapping. If the value in the bucket is less than d+f, the value in the bucket is replaced with d+f; otherwise, no update is performed.

3. The data flow frequency estimation method based on an RNN dual model according to claim 2, characterized in that, The trigger condition for the second round of elimination is that although the data is determined to be global high-frequency data and enters the high-frequency data area, the number of global buckets is limited, and the frequency of the data is lower than the frequency of all global high-frequency data in the high-frequency data area. Compared with the elimination conditions of the last round of global high-frequency data, the second round of elimination needs to meet ①② but not condition ③. The data eliminated in the second round will be stored in the low-frequency data area in a conservative update manner.

4. The data flow frequency estimation method based on an RNN dual model according to claim 2, characterized in that, There are two ways to reset the timer: First, the timer resets to zero when the countdown ends; second, if the timer has not finished counting down, it means that the local bucket is not empty, and when new local high-frequency data that is different from the data in the local bucket comes in, the timer is passively reset to zero.

5. An application of the data flow frequency estimation method based on the RNN dual model as described in any one of claims 1-4, wherein the estimation method is used for network traffic monitoring to monitor network bursts.

6. A method for estimating data stream frequency for network traffic monitoring, characterized in that, The steps of this method are: 1) Dataset Preprocessing: The format of data tuples in the dataset is set to <target host address, target port number, packet size (unit: bytes), label (0 indicates global high frequency, 1 indicates local high frequency, 2 indicates low frequency data)>. First, the frequency of packets of different sizes in historical network traffic monitoring data is counted. Then, the packets are sorted from largest to smallest according to their frequency, and the top 10% of packets after sorting are taken as global high frequency data. Finally, the matching tuples are labeled with a value of 0 according to the packet size of these high frequency data. At this point, the labeling of global high frequency data is completed. Next, labels are added to the local high-frequency data in the dataset. First, a sliding window of size 10 is taken on the data stream and slides sequentially along the data stream. When a data packet of a certain size appears in the window at a frequency greater than or equal to 7, if the matching tuple is not labeled with a 0, then the tuple is labeled with a value of 1. At this point, the labeling of the local high-frequency data is completed. Finally, the unlabeled tuples are labeled with a value of 2, indicating that the tuple belongs to low-frequency data. The final label values ​​of 0, 1, and 2 represent global high frequency, local high frequency, and low frequency, respectively. This completes the data set labeling work. 2) Construct a dual RNN model The RNN dual model includes an RNN-Global model and an RNN-Local model, which are cascaded together. Both RNN models adopt an n-to-one structure. The RNN-Global model is used to determine whether the data is globally high-frequency data, and the RNN-Local model is used to further determine whether the data that is not globally high-frequency data determined by the RNN-Global model is locally high-frequency data. If the data is determined to be globally high-frequency data by the RNN-Global model, it exits the RNN dual model; if it is determined to be low-frequency data by the RNN-Global model, it will then be further determined by the RNN-Local model to determine whether it is locally high-frequency data. After the dataset is labeled, the RNN dual model is trained. The first two terms of the tuple are taken as the basic unit of training input. The output of the RNN dual model represents the prediction score for the label values ​​0, 1 and 2. The larger the value, the greater the probability of the corresponding prediction result. After multiple rounds of training and evaluation, the trained RNN dual model is obtained and saved. 3) Data storage area partitioning: The data storage area is divided into a high-frequency data area and a low-frequency data area. There are two types of buckets in the high-frequency data area: a global bucket that stores global high-frequency data and a local bucket that stores local high-frequency data. The global buckets are arranged in an array and contain multiple buckets. The buckets in the global bucket are traversed sequentially. The global bucket stores a data and its corresponding frequency. The local bucket has one number and stores the data, the time value t, and the frequency. The low-frequency data area stores data that is determined to be low-frequency by the RNN dual model and "high-frequency data" that has been eliminated from the high-frequency data area. The storage structure of the low-frequency data area is based on Conservative-Update Sketch—multiple hash functions correspond to multiple bucket sequences, and the value range of the hash function is the length of the bucket sequence. 4) Next, the trained RNN dual model is used to process the specific data stream. The host address and port number of the data packet are extracted as the input of the RNN dual model, and the size of the data packet is used as the data in the frame. The RNN dual model determines that the data packet is low frequency, performs hash calculation on the data, and stores it in the low frequency data area in a conservative update manner. The RNN dual model identifies data as locally high-frequency data. First, it checks if the data is in the local bucket. If it exists, the frequency of the local bucket is incremented by 1. If it does not exist and the bucket is empty, the data value and the frequency value of 1 are stored in the local bucket, and a new timer is started. When a new locally high-frequency data that is different from the existing locally high-frequency data in the local bucket comes in, the previous locally high-frequency data will be eliminated from the high-frequency data area. At this time, the timer is passively reset to zero, and the final round of elimination of locally high-frequency data is performed. The new locally high-frequency data is put into the local bucket with a frequency value of 1, and a new timer is started. If the data is identified as high-frequency data using the RNN dual model, then all global buckets within the high-frequency data region are traversed to find the corresponding bucket and record the bucket b with the lowest frequency. If a corresponding bucket is found, the frequency is incremented by 1. If no corresponding bucket is found, the data is conservatively updated to the low-frequency data region to estimate the frequency g. If an empty global bucket exists, a global bucket is allocated to store the data and the frequency value f. If no empty global bucket exists, the frequency in bucket b is compared with g to determine whether the triggering condition for the final round of elimination is met. If not, the previous conservative update belongs to the next round of elimination. 5) After a period of data processing, the statistical information of all data packets is stored in the high-frequency data area and the low-frequency data area. When querying the frequency of data packets of the target size, first traverse all buckets in the high-frequency data area. If the bucket corresponding to the data packet of the target size is found to be a global bucket, the frequency in the bucket is returned. If it is a local bucket, then enter the low-frequency data area and use hash calculation to return the minimum value in the mapped bucket and the sum of the values ​​in the local bucket. Otherwise, enter the low-frequency data area and use hash calculation to return the minimum value in the mapped bucket. This minimum value is the estimated frequency of the data packet of the target size.

7. A computer-readable storage medium storing a computer program adapted to execute, when loaded by a computer, the data flow frequency estimation method based on an RNN dual model as described in any one of claims 1-4 or the data flow frequency estimation method for network traffic monitoring as described in claim 5.

8. An electronic device for performing the data flow frequency estimation method based on an RNN dual model as described in any one of claims 1-4, comprising: The data acquisition unit is used to acquire data streams labeled with low frequency, global high frequency, and local high frequency. Classification models are used to classify data streams into low-frequency, global high-frequency, and local high-frequency categories. The storage unit is used to store high-frequency data area and low-frequency data separately. The high-frequency data area stores global high-frequency data and local high-frequency data, while the low-frequency data area is used to store low-frequency data and obsolete data. The update unit uses a first-round, second-round, and final-round elimination mechanism to update the high-frequency data area and the low-frequency data area; The query unit is used to estimate the frequency of the target data.

Images