Layered large flow detection method compatible with programmable hardware

Abstract

The invention relates to the technical field of flow detection, and particularly provides a hierarchical large flow detection method compatible with programmable hardware, which comprises the following steps: according to the flow layer number of storage slots of a storage bucket of a data architecture, putting the storage slots into a corresponding storage slot list, the storage slots are used for storing identification fields of the received to-be-detected traffic, the identification fields are generated according to the number of traffic layers and a traffic label of the to-be-detected traffic, and the number of the storage slot lists is the same as the total number of traffic layers; traversing the storage slots in the storage slot list to obtain attribute fields of the currently traversed storage slots; calculating a condition count value of the to-be-detected traffic according to the attribute field; and under the condition that the condition count value exceeds a preset threshold value, determining the to-be-detected traffic corresponding to the condition count value as the target type traffic. The invention aims to solve the problems of low flow detection efficiency and high difficulty in compatibility with programmable hardware caused by low storage space utilization rate of a hierarchical large flow detection method in the related technology.

Description

Technical Field

[0001] The invention relates to the technical field of flow detection, and in particular to a layered large flow detection method compatible with programmable hardware. Background Art

[0002] Existing hierarchical large flow detection methods can be roughly divided into two categories: tree-based methods (Trie-based) and compact data structure-based methods (sketch-based).

[0003] Tree-based methods usually use a tree storage structure to store the information of each network flow and dynamically track possible hierarchical large flows and remove small flows to reduce space overhead. Such methods need to frequently create and delete tree nodes during the measurement cycle, so the throughput is very low; and the dynamic storage structure cannot be deployed in a programmable switch that only supports static memory allocation.

[0004] Methods based on compact data structures, such as SpaceSaving and MVPipe, allocate a separate compact data structure (sketch) to each layer of network flow. Since the number of layered large flows in each layer is not uniform, such methods will cause a lot of storage space waste.

[0005] On the basis of low space utilization, when the space size of the traffic forwarding device remains consistent, the number of traffic detections performed at the same time will be reduced, which means that the traffic detection efficiency is reduced.

[0006] In addition, there are many restrictions when deploying in programmable switches. Programmable switches usually do not support dynamic memory allocation and only contain a limited number of processing stages in each packet processing pipeline. Each stage only allows one read and write operation, which puts higher requirements on the deployed traffic measurement algorithm, making it difficult for existing traffic detection methods to be compatible with programmable hardware.

[0007] With regard to the layered large traffic detection method in the related technology, there is low utilization of storage space, resulting in low traffic detection efficiency, and it is difficult to be compatible with programmable hardware. No effective solution has been proposed yet. Summary of the invention

[0008] The present invention provides a layered large flow detection method compatible with programmable hardware, which at least solves the problems of low storage space utilization, low throughput and great difficulty in compatibility with programmable hardware in the layered large flow detection method in the related art.

[0009] The present invention provides a layered large flow detection method compatible with programmable hardware, including: placing the storage slots into a corresponding storage slot list according to the traffic layer number of the storage slots of the storage bucket of the data architecture, wherein the storage bucket includes multiple storage slots, and the storage slots are used to store the identification field of the received traffic to be detected, and the identification field is generated according to the traffic layer number and the flow label of the traffic to be detected, and the number of the storage slot list is the same as the total number of traffic layers; traversing the storage slots in the storage slot list to obtain the attribute fields of the currently traversed storage slots, wherein the attribute fields include a key field and a count field; calculating the conditional count value of the traffic to be detected according to the attribute field; when the conditional count value exceeds a preset threshold, determining that the traffic to be detected corresponding to the conditional count value is a target type traffic, wherein the target type traffic is traffic that exceeds a preset traffic threshold.

[0010] As an optional embodiment, the conditional count value of the traffic to be detected is calculated according to the attribute field, including: taking the value of the count field of the storage slot and the total number of traffic layers in the storage slot list where the storage slot is located as the count estimate; subtracting the estimated value of the descendant key in the output set from the count estimate to obtain the conditional count value, wherein the descendant key is the descendant key of the key field of the storage slot, the output set is used to record the target type traffic, and the estimated value of the descendant key is recorded in the attribute parameters of the output set.

[0011] As an optional embodiment, when the condition count value exceeds a preset threshold, after determining that the traffic to be detected corresponding to the condition count value is a target type traffic, the method further includes: adding the target type traffic to the output set, and updating the estimated value of the descendant key of the output set; judging whether the storage slots of the storage bucket have been traversed; when the traversal of the storage slots of the storage bucket has been completed, outputting the output set; when the traversal of the storage slots of the storage bucket has not been completed, executing the step of traversing the storage slots in the storage slot list to obtain the attribute fields of the currently traversed storage slots.

[0012] As an optional embodiment, before placing the storage slot of the storage bucket of the data architecture into the corresponding storage slot list, the method also includes: receiving the traffic to be detected and obtaining the flow label of the data packet of the traffic to be detected; determining the identification field of the traffic to be detected based on the flow label and the randomly generated traffic layer number; determining the storage bucket of the traffic to be detected based on the hash result of the identification field; storing the identification field in the storage slot in the storage bucket, and updating the attribute field of the storage slot.

[0013] As an optional embodiment, the identification field of the traffic to be detected is determined according to the flow label and the randomly generated traffic layer number, including: determining the value range of the traffic layer number according to the traffic stratification method; based on the value range, randomly selecting a random number within the value range as the traffic layer number; selecting a prefix field of a corresponding number of bits in the flow label according to the traffic layer number as the identification field; determining the storage bucket of the traffic to be detected according to the hash result of the flow label, including: performing a hash operation on the identification field through a hash function to obtain the hash result; using the hash result as a storage bucket identifier to search for a corresponding candidate storage bucket, wherein the candidate storage bucket is a virtual bucket provided in advance by the data architecture for the traffic to be detected; and using the candidate storage bucket as the storage bucket of the traffic to be detected.

[0014] As an optional embodiment, the identification field is stored in a storage slot in the storage bucket, and the attribute field of the storage slot is updated, including: determining whether there is a storage slot in the candidate storage bucket that has stored the identification field based on the identification field; if there is a storage slot in the candidate storage bucket that has stored the identification field, adding one to the count field of the storage slot; if there is no storage slot in the candidate storage bucket that has stored the identification field, determining whether there is an empty storage slot in the candidate storage bucket; if there is an empty storage slot in the candidate storage bucket, setting the key field of the empty storage slot to the identification field, setting the layer number field to the number of traffic layers, and setting the count field to one; if there is no empty storage slot in the candidate storage bucket, searching for the storage slot with the smallest count field in the candidate storage bucket, and replacing the identification field with the key field of the storage slot with the smallest count field with a preset probability, wherein the preset probability is the inverse of the minimum value of the count field.

[0015] As an optional embodiment, a right child count field is also provided in each storage slot, which is used to record the number of right children of the key field in the storage slot; after determining the identification field of the traffic to be detected according to the flow label and the randomly generated traffic layer number, the method also includes: determining an odd parameter according to the traffic layer number, wherein, when the traffic layer number is an even number, adding one to the traffic layer number obtains the corresponding odd parameter, and when the traffic layer number is an odd number, the traffic layer number is directly used as the odd parameter; according to the odd parameter, a prefix field of a corresponding number of bits is selected in the flow label as an odd field; according to the odd hash result of the odd field, the storage bucket of the traffic to be detected is determined; according to the identification field, the odd field is stored in the storage slot in the storage bucket, and the attribute field of the storage slot is updated.

[0016] As an optional embodiment, determining the storage bucket of the traffic to be detected based on the odd hash result of the odd field, including: performing a hash operation on the odd field through a hash function to obtain the odd hash result; using the odd hash result as a bucket identifier to search for a corresponding candidate bucket, wherein the candidate bucket is a virtual bucket pre-provided by the data architecture for the traffic to be detected; and using the candidate bucket as the storage bucket of the traffic to be detected.

[0017] As an optional embodiment, according to the identification field, the odd field is stored in the storage slot in the storage bucket, and the attribute field of the storage slot is updated, including: determining whether there is a storage slot in the candidate storage bucket that has already stored the odd field according to the odd field; if there is a storage slot in the candidate storage bucket that has already stored the odd field, adding one to the count field of the storage slot of the odd field, and determining whether the identification field is the right child of the odd field; if there is a storage slot in the candidate storage bucket that has already stored the odd field, and the identification field is the right child of the odd field, adding one to the right child count field of the storage slot; if there is a storage slot in the candidate storage bucket that has already stored the odd field, but the identification field is not the right child of the odd field, ending the operation of storing the identification field in the storage slot in the storage bucket and maintaining the attribute field of the storage slot; if there is no storage slot in the candidate storage bucket that has already stored the identification field, storing the odd field in the candidate storage bucket.

[0018] As an optional embodiment, storing the odd field into the candidate storage bucket includes: determining whether there is an empty storage slot in the candidate storage bucket; if there is an empty storage slot in the candidate storage bucket, setting the key field of the empty storage slot to the odd field, setting the layer number field to the odd parameter, and setting the count field to one; if there is no empty storage slot in the candidate storage bucket, searching for the storage slot with the smallest count field in the candidate storage bucket, and replacing the odd field with the key field of the storage slot with the smallest count field with a preset probability.

[0019] The flow detection method provided by the embodiment of the invention adopts a structure based on a storage bucket and a storage slot. The storage slot is used to store the identification field of the received flow to be detected. The identification field is generated according to the flow layer number and the flow label of the flow to be detected. The flow can be stored according to the number of layers, so that the flow of different layers can share the storage space, improving the space utilization. Based on this, only one update operation is required for each data packet, and each update operation only needs to use a small number of processing stages, so that it can be compatible with deployment in a programmable switch.

[0020] During detection, according to the traffic layer number of the storage slots of the storage bucket of the data architecture, the storage slots are placed in the corresponding storage slot list, the storage slots in the storage slot list are traversed, the attribute fields of the currently traversed storage slots are obtained, and the conditional count value of the traffic to be detected is calculated. Then, when the conditional count value reaches the preset threshold, the corresponding traffic to be detected is determined to be the target type traffic. It is achieved on the basis of higher space utilization to increase the amount of traffic detected in the same time. It solves the problem of low storage space utilization, low detection efficiency, and difficulty in compatibility with programmable hardware in the layered large traffic detection method in the related technology, and achieves the technical effect of reducing the space occupied by traffic, improving space utilization, and then improving detection efficiency, and compatibility with programmable hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

[0022] Figure 1 The present invention is a flowchart of a hierarchical large flow detection method compatible with programmable hardware according to an embodiment of the present invention.

[0023] Figure 2 It is a schematic diagram of the flow update process of the implementation mode created by the present invention.

[0024] Figure 3 It is a schematic diagram of a flow update process using a compression algorithm according to an implementation mode of the present invention.

[0025] Figure 4 It is a schematic diagram of the flow detection query process of the implementation mode created by the present invention.

[0026] Figure 5 It is a schematic diagram of a hierarchical large flow detection device compatible with programmable hardware according to an embodiment of the present invention.

[0027] Figure 6 It is a structural schematic diagram of the electronic device created by the present invention. DETAILED DESCRIPTION

[0028] The embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0029] Network traffic measurement provides key data support for a variety of network applications such as attack detection, load balancing, and traffic engineering. Data packets in the network can be abstracted into different network flows, each of which contains a flow label. The flow label can be defined according to different application needs, usually the source IP address or destination IP address.

[0030] In recent years, heavy hitter detection has attracted widespread attention from researchers. The goal of heavy hitter detection is to find a small number of network flows that contain a large proportion of traffic. This task can help achieve load balancing, traffic engineering, etc. However, only heavy hitter detection cannot complete anomaly detection, especially for distributed denial of service (DDoS) attacks.

[0031] For example, an attacker can control a large number of zombie machines, each of which only sends a small number of packets to the target, but the total number of packets they send can exhaust the resources of the target and make it unable to work normally. In this scenario, more complex traffic measurements are required to identify potential DDoS attacks.

[0032] IP addresses have a hierarchical feature. Each IP address can be hierarchical by byte (5 layers) or by bit (33 layers). Based on this feature, researchers proposed the problem of hierarchical heavy hitter detection. Compared with the ordinary heavy hitter detection problem, hierarchical heavy hitter detection not only finds the heavy hitters whose flow size exceeds the threshold, but also identifies a set of network flows whose total flow size exceeds the threshold, which is identified by the IP address prefix. By performing hierarchical heavy hitter detection, even if each machine in the botnet only sends a small number of data packets, the traffic anomalies of the entire botnet can be identified.

[0033] Like other network traffic measurement tasks, layered large flow detection faces many challenges and difficulties. First, network traffic measurement usually requires the use of special network processing chips, and the storage space of such chips is very limited (less than 15 MiB). In a typical backbone network, the amount of data transmitted per second can reach tens of GB, so it is impossible to store the information of each network flow on the chip.

[0034] The layered large flow detection method in the related art has low storage space utilization, resulting in low flow detection efficiency, and no effective solution has been proposed so far.

[0035] In order to solve the above technical problems, the embodiments of the present invention provide a hierarchical large flow detection method compatible with programmable hardware. Figure 1 is a flowchart of a hierarchical large flow detection method compatible with programmable hardware according to an embodiment of the present invention, such as Figure 1 As shown, the hierarchical large flow detection method compatible with programmable hardware includes the following steps:

[0036] Step S101, according to the traffic layer number of the storage slot of the storage bucket of the data architecture, the storage slot is placed in the corresponding storage slot list, wherein the storage bucket includes multiple storage slots, and the storage slot is used to store the identification field of the received traffic to be detected, and the identification field is generated according to the traffic layer number and the flow label of the traffic to be detected, and the total number of traffic layers of the storage slot list is the same;

[0037] Step S102, traversing the storage slots in the storage slot list to obtain the attribute fields of the currently traversed storage slot, wherein the attribute fields include a key field and a count field;

[0038] Step S103, calculating the condition count value of the flow to be detected according to the attribute field;

[0039] Step S104, when the condition count value exceeds the preset threshold, determine that the to-be-detected traffic corresponding to the condition count value is the target type traffic, wherein the target type traffic is the traffic that exceeds the preset traffic threshold.

[0040] The layered large flow detection method compatible with programmable hardware provided by the embodiment of the invention adopts a structure based on storage buckets and storage slots. The storage slots are used to store the identification field of the received traffic to be detected. The identification field is generated according to the number of traffic layers and the flow label of the traffic to be detected. The traffic can be stored according to the number of layers, so that traffic at different layers can share storage space, improving space utilization. Based on this, only one update operation is required for each data packet, and each update operation only requires a small number of processing stages, so that it can be compatible with deployment in programmable switches.

[0041] During detection, according to the traffic layer number of the storage slots of the storage bucket of the data architecture, the storage slots are placed in the corresponding storage slot list, the storage slots in the storage slot list are traversed, the attribute fields of the currently traversed storage slots are obtained, and the conditional count value of the traffic to be detected is calculated. Then, when the conditional count value reaches the preset threshold, the corresponding traffic to be detected is determined to be the target type traffic. It is achieved on the basis of higher space utilization to increase the amount of traffic detected in the same time. It solves the problem of low storage space utilization, low detection efficiency, and difficulty in compatibility with programmable hardware in the layered large traffic detection method in the related technology, and achieves the technical effect of reducing the space occupied by traffic, improving space utilization, and then improving detection efficiency, and compatibility with programmable hardware.

[0042] The execution subject of the above steps can be a communication device such as a router, a gateway, etc. for traffic transmission and forwarding. The above execution subject can include a memory and a controller, the memory is used to at least store program data running according to the above steps, and the controller can act according to the above steps.

[0043] The above data architecture is also the data architecture of the memory of the above execution subject, and data is stored and read according to the data architecture. The above execution subject can receive and store the traffic passing through the execution subject according to the detection frequency and detection period, and at the end of the detection period, detect the received and stored traffic, and query the traffic data detected as the target type of traffic. Therefore, the above data architecture will also affect the detection of the stored data.

[0044] The above-mentioned target type traffic is the traffic that exceeds the preset traffic threshold, which can also be called large traffic. Large traffic has a large amount of data and requires more resources in the forwarding and recording process. It often requires different communication devices to cooperate. Therefore, the identification of large traffic is a key technology in traffic detection.

[0045] The above data architecture can be a compact data structure sketch, which includes multiple storage buckets, each of which includes multiple storage slots. The storage slot is used to store the identification field of the received traffic to be detected, and the identification field is generated according to the number of traffic layers and the flow label of the traffic to be detected. The above data architecture can enable traffic at different layers to share storage space, thereby improving space utilization.

[0046] When storing the received traffic into the storage bucket and storage slot, the corresponding storage bucket and storage slot are searched according to the identification field to store the traffic. Storing the traffic data is to update the stored traffic, and the specific steps will be described later.

[0047] Based on the above data architecture, when detecting the stored traffic, multiple storage slot lists can be generated first, and the number of storage slot lists is the same as the total number of traffic layers. It should be noted that the value range of the traffic layer number is different in different layering situations. For example, the IP address of the traffic can be layered according to bytes and divided into 5 layers; or layered according to bits and divided into 33 layers. The total number of traffic layers is also the total number of traffic layers.

[0048] According to the traffic layer number of the storage slot of the storage bucket of the data architecture, the storage slot is placed in the corresponding storage slot list. That is, the data content corresponding to the storage slot is placed in the storage slot list. In this way, each storage slot stores the traffic data of the same layer. During detection, it is only necessary to estimate the condition count value of the traffic to be detected through the estimation algorithm based on the attribute field in the storage slot.

[0049] The condition count value can represent the estimated level of the data volume of the traffic to be detected. The higher the condition count value, the larger the data volume of the traffic to be detected. Conversely, the lower the condition count value, the smaller the data volume of the traffic to be detected. The specific calculation method will be described later.

[0050] The calculation of conditional count values ​​mainly depends on the attribute fields of the storage slot, including the key field and the count field. The identification fields of the data packets in the traffic, such as the prefix part of the IP address, can be stored and counted, so the size of the same traffic with the same IP address can be reflected through calculation.

[0051] When the condition count value exceeds the preset threshold, it can be determined that the to-be-detected traffic corresponding to the condition count value is the target type traffic, that is, the large traffic that exceeds the preset traffic threshold.

[0052] As an optional embodiment, the conditional count value of the traffic to be detected is calculated based on the attribute field, including: taking the value of the count field of the storage slot and the total number of traffic layers in the storage slot list where the storage slot is located as the count estimate; subtracting the estimated value of the descendant key in the output set from the count estimate to obtain the conditional count value, wherein the descendant key is the descendant key of the key field of the storage slot, the output set is used to record the target type traffic, and the estimated value of the descendant key is recorded in the attribute parameters of the output set.

[0053] When calculating the conditional count value, the product of the value of the count field of the storage slot and the total number of traffic layers in the storage slot list where the storage slot is located is first used as the count estimate. Since the traffic of different traffic layers corresponds to different fields of the IP address of the traffic data packet, this count estimate can be understood as the theoretical maximum cluster of data packets of the traffic, that is, the maximum value of the traffic.

[0054] Then, the estimated value of the descendant key in the output set is subtracted from the estimated count value to obtain the conditional count value. The descendant key is the descendant key of the key field corresponding to the above count field of the storage slot, which can represent the amount of repeated calculations in the data packets of the traffic, and then estimate the size of the traffic by taking the difference, which is the above conditional count value.

[0055] The output set is used to record the target type traffic that has been found. The estimated value of the descendant key is recorded in the attribute parameter of the output set. Or it can be obtained by querying the output set. It can represent the data packets counted in other traffic.

[0056] Through the above method, the condition count value can be simply and quickly estimated to serve as the basis for flow detection, thereby improving the efficiency of flow detection.

[0057] As an optional embodiment, when the condition count value exceeds a preset threshold, after determining that the traffic to be detected corresponding to the condition count value is a target type traffic, the method also includes: adding the target type traffic to an output set, and updating the estimated value of the descendant key of the output set; judging whether the storage slots of the storage bucket have been traversed; when the storage slots of the storage bucket have been traversed, outputting the output set; when the storage slots of the storage bucket have not been traversed, executing the step of traversing the storage slots in the storage slot list and obtaining the attribute fields of the currently traversed storage slots.

[0058] It should be noted that, when the condition count value does not exceed the preset threshold, it is determined that the to-be-detected traffic corresponding to the condition count value is not the target type traffic.

[0059] Since the received traffic may not be a large traffic, that is, there may be more than one traffic of the target type, after determining whether the traffic to be detected is of the target type according to the condition count value, it is necessary to continue to facilitate the storage slot list until all traffic is detected.

[0060] That is, when the storage slots of the bucket are traversed, the output set is output; when the storage slots of the bucket are not traversed, the storage slots in the storage slot list are traversed, the attribute fields of the currently traversed storage slots are obtained, and the traversal continues.

[0061] However, in the case where the estimated value of the descendant key is recorded by the output set, after adding the target type traffic to the output set, the estimated value of the descendant key of the output set needs to be updated for subsequent calculations.

[0062] As an optional embodiment, before placing the storage slot of the storage bucket of the data architecture into the corresponding storage slot list, the method also includes: receiving the traffic to be detected and obtaining the flow label of the data packet of the traffic to be detected; determining the identification field of the traffic to be detected based on the flow label and the randomly generated traffic layer number; determining the storage bucket of the traffic to be detected based on the hash result of the identification field; storing the identification field in the storage slot in the storage bucket, and updating the attribute field of the storage slot.

[0063] Before placing the storage slots of the storage bucket of the data architecture into the corresponding storage slot list, the stored data may be updated according to the received traffic to be detected.

[0064] The flow label of the data packet of the above-mentioned traffic to be detected is f. The flow label can be defined according to different task requirements. In this embodiment, the source IP address is used as the flow label.

[0065] The identification field of the traffic to be detected is determined based on the flow label and the randomly generated traffic layer number. The prefix F(f,r) of the first r bits of the flow label f ​​can be used as the identification field.

[0066] F(f,r) is used as the input of the hash function to obtain a hash result h(F(f,r)), which is used as the identifier of the assigned candidate bucket as the basis for bucket search. That is, the bucket of the traffic to be detected is determined based on the hash result of the identifier field.

[0067] The compact data structure provides a candidate storage bucket for each flow. In this embodiment, multiple candidate storage locations are allocated for each flow, and an attempt is made to find an empty storage location for storage when inserting. In some embodiments, the identifier of the storage bucket can be a subscript, which is generated when the data structure generates the candidate storage bucket, and the subscript has a one-to-one mapping relationship with the storage bucket.

[0068] Through the above steps, each data packet only needs to be updated once, which improves the throughput while reducing the number of processing stages required by the algorithm and has better hardware compatibility.

[0069] As an optional embodiment, the identification field of the traffic to be detected is determined based on the flow label and the randomly generated traffic layer number, including: determining the value range of the traffic layer number according to the traffic stratification method; based on the value range, randomly selecting a random number within the value range as the traffic layer number; and selecting a prefix field of corresponding number of bits in the flow label according to the traffic layer number as the identification field.

[0070] The flow label of the data packet of the above-mentioned traffic to be detected is f. The flow label can be defined according to different task requirements. In this embodiment, the source IP address is used as the flow label.

[0071] Then a random number r is generated as the traffic layer number. The value range of r varies according to different layering methods. When layering by bytes, the value range of r is 0 to 32; when layering by bits, the value range of r is 0 to 4.

[0072] A prefix field with corresponding bits is selected in the flow label according to the number of traffic layers as the identification field. The prefix F(f,r) of the first r bits of the flow label f ​​can be used as the identification field.

[0073] As an optional embodiment, the storage bucket of the traffic to be detected is determined according to the hash result of the flow label, including: performing a hash operation on the identification field through a hash function to obtain a hash result; using the hash result as the storage bucket identifier to search for the corresponding candidate storage bucket, wherein the candidate storage bucket is a virtual bucket pre-provided by the data architecture for the traffic to be detected; and using the candidate storage bucket as the storage bucket of the traffic to be detected.

[0074] Using F(f,r) as the input of the hash function, a hash result h(F(f,r)) is obtained. This hash result h(F(f,r)) is consistent with the index of an existing storage bucket, or has a one-to-one mapping relationship. That is, the hash function is used to perform a hash operation on the identification field to obtain a hash result.

[0075] The identifier of the candidate bucket to be allocated is used as the basis for searching the bucket to find the corresponding bucket. That is, the hash result is used as the bucket identifier to find the corresponding candidate bucket.

[0076] Then the identification field is stored in the storage slot in the bucket, and the attribute field of the storage slot is updated. The details are as follows.

[0077] As an optional embodiment, the identification field is stored in the storage slot in the storage bucket, and the attribute field of the storage slot is updated, including: determining whether there is a storage slot in the candidate storage bucket that has already stored the identification field based on the identification field; if there is a storage slot in the candidate storage bucket that has already stored the identification field, adding one to the count field of the storage slot; if there is no storage slot in the candidate storage bucket that has already stored the identification field, determining whether there is an empty storage slot in the candidate storage bucket; if there is an empty storage slot in the candidate bucket, setting the key field of the empty storage slot to the identification field, setting the layer number field to the number of traffic layers, and setting the count field to one; if there is no empty storage slot in the candidate storage bucket, searching for the storage slot with the smallest count field in the candidate storage bucket, and replacing the identification field with the key field of the storage slot with the smallest count field with a preset probability, wherein the preset probability is the inverse of the minimum value of the count field.

[0078] After the bucket is determined, multiple storage slots in the bucket are traversed. If a storage slot that has stored F(f,r) information is found during the traversal, the count field in the storage slot is directly increased by 1. That is, if there is a storage slot in the candidate bucket that has stored the identification field, the count field of the storage slot is increased by 1.

[0079] If an empty storage slot is found, the index of the empty storage slot is recorded. After the traversal, if no storage slot that has stored F(f,r) information is found, but the index of the empty storage slot is recorded, F(f,r) is stored in the empty storage slot, that is, the key field of the storage slot is set to F(f,r), the layer number field is set to r, and the count field is set to 1.

[0080] That is, when there is no storage slot in the candidate bucket that has stored the identification field, determine whether there is an empty storage slot in the candidate bucket; when there is an empty storage slot in the candidate bucket, set the key field of the empty storage slot to the identification field, set the layer number field to the number of traffic layers, and set the count field to one.

[0081] If neither the storage slot that has stored the F(f,r) information is found nor the index of the empty storage slot is recorded, the storage slot with the smallest count field value among all storage slots is selected for replacement, and the probability of replacement is the inverse of the smallest count field value.

[0082] That is, when there is no empty storage slot in the candidate storage bucket, the storage slot with the smallest count field in the candidate storage bucket is searched, and the key field of the storage slot with the smallest count field is replaced with the identification field with a preset probability. The preset probability is the inverse of the minimum value of the count field.

[0083] Since the replacement is based on probability, the smaller the count field, the greater the replacement probability, and the larger the count field, the smaller the replacement probability, so the probability of replacing a large flow is minimized. If the replacement is successful, the key field of the storage slot is set to F(f,r), the layer number field is set to r, and the count field value remains unchanged.

[0084] That is, when the key field of the storage slot with the smallest count field is replaced with the identification field, the key field of the storage slot with the smallest count field is replaced with the identification field.

[0085] Through the above strategy, the selection of storage space depends only on the flow label and the identification field related to the layer number, so that data packets of traffic at different layers share the storage space, thereby improving the utilization of storage space.

[0086] As an optional embodiment, a right child count field is also provided in each storage slot, which is used to record the number of right children of the key field in the storage slot; after determining the identification field of the traffic to be detected according to the flow label and the randomly generated traffic layer number, the method also includes: determining an odd parameter according to the traffic layer number, wherein, when the traffic layer number is an even number, the traffic layer number is added by one to obtain the corresponding odd parameter, and when the traffic layer number is an odd number, the traffic layer number is directly used as the odd parameter; according to the odd parameter, a prefix field of the corresponding number of bits is selected in the flow label as the odd field; according to the odd hash result of the odd field, the storage bucket of the traffic to be detected is determined; according to the identification field, the odd field is stored in the storage slot in the storage bucket, and the attribute field of the storage slot is updated.

[0087] The above right child may be the Right-Child field. Specifically, the Right-Child count field is used to record the right child count of the key in the storage slot. For example, if the key field in the storage slot is 10.10.0.0 / 24, then its right child is 10.10.0.1 / 25. It should be noted that in some other embodiments, it may also be the Left-Child field. These are all concepts in the tree data structure.

[0088] In the case where the right child field exists, the steps and processes for updating the stored data will also change according to the received traffic to be detected. The details are as follows.

[0089] After determining the identification field of the traffic to be detected based on the flow label and the randomly generated traffic layer number, the odd parameter is also determined based on the traffic layer number. Specifically, when the traffic layer number is an even number, the traffic layer number is added by one to obtain the corresponding odd parameter. When the traffic layer number is an odd number, the traffic layer number is directly used as the odd parameter.

[0090] That is, the value of the odd parameter m is determined by the number of traffic layers r. If the number of traffic layers r is an even number, the odd parameter m=r+1; otherwise, the odd parameter m=r.

[0091] According to the odd parameter, a prefix field of the corresponding number of bits is selected in the flow label as the odd field. The prefix F(f,m) of the first m bits of the flow label f ​​can be used as the odd field.

[0092] According to the odd hash result of the odd field, the storage bucket of the traffic to be detected is determined. The odd field F(f,m) is used as the input of the hash function to obtain an odd hash result h(F(f,m)). The location of the storage bucket is determined based on the odd hash result h(F(f,m)).

[0093] As an optional embodiment, the storage bucket of the traffic to be detected is determined according to the odd hash result of the odd field, including: performing a hash operation on the odd field through a hash function to obtain an odd hash result; using the odd hash result as a bucket identifier to search for a corresponding candidate bucket, wherein the candidate bucket is a virtual bucket pre-provided by the data architecture for the traffic to be detected; and using the candidate bucket as the storage bucket for the traffic to be detected.

[0094] Using F(f,m) as the input of the hash function, we get an odd hash result h(F(f,m)), which is consistent with the index of an existing bucket, or has a one-to-one mapping relationship. That is, we get the hash result by performing a hash operation on the identification field through the hash function.

[0095] The identifier of the candidate bucket to be allocated is used as the basis for searching the bucket to find the corresponding bucket. That is, the hash result is used as the bucket identifier to find the corresponding candidate bucket.

[0096] Then, according to the identification field, the odd field is stored in the storage slot in the storage bucket, and the attribute field of the storage slot is updated. The details are as follows.

[0097] As an optional embodiment, according to the identification field, the odd field is stored in the storage slot in the storage bucket, and the attribute field of the storage slot is updated, including: determining whether there is a storage slot in the candidate storage bucket that has already stored the odd field according to the odd field; if there is a storage slot in the candidate storage bucket that has already stored the odd field, adding one to the count field of the odd field storage slot, and determining whether the identification field is the right child of the odd field; if there is a storage slot in the candidate storage bucket that has already stored the odd field, and the identification field is the right child of the odd field, adding one to the right child count field of the storage slot; if there is a storage slot in the candidate storage bucket that has already stored the odd field, but the identification field is not the right child of the odd field, ending the operation of storing the identification field in the storage slot in the storage bucket, and maintaining the attribute field of the storage slot; if there is no storage slot in the candidate storage bucket that has already stored the identification field, the odd field is stored in the candidate storage bucket.

[0098] After calculating the index of the bucket, that is, the odd field F(f,m), determine whether there is a storage slot in the corresponding candidate bucket that has stored the odd field F(f,m). If there is a storage slot in the bucket that has stored the odd field F(f,m), directly increase the count value of the storage slot by 1. And determine whether the identification field F(f,r) is the right child of the odd field F(f,m).

[0099] That is, when there is a storage slot in the candidate storage bucket that has stored an odd field, the count field of the odd field storage slot is increased by one, and it is determined whether the identification field is the right child of the odd field.

[0100] If the identification field F(f,r) is the right child of the odd field F(f,m), the right child count of the storage slot is increased by 1. If the identification field F(f,r) is not the right child of the odd field F(f,m), the update operation is terminated.

[0101] That is, if there is a storage slot in the candidate storage bucket that has stored an odd field, and the identification field is the right child of the odd field, the right child count field of the storage slot is increased by 1. If there is a storage slot in the candidate storage bucket that has stored an odd field, but the identification field is not the right child of the odd field, the operation of storing the identification field into the storage slot in the storage bucket is terminated, and the attribute field of the storage slot is maintained.

[0102] If there is no storage slot in the candidate storage bucket that has stored the identification field, the odd field is stored in the candidate storage bucket. The details are as follows.

[0103] As an optional embodiment, storing an odd field in a candidate storage bucket includes: determining whether there is an empty storage slot in the candidate storage bucket; if there is an empty storage slot in the candidate storage bucket, setting the key field of the empty storage slot to an odd field, setting the layer number field to an odd parameter, and setting the count field to one; if there is no empty storage slot in the candidate storage bucket, searching for the storage slot with the smallest count field in the candidate storage bucket, and replacing the odd field with the key field of the storage slot with the smallest count field with a preset probability.

[0104] If there is no storage slot in the current bucket that has stored the odd field F(f,m), determine whether there is an empty storage slot in the current bucket. If there is an empty storage slot, directly set the key of the storage slot to the odd field F(f,m), the number of layers to m, and the count value to 1.

[0105] That is, determine whether there is an empty storage slot in the candidate bucket; if there is an empty storage slot in the candidate bucket, set the key field of the empty storage slot to an odd field, set the layer number field to an odd parameter, and set the count field to one.

[0106] There is no storage slot in the bucket that has already stored F(f,r) and there is no empty storage slot. Find the storage slot with the smallest count value in the bucket and replace the key of the storage slot with F(f,m) with the probability of the inverse of the count value.

[0107] That is, when there is no empty storage slot in the candidate storage bucket, the storage slot with the smallest count field in the candidate storage bucket is searched, and the key field of the storage slot with the smallest count field is replaced with the odd field with a preset probability.

[0108] In the process of storing traffic into storage slots according to identification fields, this embodiment stores identification fields of multiple traffic in one bucket by referring to the count of the right child, thereby achieving a certain degree of data compression and further improving memory usage efficiency.

[0109] It should be noted that this embodiment also provides an optional implementation method, which is described in detail below. This implementation method provides a layered large flow detection method that can achieve high memory efficiency and hardware compatibility, aiming to use a single compact data structure to store network flow information in different layers to improve space utilization. At the same time, the present invention relates to a sampling-based update method, which only requires one update operation for each data packet, so each update operation only requires a small number of processing stages and can be deployed in a programmable switch.

[0110] This implementation scheme designs a layered large flow detection method that can achieve high memory efficiency and hardware compatibility, and uses a single compact data structure to store network flow information in different layers, thereby achieving higher space utilization and better hardware compatibility.

[0111] The key points of this implementation are: (1) by designing a single compact data structure based on a bucket structure, network flows at different layers share storage space, thereby improving space utilization; (2) a sampling-based update method is designed, which only needs to be updated once for each data packet, thereby improving throughput while reducing the number of processing stages required by the algorithm and having better hardware compatibility; (3) a compression method is designed, which further improves memory usage efficiency by storing information of multiple network flows or prefixes in one bucket.

[0112] The data structure of this embodiment mainly includes an initialization module, an update module and a query module. The initialization module is responsible for initializing the compact data structure according to user needs. The update module performs an insert operation on the arriving data packet and inserts the flow elements contained in the data packet into the compact data structure; the query module is responsible for providing a query function for layered large flows after the measurement cycle ends. Users can use this module to obtain the flow label and corresponding flow size of the detected layered large flow. The detailed technical solution is as follows:

[0113] This implementation provides an initialization module that can initialize the compact data structure according to user needs:

[0114] (1) Initialize the number of buckets of the compact data structure: The compact data structure has w buckets.

[0115] (2) Initialize each bucket: Each bucket contains k storage slots. The compact data structure provides a candidate storage bucket for each network flow or prefix. Therefore, this implementation allocates multiple candidate storage locations for each network flow or prefix, and tries to find an empty storage location for storage during insertion.

[0116] (3) Initialize each storage slot: Each storage slot contains three fields: key, level, and count. During initialization, the key, level, and count fields are all set to 0.

[0117] This embodiment provides an update module. The update module is responsible for performing an update operation of the compact data structure on each incoming data packet, and its specific implementation is as follows:

[0118] (1) Extract the flow label from the data packet.

[0119] Each data packet includes a flow label f, and the flow label can be defined according to different task requirements. The present invention uses the source IP address as the flow label.

[0120] (2) Determine the bucket for storing the flow based on the hash result of the flow label.

[0121] When determining the storage bucket, first generate a random number r, and then select the prefix F(f,r) of the flow label f ​​as the label to be updated. The value range of r varies according to different stratification methods. When stratified by bytes, the value range of r is 0 to 32; when stratified by bits, the value range of r is 0 to 4.

[0122] After determining the tag to be updated, it is used as the input of the hash function to obtain a hash result h(F(f,r)), which is used as the subscript of the bucket. By randomly selecting a prefix for updating, the present invention achieves high update throughput and better hardware compatibility.

[0123] (3) Find an available storage slot in the storage bucket for storage or replace a non-empty storage slot;

[0124] After determining the storage bucket, traverse the k storage slots in the storage bucket. During the traversal process, if a storage slot that has stored F(f,r) information is found, the count field in the storage slot is directly increased by 1; if an empty storage slot is found, the index of the empty storage slot is recorded.

[0125] After the traversal is completed, if the storage slot that has stored the F(f,r) information is not found, but the subscript of the empty storage slot is recorded, F(f,r) is stored in the empty storage slot, that is, the key field of the storage slot is set to F(f,r), the layer number field is set to r, and the count field is set to 1.

[0126] If neither the storage slot that has stored the F(f,r) information is found nor the index of the empty storage slot is recorded, the storage slot with the smallest count field value among all storage slots is selected for replacement, and the probability of replacement is the inverse of the smallest count field value.

[0127] If the replacement is successful, set the key field of the storage slot to F(f,r), set the layer number field to r, and keep the count field value unchanged.

[0128] This embodiment further provides an update module using a compression algorithm. The use of a compression algorithm can improve the space utilization and detection accuracy of the present invention in the layered large flow detection based on bit layers. The update module using the compression algorithm includes the following changes:

[0129] (1) In each storage slot, a right-child count field is added to record the right child count of the key in the storage slot. For example, if the key in the storage slot is 10.10.0.0 / 24, then its right child is 10.10.0.1 / 25.

[0130] (2) When determining the storage bucket, a random number r is also generated first, and then the prefix F(f,m) of the flow label f ​​is selected as the label to be updated. The value of m is determined by r. If r is an even number, m=r+1; otherwise, m=r. Then F(f,m) is used as the input of the hash function to determine the location of the storage bucket.

[0131] (3) After the storage bucket is determined, the process of finding the storage slot is the same as not using the compression algorithm. However, when updating the value of the count field after the storage slot is determined, it is necessary to additionally determine whether F(f,r) is the right child of F(f,m). If so, it is necessary to additionally update the right child count field of the storage slot; otherwise, no additional operation is performed.

[0132] This implementation provides a query module that queries and detects large traffic and outputs:

[0133] (1) First, construct a list of storage slots A0, A1, ..., A with the same number of layers as the total number of layers. H-1 , and initialize these storage slot lists to empty. Then traverse each bucket in the compact data structure, and add each storage slot in each bucket to the corresponding storage slot list according to the layer number field.

[0134] (2) Initialize an empty output set.

[0135] (3) Starting from the storage slot list with subscript 0, traverse each storage slot list in ascending order of the subscript. When traversing each storage slot list, first take out the key field in the storage slot, then take out the count field in the storage slot, and multiply the count field by the total number of layers as the key count estimate.

[0136] Since the criterion for determining whether a key is a layered large flow is whether its condition count value exceeds a threshold, it is necessary to additionally calculate the condition count value of a key.

[0137] The conditional count for a key is calculated by subtracting the count estimates of all keys in the output set that are descendants of the key from the count estimate. If the conditional count for a key exceeds a given threshold, it is added to the output set.

[0138] (4) If the update module with compression algorithm is used, the query module also needs to be modified. When initializing the storage slot list, each storage slot will be transformed into three storage slots. In addition to the original storage slot, assuming that the key in the original storage slot is key, the level is r, the count value is c, and the right child count is rc, then the keys of the additional two storage slots are the right child and left child of key, the level is r-1, and the count values ​​are rc and c-rc respectively. The rest of the steps of the update module remain unchanged.

[0139] Specifically, an embodiment of the present invention proposes a layered large flow detection method that is space-efficient and compatible with programmable hardware. In order to describe the method more clearly, a more detailed introduction is given below with reference to the accompanying drawings.

[0140] Figure 2 It is a schematic diagram of the flow update process of the implementation mode created by the present invention, such as Figure 2 As shown, the specific implementation steps of the update operation in this embodiment are as follows:

[0141] S11: Obtain the data packet f to be inserted.

[0142] S12: Randomly select a prefix of f or f itself F(f,r), where r is a random number and F(f,r) represents the prefix of f at the rth level.

[0143] S13: Calculate the bucket index using the hash function: h(F(f,r));

[0144] S14: After calculating the index of the storage bucket, determine whether there is a storage slot in the corresponding candidate storage bucket that has already stored F(f,r). If yes, execute S15; otherwise, execute S16.

[0145] S15: If there is a storage slot in the storage bucket that has already stored F(f,r), the count value of the storage slot is directly increased by 1.

[0146] S16: If there is no storage slot in the current storage bucket that has stored F(f,r), determine whether there is an empty storage slot in the current storage bucket. If yes, execute S17; otherwise, execute S18.

[0147] S17: If there is an empty storage slot, directly set the key of the storage slot to F(f,r), the number of layers to r, and the count value to 1.

[0148] S18: There is no storage slot in the bucket that has already stored F(f,r) and there is no empty storage slot. Find the storage slot with the smallest count value in the bucket and replace the key of the storage slot with F(f,r) with a probability of the inverse of the count value.

[0149] Figure 3 Schematic diagram of a flow update process using a compression algorithm according to an embodiment of the present invention. Figure 3 As shown, the specific implementation steps of the update operation using the compression algorithm in this embodiment are as follows:

[0150] S201: Obtain a data packet f to be inserted.

[0151] S202: Randomly select a prefix of f or f itself F(f,r), where r is a random number and F(f,r) represents the prefix of f at the rth level.

[0152] S203: Calculate the bucket index using a hash function: h(F(f,m)), if r is an even number, m=r+1; otherwise, m=r.

[0153] S204: After calculating the index of the storage bucket, determine whether there is a storage slot in the corresponding candidate storage bucket that has already stored F(f,m). If yes, execute S25; otherwise, execute S28.

[0154] S205: If there is a storage slot in the storage bucket that has already stored F(f,m), the count value of the storage slot is directly increased by 1, and S26 is executed.

[0155] S206: Determine whether F(f, r) is the right child of F(f, m). If so, execute S27; otherwise, end the update operation.

[0156] S207: If F(f,r) is the right child of F(f,m), the right child count of the storage slot is increased by 1.

[0157] S208: If there is no storage slot in the current storage bucket that has stored F(f,m), determine whether there is an empty storage slot in the current storage bucket. If yes, execute S209; otherwise, execute S210.

[0158] S209: If there is an empty storage slot, directly set the key of the storage slot to F(f,m), the number of layers to m, and the count value to 1.

[0159] S210: There is no storage slot in the bucket that has already stored F(f, r) and there is no empty storage slot. Find the storage slot with the smallest count value in the bucket and replace the key of the storage slot with F(f, m) with a probability of the inverse of the count value.

[0160] Figure 4 It is a schematic diagram of the flow detection query process of the implementation mode created by the present invention, such as Figure 4 As shown, the specific implementation steps of the query operation in this embodiment are as follows:

[0161] S301: Initialize H empty storage slot lists, where H is the total number of layers.

[0162] S302: traverse each storage slot in all buckets, and put them into the corresponding storage slot list according to the number of layers of the storage slot. If a compression algorithm is used, each storage slot should be converted into three new storage slots according to the conversion method described in the invention content and then put into the storage slot list respectively.

[0163] S303: Initialize an empty output set.

[0164] S304: Start traversing from the storage slot list with index 0.

[0165] S305: For each storage slot traversed, first take out the key therein, and then calculate its count estimate by multiplying the count value by the total number of layers.

[0166] S306: Calculate the conditional count value by subtracting the count estimate of all the keys in the output set that are descendants of the key from the count estimate.

[0167] S307: Determine whether the calculated condition count value is greater than a threshold value. If so, execute S308; otherwise, execute 309.

[0168] S308: If it is determined whether the calculated condition count value is greater than the threshold, the key is added to the output set and S309 is executed.

[0169] S309: Determine whether all storage slots have been traversed. If so, execute S310; otherwise, traverse the next storage slot and execute S305.

[0170] S310: After traversing all storage slots, an output set is returned.

[0171] The algorithm proposed in this embodiment no longer uses multiple compact data structures in traditional layered large flow detection solutions. While improving space utilization, it is also very friendly to programmable hardware (such as FPGAs and P4 chips) because all network flow and prefix information are stored in a compact data structure, and each update uses a sampling method to perform only one memory access, which greatly reduces the processing stages required for deployment on programmable hardware.

[0172] In addition, the embodiment of the present invention also designs a compression algorithm based on the right child count field, which can store the identification fields of multiple storage slots in one storage slot, further improving the memory usage efficiency and the layered large flow detection accuracy.

[0173] Figure 5 Schematic diagram of a hierarchical large flow detection device compatible with programmable hardware according to an embodiment of the present invention. Figure 5 As shown, based on the above-mentioned layered large flow detection method compatible with programmable hardware provided by the embodiment of the invention, the embodiment of the invention also provides a layered large flow detection device compatible with programmable hardware, which is applied to layered large flow detection. The device includes: a storage slot extraction module 501, an attribute field acquisition module 502, a count value calculation module 503, and a target flow determination module 504. The device is described in detail below.

[0174] The storage slot extraction module 501 is used to put the storage slot into the corresponding storage slot list according to the traffic layer number of the storage slot of the storage bucket of the data architecture, wherein the storage bucket includes multiple storage slots, and the storage slot is used to store the identification field of the received traffic to be detected, and the identification field is generated according to the traffic layer number and the flow label of the traffic to be detected. The number of storage slot lists is the same as the total number of traffic layers.

[0175] The attribute field acquisition module 502 is connected to the storage slot extraction module 501 and is used to traverse the storage slots in the storage slot list and obtain the attribute fields of the currently traversed storage slot, wherein the attribute fields include a key field and a count field.

[0176] The count value calculation module 503 is connected to the attribute field acquisition module 502 and is used to calculate the condition count value of the traffic to be detected according to the attribute field.

[0177] The target flow determination module 504 is connected to the above-mentioned count value calculation module 503, and is used to determine that the flow to be detected corresponding to the conditional count value is the target type flow when the conditional count value exceeds the preset threshold, wherein the target type flow is the flow that exceeds the preset flow threshold.

[0178] The hierarchical large flow detection device compatible with programmable hardware provided by the embodiment of the invention adopts a structure based on storage buckets and storage slots. The storage slots are used to store the identification field of the received traffic to be detected. The identification field is generated according to the number of traffic layers and the flow label of the traffic to be detected. The traffic can be stored according to the number of layers, so that traffic at different layers can share storage space, thereby improving space utilization. Based on this, only one update operation is required for each data packet, and each update operation only requires a small number of processing stages, so that it can be compatible with deployment in a programmable switch.

[0179] During detection, according to the traffic layer number of the storage slots of the storage bucket of the data architecture, the storage slots are placed in the corresponding storage slot list, the storage slots in the storage slot list are traversed, the attribute fields of the currently traversed storage slots are obtained, and the conditional count value of the traffic to be detected is calculated. Then, when the conditional count value reaches the preset threshold, the corresponding traffic to be detected is determined to be the target type traffic. It is achieved on the basis of higher space utilization to increase the amount of traffic detected in the same time. It solves the problem of low storage space utilization, low detection efficiency, and difficulty in compatibility with programmable hardware in the layered large traffic detection method in the related technology, and achieves the technical effect of reducing the space occupied by traffic, improving space utilization, and then improving detection efficiency, and compatibility with programmable hardware.

[0180] The embodiments of the present invention also provide a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiments of the present invention.

[0181] The embodiments of the present invention further provide a computer program product, including a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to execute the method of the embodiments of the present invention.

[0182] The embodiment of the invention also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform the method of the embodiment of the invention when executed by the at least one processor.

[0183] refer to Figure 6, a block diagram of an electronic device that can be used as a server or client of an embodiment of the invention will now be described, which is an example of hardware devices that can be applied to various aspects of the invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the invention described and / or required herein.

[0184] like Figure 6 As shown, the electronic device includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0185] Multiple components in the electronic device are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 can be any type of device that can input information to the electronic device, and the input unit 606 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit 607 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 can include but is not limited to a disk, an optical disk. The communication unit 609 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, and / or a wireless communication transceiver, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0186] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a CPU, a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the method embodiments created by the present invention may be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via a ROM 602 and / or a communication unit 609. In some embodiments, the computing unit 601 may be configured to perform the above-described method in any other appropriate manner (e.g., by means of firmware).

[0187] The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the computer program, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0188] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0189] It should be noted that the term "including" and its variations used in the embodiments of the present invention are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0190] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0191] The various steps described in the method implementation methods provided by the embodiments of the present invention can be performed in different orders and / or in parallel. In addition, the method implementation methods may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0192] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same and similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

[0193] The above-described embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of protection. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A hierarchical large flow detection method compatible with programmable hardware, characterized in that: include: According to the number of traffic layers of the storage slots of the storage bucket of the data architecture, the storage slots are placed in a corresponding storage slot list, wherein the storage bucket includes a plurality of storage slots, the storage slots are used to store an identification field of the received traffic to be detected, the identification field is generated according to the number of traffic layers and a flow label of the traffic to be detected, and the number of the storage slot list is the same as the total number of traffic layers; Traversing the storage slots in the storage slot list to obtain the attribute fields of the currently traversed storage slot, wherein the attribute fields include a key field and a count field; Calculate the conditional count value of the flow to be detected according to the attribute field; In the case where the condition count value exceeds a preset threshold, it is determined that the to-be-detected traffic corresponding to the condition count value is a target type traffic, wherein the target type traffic is traffic that exceeds a preset traffic threshold.

2. The method according to claim 1, characterized in that Calculating the conditional count value of the flow to be detected according to the attribute field includes: The product of the value of the count field of the storage slot and the total number of flow layers of the storage slot list where the storage slot is located is used as the count estimate value; The conditional count value is obtained by subtracting the estimated value of the descendant key in the output set from the count estimate, wherein the descendant key is the descendant key of the key field of the storage slot, the output set is used to record the target type traffic, and the estimated value of the descendant key is recorded in the attribute parameters of the output set.

3. The method according to claim 2, characterized in that When the condition count value exceeds the preset threshold, after determining that the to-be-detected traffic corresponding to the condition count value is the target type traffic, the method further includes: Add the target type traffic to the output set, and update the estimated value of the descendant key of the output set; Determine whether the storage slots of the storage bucket have been traversed; When the storage slot traversal of the storage bucket is completed, outputting the output set; In the case that the storage slots of the storage bucket have not been traversed completely, a step of traversing the storage slots in the storage slot list to obtain the attribute fields of the currently traversed storage slot is performed.

4. The method according to any one of claims 1 to 3, characterized in that Before placing the storage slots of the storage buckets of the data architecture into the corresponding storage slot list, the method further includes: Receiving the traffic to be detected, and obtaining the flow label of the data packet of the traffic to be detected; Determining the identification field of the to-be-detected traffic according to the flow label and the randomly generated traffic layer number; Determine the storage bucket of the traffic to be detected according to the hash result of the identification field; The identification field is stored in a storage slot in the storage bucket, and the attribute field of the storage slot is updated.

5. The method according to claim 4, characterized in that Determining, according to the flow label and the randomly generated flow layer number, an identification field of the flow to be detected, including: Determine the value range of the number of traffic layers according to the traffic stratification method; Based on the value range, randomly selecting a random number within the value range as the traffic layer number; According to the traffic layer number, a prefix field with a corresponding number of bits is selected in the flow label as the identification field; Determining a storage bucket of the to-be-detected traffic according to a hash result of the flow label includes: Performing a hash operation on the identification field by using a hash function to obtain the hash result; Using the hash result as a bucket identifier, searching for a corresponding candidate bucket, wherein the candidate bucket is a virtual bucket pre-provided by the data architecture for the traffic to be detected; The candidate storage bucket is used as the storage bucket of the traffic to be detected.

6. The method according to claim 4, characterized in that Storing the identification field in a storage slot in the storage bucket and updating the attribute field of the storage slot includes: Determine, according to the identification field, whether there is a storage slot in the candidate storage bucket that has stored the identification field; If there is a storage slot in the candidate storage bucket that has stored the identification field, increment the count field of the storage slot by one; If no storage slot in the candidate storage bucket has stored the identification field, determining whether there is an empty storage slot in the candidate bucket; In the case where there is an empty storage slot in the candidate storage bucket, the key field of the empty storage slot is set to the identification field, the layer number field is set to the traffic layer number, and the count field is set to one; When there is no empty storage slot in the candidate storage bucket, find the storage slot with the smallest count field in the candidate storage bucket, and replace the identification field with the key field of the storage slot with the smallest count field with a preset probability, wherein the preset probability is the inverse of the minimum value of the count field.

7. The method according to claim 4, characterized in that Each storage slot is also provided with a right child count field for recording the number of right children of the key field in the storage slot; After determining the identification field of the to-be-detected traffic according to the flow label and the randomly generated traffic layer number, the method further includes: Determine an odd parameter according to the number of traffic layers, wherein, when the number of traffic layers is an even number, add one to the number of traffic layers to obtain a corresponding odd parameter, and when the number of traffic layers is an odd number, use the number of traffic layers directly as the odd parameter; According to the odd parameter, a prefix field of a corresponding number of bits is selected in the flow label as an odd field; Determine a storage bucket for the traffic to be detected according to the odd hash result of the odd field; According to the identification field, the odd field is stored in a storage slot in the storage bucket, and the attribute field of the storage slot is updated.

8. The method according to claim 7, characterized in that Determining a storage bucket of the traffic to be detected according to the odd hash result of the odd field includes: Performing a hash operation on the odd field using a hash function to obtain the odd hash result; Using the odd hash result as a bucket identifier, searching for a corresponding candidate bucket, wherein the candidate bucket is a virtual bucket pre-provided by the data architecture for the traffic to be detected; The candidate storage bucket is used as the storage bucket of the traffic to be detected.

9. The method according to claim 7, characterized in that: According to the identification field, storing the odd field into a storage slot in the storage bucket, and updating the attribute field of the storage slot, including: Determine, according to the odd field, whether there is a storage slot in the candidate storage bucket that has stored the odd field; In the case that there is a storage slot in the candidate storage bucket that has stored the odd field, incrementing the count field of the storage slot of the odd field by one, and determining whether the identification field is the right child of the odd field; If there is a storage slot in the candidate storage bucket that has stored the odd field, and the identification field is the right child of the odd field, increment the right child count field of the storage slot by one; If there is a storage slot in the candidate storage bucket that has stored the odd field, but the identification field is not the right child of the odd field, the operation of storing the identification field into the storage slot in the storage bucket is terminated, and the attribute field of the storage slot is retained; In the case that no storage slot in the candidate storage bucket has stored the identification field, the odd field is stored in the candidate storage bucket.

10. The method according to claim 9, characterized in that Storing the odd field in the candidate storage bucket includes: Determining whether there is an empty storage slot in the candidate storage bucket; In the case where there is an empty storage slot in the candidate storage bucket, setting the key field of the empty storage slot to the odd field, setting the layer number field to the odd parameter, and setting the count field to one; In the case that there is no empty storage slot in the candidate storage bucket, the storage slot with the smallest count field in the candidate storage bucket is searched, and the key field of the storage slot with the smallest count field is replaced with the odd field with a preset probability.

Images