Flow processing method and device based on Sketch and flow result query method and device based on Sketch

By building a multi-layer Sketch data structure in a programmable switch and adopting compression and probability compensation methods, the problem of not being able to simultaneously implement packet granularity and byte granularity statistics in the prior art is solved, and accurate monitoring and data storage of network traffic is achieved. .

CN118972273APending Publication Date: 2024-11-15DOUYIN VISION CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411039355.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing Sketch-based network traffic measurement technology cannot achieve accurate packet granularity and byte granularity statistics at the same time, resulting in statistical overflow and accuracy loss in data center network status monitoring.

Method used

By constructing two Sketch data structures in a programmable switch, they are used for packet granularity and byte granularity statistics respectively, and inserting the byte granularity data statistics results into the Sketch storage space using compression and probability compensation methods, the packet granularity and word are achieved. Accurate statistics of granularity data.

Benefits of technology

Accurate statistics of packet granularity and byte granularity data are realized, which reduces the order of magnitude of byte granularity data statistics, and improves the accuracy of data storage, avoids counter overflow and accuracy loss.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118972273A_ABST
    Figure CN118972273A_ABST
Patent Text Reader

Abstract

The invention relates to a Sketch-based traffic processing method and device, and a traffic result query method and device. The method comprises the following steps: respectively carrying out packet granularity and byte granularity data statistics on to-be-detected traffic; inserting the data statistical result of the packet granularity into a Sketch storage space of the programmable switch, and inserting the data statistical result of the byte granularity into the Sketch storage space of the programmable switch in a compression and probability compensation mode; wherein the compression and probability compensation mode is used for performing order-of-magnitude conversion on the data statistical result of the byte granularity. The compression and probability compensation mode can reduce the order of magnitude of the data statistics result of the byte granularity, and can also ensure the precision of the data statistics result of the byte granularity, so that the data statistics of the packet granularity and the data statistics of the byte granularity can be simultaneously and accurately realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a flow processing method based on Sketch, a flow result query method and a device. Background Art

[0002] Network traffic measurement is the basis of network operation and maintenance, and is widely used in traffic engineering, anomaly detection, troubleshooting, and network billing. Through network traffic measurement, the efficiency and reliability of the network can be quickly and efficiently evaluated, and problems can be discovered and solved to improve network performance and availability.

[0003] Sketch-based network traffic measurement is a common network traffic measurement method in related technologies. By sacrificing a certain degree of accuracy to record network traffic information instead of information about each data packet, the memory resources required for network traffic measurement can be significantly reduced. Related technologies can usually accurately implement data statistics at the packet granularity, but cannot accurately implement data statistics at the byte granularity at the same time. Summary of the invention

[0004] The present disclosure provides a flow processing method based on Sketch, a flow result query method and a device.

[0005] According to a first aspect of the present disclosure, a Sketch-based traffic processing method is provided, which is applied to a programmable switch. The method includes:

[0006] Perform data statistics at packet and byte granularity for the traffic to be inspected;

[0007] The data statistics results at the packet granularity are inserted into the Sketch storage space of the programmable switch, and the data statistics results at the byte granularity are inserted into the Sketch storage space of the programmable switch through compression and probability compensation; wherein the compression and probability compensation method is used to convert the data statistics results at the byte granularity by an order of magnitude.

[0008] According to a second aspect of the present disclosure, a flow result query method based on Sketch is provided, which is applied to a server, and the method includes:

[0009] Obtain identification information of the traffic to be queried, and perform traffic query processing on the Sketch storage space of the programmable switch based on the identification information;

[0010] Obtain a first query result of a packet granularity of the Sketch storage space, and obtain a second query result of a byte granularity of the Sketch storage space;

[0011] The first query result and the second query result are mutually verified to obtain a first target query result corresponding to the packet granularity and a second target query result corresponding to the byte granularity after mutual verification.

[0012] According to a third aspect of the present disclosure, a Sketch-based traffic processing device is provided, which is applied to a programmable switch, and the device includes:

[0013] A data statistics module is used to perform data statistics at packet and byte granularities on the traffic to be detected;

[0014] A data insertion module is used to insert the data statistics results of the packet granularity into the Sketch storage space of the programmable switch, and insert the data statistics results of the byte granularity into the Sketch storage space of the programmable switch through compression and probability compensation; wherein the compression and probability compensation method is used to convert the data statistics results of the byte granularity into an order of magnitude.

[0015] According to a fourth aspect of the present disclosure, a flow result query device based on Sketch is provided, which is applied to a server, and the device includes:

[0016] A data query module, which obtains identification information of the traffic to be queried, and performs traffic query processing on the Sketch storage space of the programmable switch based on the identification information;

[0017] A query result acquisition module, used to acquire a first query result of a packet granularity of the Sketch storage space, and acquire a second query result of a byte granularity of the Sketch storage space;

[0018] A data verification module is used to perform mutual verification through the first query result and the second query result to obtain the first target query result corresponding to the packet granularity and the second target query result corresponding to the byte granularity after mutual verification.

[0019] According to a fifth aspect of the present disclosure, an electronic device is provided. The electronic device comprises: a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0020] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.

[0021] The Sketch-based traffic processing method, traffic result query method and device provided by the embodiments of the present disclosure respectively perform data statistics of packet granularity and byte granularity on the traffic to be detected, insert the data statistics results of the packet granularity into the Sketch storage space of the programmable switch, and insert the data statistics results of the byte granularity into the Sketch storage space of the programmable switch by compression and probability compensation. Since the compression and probability compensation methods can reduce the order of magnitude of the data statistics results of the byte granularity and can also ensure the accuracy of the data statistics results of the byte granularity, it is possible to achieve accurate data statistics of the packet granularity and the byte granularity at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A schematic diagram of a flow insertion process provided by an exemplary embodiment of the present disclosure;

[0024] Figure 2 A schematic diagram of a Sketch data structure provided for an exemplary embodiment of the present disclosure;

[0025] Figure 3 A schematic diagram of a Sketch data structure provided by an exemplary embodiment of the present disclosure during flow insertion;

[0026] Figure 4 A schematic diagram of a flow query process provided by an exemplary embodiment of the present disclosure;

[0027] Figure 5 A schematic diagram of a process flow query based on a Sketch data structure provided for an exemplary embodiment of the present disclosure;

[0028] Figure 6 A schematic diagram of resource allocation provided for an exemplary embodiment of the present disclosure;

[0029] Figure 7 A schematic diagram of a resource allocation process provided for an exemplary embodiment of the present disclosure;

[0030] Figure 8 A flow chart of a Sketch-based traffic processing method provided for an exemplary embodiment of the present disclosure;

[0031] Fig. 9 Another flow chart of a Sketch-based traffic processing method provided for an exemplary embodiment of the present disclosure;

[0032] Fig.10A schematic block diagram of functional modules of a Sketch-based traffic processing device provided for an exemplary embodiment of the present disclosure;

[0033] Fig.11 A schematic block diagram of functional modules of a Sketch-based traffic processing device provided for another exemplary embodiment of the present disclosure;

[0034] Fig.12 A structural block diagram of an electronic device provided for an exemplary embodiment of the present disclosure;

[0035] Fig.13 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0037] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0038] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0039] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0040] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0041] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0042] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0043] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending a prompt message to the user may be, for example, a pop-up window, in which the prompt message may be presented in text. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0044] In the related technologies, the network traffic measurement method based on Sketch often cannot achieve accurate packet granularity statistics and byte granularity statistics at the same time, that is, the related technology of network traffic measurement based on Sketch can achieve relatively accurate packet granularity statistics on programmable switches, but cannot achieve accurate byte granularity statistics. In actual application, including when monitoring the network status of a data center, the statistics of byte granularity are equally important. Compared with packet granularity statistics, byte granularity statistics are often three orders of magnitude higher than packet granularity statistics, which makes the counters in programmable switches prone to overflow.

[0045] For example, a data stream may consist of multiple data packets, and a data packet may include multiple bytes. For example, a data packet may include 1500 bytes to 9000 bytes, so the number of bytes is three orders of magnitude higher than the number of packets. When implementing packet granularity statistics and byte granularity statistics at the same time, it will bring severe challenges to the original data structure of the programmable switch.

[0046] Therefore, in order to solve the above technical problems, the technical solution provided by the embodiment of the present disclosure can simultaneously realize accurate statistics of the two dimensions of packet granularity and byte granularity, and the embodiment of the present disclosure can implement Sketch-based algorithms on the chip of the programmable switch, such as: CM-Sketch, ElasticSketch, Tower Sketch, etc. In the embodiment, Tower Sketch is taken as an example, and the technical solution of the embodiment of the present disclosure obtained by converting Tower Sketch is referred to as ByteSketch.

[0047] In the embodiment, in the process of constructing the data structure in the programmable switch, two Sketch data structures may be established to store the statistics of the packet granularity and the statistics of the byte granularity respectively. Each Sketch data structure established may include d layers, for example, d may be 3, but the embodiment is not limited thereto. Figure 2 As shown, Figure 2 is a schematic diagram of a 3-layer Sketch data structure in an embodiment of the present disclosure, Figure 2 It includes the Sketch data structure of packet granularity statistics and the Sketch data structure of byte granularity statistics. Figure 2 The 1 in the Sketch data structure of the mid-pack granularity statistics indicates the true value when the value in the bucket is 1.

[0048] The i-th layer may include w i counters and a hash function h(.), each counter can contain δ i Each layer can contain the same amount of space, and as the number of layers increases, δ i will increase, w i In the Sketch data structure with byte granularity, the i-th layer has a compression ratio of 2 C(i) , which means that if the value stored in the calculator of this layer is y, the actual value stored is y*2 C(i) Among them, if each layer's δ i and w i If the number of counters in each layer is the same, the number of counters in each layer can be obtained by dividing the size of the layer space by the size of the space occupied by a single counter.

[0049] After constructing the above two Sketch data structures, you can insert data into the network traffic. Figure 1 As shown, the process of traffic insertion may include the following steps:

[0050] Step S11, parsing the flow identifier in the data packet.

[0051] In an embodiment, when traffic is inserted, the fields of interest in the data packet can be extracted as identifiers for identifying the flow in the sketch statistics, and these fields are arranged consecutively in a certain order to form the ID (identification) of the flow, that is, the flow identifier, which can be marked as e in the embodiment.

[0052] Specifically, the data packet can be identified by a two-tuple consisting of <source IP address, destination IP address>, or by a five-tuple consisting of <source IP address, destination IP address, source port number, destination port number, protocol number>, etc., and the extracted message length or data packet length can be recorded as x.

[0053] Step S12, determining whether measurement is permitted.

[0054] In the embodiment, the Sketch admission table can be queried to check whether a message or data packet is allowed to be counted in the Sketch data structure. The Sketch admission table is almost the same as the common admission table, including port number, protocol number and priority registration, with specific layer 2 and 3 addresses, and whether it needs to be included in the evaluation range can be determined through fields such as vlan mac and vni.

[0055] Step S13: if the measurement is not permitted, the value of any register in the programmable switch is not modified.

[0056] Messages or data packets that fail to pass the admission decision will be bypassed, while messages or data packets that pass the admission decision will continue to be processed.

[0057] Step S14, performing calculations using 2*d hash functions in the Sketch data structure.

[0058] Step S15: Use the values ​​calculated by the first d hash functions to locate the counter position of the Sketch of the packet granularity statistics, and use the values ​​calculated by the last d hash functions to locate the counter position of the Sketch of the byte granularity statistics.

[0059] In the embodiment, two Sketch data structures are constructed as described above, and each Sketch data structure includes d different hash functions. In this way, 2*d different hash functions can be calculated, and the values ​​calculated by the first d hash functions are used to locate the counter position of the Sketch of the packet granularity statistics. The values ​​calculated by the last d hash functions are used to locate the counter position of the Sketch of the byte granularity statistics. Among them, the above functions in the embodiment can be as independent of each other as possible, and the higher the degree of independence, the higher the measurement accuracy.

[0060] Step S16, locate d counters mapped to packet granularity, saturate and add 1, and locate d counters mapped to byte granularity, saturate and add corresponding increment values.

[0061] In the embodiment, the corresponding increment value V of each layer of Sketch for which byte granularity statistics need to be calculated is i (x). The most accurate statistics under data compression is to add a floating point number to the counter, which does not lose any precision. However, the counters that are easy to implement in the programmable switch chip count integers and cannot calculate the decimal part. For example, in order to facilitate the counter to meet the data statistics of byte granularity, 1500 bytes can be compressed to 1.5k bytes. The 1 in 1.5 can be realized by the counter, while 0.5 cannot be calculated in the programmable switch.

[0062] Therefore, the embodiment of the present disclosure adopts compression and probability compensation, that is, the calculation is performed by combining shift compression and probability compensation. While achieving compression, the loss of accuracy can be reduced while ensuring the Bayesian unbiasedness of the compression process in a statistical sense, so that the storage capacity of data in the Sketch data structure can be increased as much as possible, and the overflow of the counter can be avoided.

[0063] Specifically, according to the above embodiment, the Sketch data structure includes d layers, wherein the i-th layer has a compression ratio of 2 C(i) , so that the above x can be divided by 2 C(i) , we can get the quotient and remainder, namely:

[0064] V i (x) = quotient + α, P (α = 1) = remainder / 2 C(i) (1)

[0065] Among them, V i (x) represents the increment value of the i-th layer in the Sketch data structure, and P(α=1) represents the probability when the counter is increased by 1. The values ​​calculated by the first d hash functions, such as hash functions h1(e), h2(e), …, hd(e), are used to locate the d mapped counters L1[h1(e)], L2[h2(e)], …, Ld[hd(e)], and then the first d mapped counters are increased by 1; the values ​​calculated by the last d hash functions, such as hash functions hd+1(e), hd+2(e), …, h2d(e), are used to locate the d mapped counters L1[h1+d(e)], L2[h2+d(e)], …, Ld[h2d(e)], which are used to locate the counter position of each layer of Sketch for byte granularity statistics and increment V1(x), V2(x)…V d(x). Where L represents the number of layers, L1 represents the first layer, L1[h1(e)] represents the counter h1(e) of the first layer, and Ld represents the dth layer.

[0066] It should be noted that the increment in the embodiment adopts a saturation addition method, which can specifically include two meanings: if the increment will cause the counter to overflow, the increment will directly set the counter to the maximum value that the counter can carry, and mark the counter as overflow; if a counter has overflowed, the value of the counter will not be modified in subsequent operations.

[0067] The same is true for other data packets that need to be counted. At the end of the statistics, the entire Sketch data structure can be uploaded so that the query results can be obtained through the query process.

[0068] During the flow insertion process, Figure 3 As shown, for example, the data stream e corresponding to the inserted data packet has a packet length x=1518. Figure 3 The left side shows the packet granularity statistics. The Sketch storage space corresponding to the Sketch data structure is used to store the packet granularity statistics results. The Sketch data structure includes three layers. The values ​​of the counters at each layer are as follows: Figure 3 As shown in the upper left part, when inserting the above data packet, the value of each layer counter in the Sketch data structure needs to be updated. Figure 3 Shown at lower left. Figure 3 The right side shows the byte granularity statistics. Similarly, the Sketch storage space corresponding to the Sketch data structure is used to store the byte granularity statistics results. The Sketch data structure can also include a three-layer structure. The value of each layer counter is as follows: Figure 3 As shown in the upper right part, when inserting the above data packet, the value of each layer counter in the Sketch data structure needs to be updated. Figure 3 Shown in the lower right.

[0069] Specifically, in the embodiment, ByteSketch can first calculate 2d hash functions: h1(e)=3, h2(e)=3, h3(e)=0, h4(e)=4, h5(e)=0, h6(e)=1.

[0070] In this way, the counters of the first d mapped packet granularity statistics can be located: L1[h1(e)]=63, L2[h2(e)]=55, L3[h3(e)]=120. Figure 3As shown in the upper left part, L1[h1(e)] represents the value of the counter with the first layer index h1(e), L2[h2(e)] represents the value of the counter with the second layer index h2(e), and L3[h3(e)] represents the value of the counter with the third layer index h3(e). Among them, L1[h1(e)] has overflowed and its value will not be increased later.

[0071] When the above data packet is inserted, the non-overflow counters are increased by 1, that is, L2[h2(e)]=55+1=56, L3[h3(e)]=120+1=121. After the insertion, L2[h2(e)] is marked as overflowed.

[0072] In addition, the counters of the byte granularity statistics mapped after positioning are: L1[h4(e)]=14, L2[h5(e)]=166, L3[h6(e)]=1120. If the length of the inserted data packet is 1518, the calculated values ​​of each layer are:

[0073]

[0074]

[0075]

[0076] TowerSketch increments the un-overflowed counter by V i (x):

[0077] Here we assume that if P>50%, then α=1 for this insertion.

[0078] L1[h4(e)]=14+6=20, L2[h5(e)]=166+95=261, L3[h6(e)]=1120+379=1499. Indicates rounding down, and % indicates remainder.

[0079] In the data query process, such as Figure 4 As shown, the following steps may be included:

[0080] Step S21: query the packet granularity statistics and byte granularity statistics.

[0081] In the embodiment, when the Sketch data structure is uploaded, the process query process can be entered. Given the queried data flow, its ID can be e.

[0082] Step S22: Calculate using 2*d hash functions in the Sketch data structure.

[0083] Step S23: locate d mappings to packet granularity counters;

[0084] Step S24: locate d counters mapped to byte granularity;

[0085] Step S25: Return the minimum value Rp of the d mapped packet granularity counters that has not overflowed;

[0086] Step S26: Return the minimum value R among the decompressed values ​​in the d mapped byte granularity counters B .

[0087] In the embodiment, 2*d hash functions h1(e), h2(e)..., h2d(e) can be calculated first, and the first d hash functions locate the counters L1[h1(e)], L2[h2(e)], ..., Ld[hd(e)] of the Sketch of d packet granularity, and then return the minimum value Rp of the counter that has not overflowed; the second d hash functions locate the counters L1[h1+d(e)], L2[h2+d(e)], ..., Ld[h2d(e)] of the Sketch of d byte granularity, and multiply the value of the corresponding register by the compression ratio 2 corresponding to each layer. C(i) , so that it can be restored to the original value before compression and then return the minimum value R of the counter that has not overflowed B .

[0088] Step S27: query byte granularity statistics and packet granularity statistics;

[0089] Step S28: Return to R B / The minimum value between the Ethernet minimum frame length and Rp, and the minimum value is used as the query result of the packet granularity;

[0090] Step S29: Return to R B and Rp*maximum transmission unit, and use the minimum value as the query result of the byte granularity.

[0091] In the embodiment, when the packet granularity statistics result needs to be queried, R B / The smaller value between the Ethernet minimum frame length and Rp. B is the minimum value of the non-overflow counter in the byte granularity statistics, so R B / The value obtained by the Ethernet minimum frame length is equivalent to the maximum number of packets, which is equal to the minimum value R in the packet granularity counter that has not overflowed. P By comparison, if Rp is greater than R B / The value of the Ethernet minimum frame length is still large, indicating that Rp is problematic, and then you can use R B / Ethernet minimum frame length is used to check Rp.

[0092] When you need to query the byte granularity statistics, return Rp*Maximum Transmission Unit and R B The smaller value. Since Rp*Maximum Transmission Unit is B The smaller value is equivalent to obtaining the statistical value of the largest byte granularity, while R B is the minimum value among the decompressed values ​​in the byte granularity counters. If R B It is larger than the value of Rp*Maximum Transmission Unit, indicating that R B There is a problem, and then R can be adjusted by the value of Rp*maximum transmission unit. B The purpose of the calibration.

[0093] In the embodiment, by performing checks between packet granularity statistics and byte granularity statistics in mutually independent Sketch storage spaces, the accuracy of measurement can be further improved, and the possibility of a small flow being queried as a large flow due to hash conflicts can be greatly reduced.

[0094] For example, in the process of traffic query, combined with the above Figure 3 And corresponding embodiments, such as Figure 5 As shown, 2d hash functions can be calculated first: h1(e)=3, h2(e)=3, h3(e)=0, h4(e)=4, h5(e)=0, h6(e)=1.

[0095] Locate the d mapped packet granularity statistics counters: L1[h1(e)]=63, L2[h2(e)]=56, L3[h3(e)]=121. L1[h1(e)] has overflowed. Then return the minimum value of the non-overflowing counter: Rp:min{121,56}=56.

[0096] Locate the d mapped byte granularity statistics counters: L1[h4(e)]=20, L2[h5(e)]=261, L3[h6(e)]=1499. Then return the decompressed minimum value of the non-overflowing byte granularity counter:

[0097] Rp: min{1499*2 2 ,261*2 4 , 20*2 8}=4176.

[0098] When querying the number of packets of data stream e, min{56,[4176 / 64]}=56 is returned; when querying the number of flows of stream e, min{4176,56*1500}=4176 is returned. Here, it is assumed that the minimum frame length of Ethernet is 64 and the maximum transmission unit is 1500.

[0099] The embodiment of the present disclosure uses different hash functions to locate the counters of the Sketch of packet granularity statistics and byte granularity statistics, and combines the mutual verification method used in the query process to enhance the statistical accuracy of byte granularity and packet granularity. Different hash functions are used to locate packet granularity statistics and byte granularity statistics respectively, which makes hash conflicts occur independently in the 2*d layer. When querying the packet granularity, the embodiment converts the statistical value of the byte granularity to the packet granularity as much as possible; when querying the granularity, the embodiment converts the statistics of the packet granularity to the byte granularity as much as possible, and increases the rigorous estimation calculated by the independent sketch, and introduces additional dimensions, which can improve the measurement accuracy. In addition, the embodiment of the present disclosure can avoid the overestimation of small flows as much as possible, and reduce the false alarm triggering in the actual application process.

[0100] Based on the above embodiments, considering that resources in a programmable switch are limited, in order to improve the resource utilization efficiency in the programmable switch and better deploy the converted Sketch to the programmable switch, in the embodiments provided in the present disclosure, resource utilization is improved by making full use of resources and reducing bandwidth loss.

[0101] In the embodiment, the Sketch-based flow measurement method will share one or more pipeline resources in an exclusive manner or with the business logic or basic forwarding logic during the deployment of the programmable switching chip. Here we take one pipeline resource as an example for explanation, and multiple pipelines can be expanded by analogy.

[0102] The outbound and inbound pipelines of this pipeline resource are used together, which can include two different pipeline deployment scenarios. The first is that the message enters from the inbound pipeline, passes through the traffic manager, and then accesses the outbound pipeline; the second is that the message enters from the outbound pipeline, passes through the port loopback and enters the inbound pipeline. These two types of deployment scenarios can be abstracted as the message first enters the first-accessed pipeline, then passes through the middle traffic manager, and then enters the last-accessed pipeline.

[0103] like Figure 6 As shown, the resource arrangement method provided by the embodiment of the present disclosure is mainly divided into an insertion preprocessing module, Sketch part 1, Sketch part 2 and two parts of Hash operation modules. In the first accessed pipeline, the resources of the insertion preprocessing module and Sketch part 1 can be preferentially allocated, and the full amount of hashing of Sketch part 1 and a small amount of hashing of Sketch part 2 can be realized. In the later accessed pipeline, the resources of Sketch part 2 and the hash in Sketch part 2 that are not realized in the first accessed pipeline are allocated.

[0104] In an embodiment, the insertion preprocessing module can use the front stage resources of the pipeline that is accessed first to implement flow monitoring process access, byte Sketch insertion value precalculation and independent logic gate centralized budgeting.

[0105] Among them, the traffic monitoring process access corresponds to the Sketch access table in the above embodiment, and a decision needs to be made first to make the sketch work.

[0106] Byte Sketch insert value pre-calculation is responsible for implementing V i (x) = quotient + α*1, P(α=1) = remainder / 2 C(i) However, it is very difficult to implement the above operation for programmable switching. Therefore, in the embodiment of the present disclosure, the obtained packet length x is shifted left by 2. C (i) The quotient is obtained by truncating the obtained packet length bit by bit, and the remainder is obtained by taking the binary expression of the packet length x (C(i), 0]. The above probability compensation P(α=1)=remainder / 2 C(i) To do the conversion:

[0107] P(α=1)=remainder / 2 C(i) <=>P(α=1)=P(random(0,2 C(i) )<remainder)(2)

[0108] Since the programmable switching chip does not support the size judgment between two 16 bits, the above formula (2) can be further converted to convert the inequality comparison into an equality comparison using saturation subtraction:

[0109] P(α=1)=P(random(0, 2 C(i) < remainder) <=> P(α=1)=P(random(0, 2 C(i) )| -| remainder == 0) (3)

[0110] Through the above conversion, it is possible to complete the V i (x) Operations on programmable switch chips.

[0111] The logic gate operation in the embodiment may refer to the operation of judgment such as if else. The insertion process of Sketch implemented in the programmable switching chip mainly includes two types of logic gate operations, one is independent logic gate operation, this part of the logic budget can be judged without reading the data stored in the counter; the other is dependent logic gate operation, this part of the logic budget needs to read the data stored in the counter to judge, and the result after judgment will affect the subsequent insertion of the counter, such as the update of the weight part in ElasticSketch, which needs to determine the ratio of positive votes. For the part with unqualified ratio, a new value will be inserted into the weight part, otherwise no corresponding update will be performed.

[0112] The disclosed embodiment performs pre-calculations by concentrating all independent logic gate operations, and distributes dependent logic gate operations to the same stage of Sketch counter resources for calculation. This allows the logic gate judgment before Sketch counter insertion to occupy an additional stage when adapting Sketch containing dependencies, thereby making Sketch counters more closely arranged and reducing the waste of available resources.

[0113] In the embodiment, by concentrating the three functions of traffic monitoring process admission, byte Sketch insertion value pre-calculation and independent logic gate centralized budgeting in the insertion preprocessing module for calculation, resources can be better used in parallel to save more available serial stages for the Sketch structure. In addition, the preprocessing part in the embodiment allocates the resources of the front end of the pipeline that is first accessed instead of other parts, because the front stage of the pipeline that is first accessed and the front stage of the pipeline that is later accessed share resources. The front stage of the pipeline that is first accessed cannot insert the Sketch counter because the counter insertion requires the hash result and the increment value to be prepared in advance. Therefore, in order to make it easier to deploy a larger Sketch, the resources of each stage need to be fully utilized, so in the embodiment, the Sketch counter needs to be deployed at the front stage of the pipeline that is later accessed. Because the resources occupied by the counter are usually different from those of the insertion preprocessing, the insertion preprocessing is mainly logic gate operation, while the Sketch part mainly uses SRAM (Static Random-Access Memory) resources for storage. Therefore, mixing insert preprocessing with counter resources will have better resource utilization, making Sketch deployment integration easier to achieve.

[0114] In the embodiment, the insertion of Sketch also requires the preparation of the hash values ​​h1(e), h2(e), ..., hd(e) corresponding to each layer in advance. Since the operation of the hash function requires only 6 hash operation units in each stage, and each counter group also occupies one. Since there is no dependency between hash functions, hi(e) only needs to be calculated before the operation is implemented in the i-th layer, and i is an integer.

[0115] Therefore, the allocation strategy of the hash operation in the embodiment adopts the method of first allocating resources to the Sketch counters, and then performing operations from front to back according to the order of Sketch counter allocation and ensuring that hi(e) is calculated before the operation is implemented in the i-th layer. In addition, the hash value of the counter allocated to Stage0 in Sketch part 2 must be calculated in the pipeline that enters first to ensure that the resources of stage0 can be fully utilized. Therefore, in the pipeline that enters first, it is necessary to perform a full hash operation on the allocated counters of Sketch part 1 and a hash value on the counter allocated to Stage0 in Sketch part 2, and complete the hash value operation of the remaining counters in Sketch part 2. In Sketch part 1 in the first-accessed pipeline and Sketch part 2 in the later-accessed pipeline, the dependent logic gates can be evenly distributed.

[0116] In the embodiment provided by the present disclosure, the counter resources of Sketch are divided into part 1 and part 2, and the resources of the pipeline accessed first and the resources of the pipeline accessed later are used respectively. Regardless of whether the intermediate module is a loopback or a traffic manager, the Sketch counter using the resources of part 2 of Sketch needs to transmit additional bridge fields, which has a certain impact on the throughput. Therefore, in the process of allocating Sketch counter resources, such as Figure 7 As shown, a counter that can preferentially allocate byte granularity is implemented in part 1.

[0117] The insertion of a packet-sized Sketch counter requires a hash as the index value to complete the insertion. The insertion of a byte-sized Sketch counter requires a hash as the index value and V i (x) The insertion can be completed by increasing the value. If the byte-granularity Sketch is allocated to Sketch part 2 for implementation, more additional bridging fields will be introduced. Therefore, in the allocation strategy of the embodiment of the present disclosure, the byte-granularity counter will be implemented in the available space of Sketch part 1 first.

[0118] After the byte-granularity counter is preferentially allocated in Sketch part 1, it is necessary to determine whether there is enough space to deploy the counter in Sketch part 1. If there is still available space left in Sketch part 1, the data structure of Sketch is first determined.

[0119] a) If the resource occupancy of each layer in the Sketch data structure is different, such as constructing a special-shaped TowerSketch in the embodiment, the space occupied by the 4-layer and 5-layer Sketch is 1 / 2 of that of other layers. Because the space occupied is smaller, more layers of counters can be implemented in the same space. When the counters of the layers that occupy more resources are deployed in part 2, the additional bridging field required for transmission will be changed from the index of the first layer to the index of the second layer. Therefore, the allocation scheme of the embodiment of the present disclosure will give priority to implementing these Sketch counters that occupy less resources in the available space of part 1. If there is enough space to deploy counters in Sketch part 1 after the above a is completed, the operation in b is performed.

[0120] b) If the resource occupancy of each layer in the Sketch data structure is the same. Since the layers with more barrels have longer indexes, in order to reduce the extra bridging fields that need to be transmitted, the embodiment of the present disclosure adopts the method of preferentially allocating the counters of the layers with more barrels to Sketch part 1. If there is enough space to deploy counters in Sketch part 1 after b is completed, the remaining registers are deployed to Sketch part 1. If all the counters of the designed Sketch are allocated resources, the resource allocation is successful. Otherwise, it is necessary to check whether Sketch part 2 has enough space to deploy all the remaining counters. If there is space, the counters are allocated to Sketch2 and the allocation is successful; otherwise, the resource share allocation fails. In this way, by making full use of pipeline resources and reasonable resource allocation, the difficulty of deployment integration on programmable switches can be reduced.

[0121] Based on the above embodiments, the present disclosure also provides a flow processing method based on Sketch, such as Figure 8 As shown, the method may include the following steps:

[0122] In step S810, data statistics are performed on the traffic to be detected at packet granularity and byte granularity respectively.

[0123] In step S820, the data statistics results at the packet granularity are inserted into the Sketch storage space of the programmable switch, and the data statistics results at the byte granularity are inserted into the Sketch storage space of the programmable switch through compression and probability compensation; wherein the compression and probability compensation methods are used to convert the data statistics results at the byte granularity by an order of magnitude.

[0124] In an embodiment, a Sketch storage space can be created in a programmable switch, and data statistics at the packet granularity and data statistics at the byte granularity can be performed at the same time. For data statistics at the packet granularity, for example, the Sketch storage space can be composed of multiple hash functions and multiple counter buckets. Each storage space of the counter bucket maintains a counter, where each hash function corresponds to a counter bucket group. When each network message arrives, it needs to be processed by each hash function operation, and according to the hash mapping value obtained by the processing, it is stored in the counter of the hash value corresponding to the corresponding bucket group.

[0125] As for the data statistics of byte granularity, according to the above embodiments, the data statistics results of byte granularity are usually three orders of magnitude higher than the data statistics results of packet granularity. If the data statistics results of byte granularity are directly stored through counters, it is easy to cause counter overflow and it is impossible to store a large number of data statistics results of byte granularity. Therefore, the embodiment inserts the data statistics results of byte granularity into the Sketch storage space of the programmable switch through compression and probability compensation.

[0126] In an embodiment, the compression and probability compensation method is used to convert the order of magnitude of the data statistical results at the byte granularity, that is, the compression and probability compensation method can compress the data statistical results at the byte granularity, reduce the order of magnitude of the data statistical results at the byte granularity, and improve the accuracy of the compressed data statistical results at the byte granularity through probability compensation, so as to facilitate the storage of the data statistical results at the byte granularity into the Sketch storage space of the programmable switch while ensuring the accuracy of the storage of the data statistical results at the byte granularity.

[0127] For example, during the traffic insertion process, if a data packet has 1500 bytes, when performing data statistics at the packet granularity, the statistical result of the packet granularity can be added to 1 in the counter, and when performing data statistics at the byte granularity, the 1500 bytes can be compressed by three orders of magnitude, that is, compressed to 1.5. At this time, the value of the counter increases by 1, and the remaining 0.5 can be processed by probability compensation, that is, there is a 0.5 probability that the value of the counter increases by 1. In this way, by continuously counting the byte granularity of the data packet, while storing the data statistical results at the byte granularity in the Sketch storage space, the accuracy of the stored data statistical results at the byte granularity can also be ensured as a whole.

[0128] The Sketch-based traffic processing method provided by the embodiment of the present disclosure performs data statistics at packet granularity and byte granularity on the traffic to be detected, inserts the data statistics results at packet granularity into the Sketch storage space of the programmable switch, and inserts the data statistics results at byte granularity into the Sketch storage space of the programmable switch by compression and probability compensation. Since the compression and probability compensation methods can reduce the order of magnitude of the data statistics results at byte granularity and can also ensure the accuracy of the data statistics results at byte granularity, it is possible to achieve accurate data statistics at packet granularity and data statistics at byte granularity at the same time.

[0129] Based on the above embodiment, in another embodiment provided by the present disclosure, the above step S820 may further specifically include a step:

[0130] In step S821, a compression ratio is obtained, and data compression is performed on the data statistics result of the byte granularity according to the compression ratio to obtain the loss accuracy of the data compression.

[0131] In the embodiment, based on Figure 2 As can be seen from the corresponding embodiments, the Sketch data structure may include d layers, so the compression of the i-th layer is 2 C(i) , if the value stored in the layer calculator is y, then the actual stored value is y*2 C(i) .

[0132] In step S822, probability compensation is performed on the data statistics results of the byte granularity after data compression based on the loss precision, and the probability compensated data is inserted into the Sketch storage space of the programmable switch.

[0133] In the embodiment, when performing byte-granular data statistics on a data packet, the length x of the data packet is obtained and x is divided by 2. C(i) , we can get the quotient and remainder, so we can get the incremental value V of the i-th layer in the Sketch data structure through the above formula (1). i (x), and the probability when the counter increases α. In this way, while compressing the data statistics of the byte granularity through the compression ratio, the loss of precision caused by the data compression can be obtained, and the probability of the data statistics of the byte granularity after the data compression can be compensated by the loss of precision, that is, the probability plus 1 can be constructed by extracting the precision lost by the compression, and the probability compensated data can be inserted into the Sketch storage space of the programmable switch, so that the statistics of floating-point operations in the programmable switch can be realized under limited logical operations.

[0134] Therefore, based on the data statistics results and compression ratio of byte granularity, the increment value and compensation probability of each counter in the Sketch storage space of the programmable switch can be obtained, and the counting results of each counter in the Sketch storage space of the programmable switch can be determined based on the increment value and the compensation probability.

[0135] Based on the above embodiment, in another embodiment provided by the present disclosure, based on Figure 2 , Figure 3 and Figure 6 As shown, a first Sketch storage space and a second Sketch storage space can be created in a programmable switch; wherein the first Sketch storage space is located in a first access part of a pipeline in the programmable switch, and the second Sketch storage space is located in a later access part of the pipeline in the programmable switch; the method further comprises the following steps:

[0136] In step S830, first priorities corresponding to the first Sketch storage space and the second Sketch storage space are obtained, and second priorities corresponding to the byte granularity and the packet granularity are obtained.

[0137] The priority of the first Sketch storage space is higher than that of the second Sketch storage space, and the priority of the byte granularity is higher than that of the packet granularity.

[0138] In step S840, based on the first priority and the second priority, a byte-granularity counter and a packet-granularity counter are respectively allocated to the first Sketch storage space and the second Sketch storage space, wherein the byte-granularity counter is used to count the data statistics results of the byte granularity, and the packet-granularity counter is used to count the data statistics results of the packet granularity.

[0139] In an embodiment, data pre-insertion processing can be performed by inserting a pre-processing module. The pre-insertion processing includes flow monitoring process access processing, byte-granularity Sketch insertion value pre-calculation processing, and centralized pre-calculation processing of independent logic gate operations. In this way, dependent logic gate operations can be evenly distributed to the first Sketch storage space and the second Sketch storage space for calculation processing.

[0140] In the embodiment, Figure 6 As shown, in the first accessed pipeline, a preprocessing module and Sketch part 1 may be inserted, and in the later accessed pipeline, Sketch part 2 may be included. Here, Sketch part 1 is equivalent to the first Sketch storage space, and Sketch part 2 is equivalent to the second Sketch storage space.

[0141] The outbound and inbound pipelines of this pipeline resource are used together, which can specifically include two different pipeline deployment scenarios. The first is that the message enters from the inbound pipeline, passes through the traffic manager, and then accesses the outbound pipeline; the second is that the message enters from the outbound pipeline, passes through the port loopback and enters the inbound pipeline. These two types of deployment scenarios can be abstracted as the message first enters the first-accessed pipeline, then passes through the middle traffic manager, and then enters the later-accessed pipeline. In the embodiment, by processing the byte-granularity Sketch insertion pre-operation and independent logic gate centralized operation in the pre-insertion processing module in the first-accessed pipeline, the transmission amount through the traffic manager or port loopback can be reduced, thereby reducing the pressure on the bandwidth. For details, please refer to the description of the above embodiment, which will not be repeated here.

[0142] Based on the above embodiments, the present disclosure also provides a flow result query method based on Sketch, which can be applied to a server, such as Fig. 9 As shown, the method may include the following steps:

[0143] In step S910, identification information of the traffic to be queried is obtained, and traffic query processing is performed on the Sketch storage space of the programmable switch based on the identification information.

[0144] In the embodiment, when a data query is required for a data flow, an identifier e of the data flow may be obtained, and a flow query may be performed in the Sketch storage space of the programmable switch through the identifier e.

[0145] In step S920, a first query result of a packet granularity of the Sketch storage space is obtained, and a second query result of a byte granularity of the Sketch storage space is obtained.

[0146] In step S930, the first query result and the second query result are mutually verified to obtain a first target query result corresponding to the packet granularity and a second target query result corresponding to the byte granularity after mutual verification.

[0147] In the embodiment, the statistical results of the packet granularity and the statistical results of the byte granularity obtained by the traffic insertion of the above embodiment are stored in the Sketch storage space respectively, so that when a data query is required for a certain data stream, the query results of the byte granularity and the packet granularity of the Sketch storage space can be obtained by performing a data query on the Sketch storage space. Since the statistical results of the packet granularity and the statistical results of the byte granularity are queries on the same data stream in different dimensions, relevant verification can be performed based on the query results of the two, so that the query results of the packet granularity and the query results of the byte granularity obtained are more accurate.

[0148] Specifically, when performing correlation verification through the query result of packet granularity and the query result of byte granularity, the first query result can be compared with the first value, and the smallest value between the first query result and the first value can be used as the first target query result; wherein the first value is determined based on the second query result and the Ethernet minimum frame length. And the second query result can be compared with the second value, and the smallest value between the second query result and the second value can be used as the second target query result; wherein the second value is determined based on the first query result and the maximum transmission unit in data transmission.

[0149] Specifically, in combination with the above embodiment, when it is necessary to query the packet granularity statistics result, the minimum value R of the decompressed value in the mapped byte granularity counter can be returned. B The smallest value between the Ethernet minimum frame length and the minimum value Rp that does not overflow in the mapped packet granularity counter is used as the query result of the packet granularity. B is the minimum value of the non-overflow counter in the byte granularity statistics, so R B / The value obtained by the Ethernet minimum frame length is equivalent to the maximum number of packets, which is equal to the minimum value R in the packet granularity counter that has not overflowed. P By comparison, if Rp is greater than R B / The value of the Ethernet minimum frame length is still large, indicating that Rp is problematic, and then you can use R B / Ethernet minimum frame length is used to check Rp.

[0150] When you need to query the byte granularity statistics, return Rp*Maximum Transmission Unit and R B The smallest value among them is taken as the query result of byte granularity. B The smaller value is equivalent to obtaining the statistical value of the largest byte granularity, while R B is the minimum value among the decompressed values ​​in the byte granularity counters. If R B It is larger than the value of Rp*Maximum Transmission Unit, indicating that R B There is a problem, and then R can be adjusted by the value of Rp*maximum transmission unit. B The purpose of verification. In this way, by checking the packet granularity statistics and byte granularity statistics in the independent Sketch storage space, the accuracy of the measurement can be further improved, and the possibility of a small flow query being a large flow due to a hash conflict can be greatly reduced. For details, please refer to the description of the above embodiment, which will not be repeated here.

[0151] In the case of dividing each functional module according to each function, the embodiment of the present disclosure provides a Sketch-based traffic processing device, which can be applied to a programmable switch. Fig.10 The following is a schematic block diagram of the functional modules of a Sketch-based traffic processing device provided by an exemplary embodiment of the present disclosure. Fig.10 As shown, the Sketch-based traffic processing device includes:

[0152] A data statistics module 10 is used to perform data statistics of packet granularity and byte granularity on the traffic to be detected;

[0153] The data insertion module 20 is used to insert the data statistics results of the packet granularity into the Sketch storage space of the programmable switch, and insert the data statistics results of the byte granularity into the Sketch storage space of the programmable switch through compression and probability compensation; wherein the compression and probability compensation method is used to convert the data statistics results of the byte granularity into an order of magnitude.

[0154] In another embodiment provided by the present disclosure, the data insertion module is specifically used to:

[0155] Obtaining a compression ratio, and performing data compression on the data statistical result of the byte granularity according to the compression ratio to obtain loss precision of the data compression;

[0156] Probabilistic compensation is performed on the data statistical result of the byte granularity after data compression based on the loss precision, and the probability-compensated data is inserted into the Sketch storage space of the programmable switch.

[0157] In another embodiment provided by the present disclosure, the device further includes:

[0158] A data acquisition module, configured to obtain an increment value and a compensation probability of each counter in the Sketch storage space of the programmable switch based on the data statistics result of the byte granularity and the compression ratio;

[0159] A counting result determination module is used to determine the counting results of each counter in the Sketch storage space of the programmable switch based on the incremental value and the compensation probability.

[0160] In another embodiment provided by the present disclosure, a first Sketch storage space and a second Sketch storage space are created in the programmable switch; wherein the first Sketch storage space is located in a first access part of a pipeline in the programmable switch, and the second Sketch storage space is located in a later access part of the pipeline in the programmable switch; the counter allocation module is further configured to:

[0161] Obtain first priorities corresponding to the first Sketch storage space and the second Sketch storage space, respectively, and obtain second priorities corresponding to the byte granularity and the packet granularity, respectively; wherein the priority of the first Sketch storage space is greater than the priority of the second Sketch storage space, and the priority of the byte granularity is greater than the priority of the packet granularity;

[0162] Based on the first priority and the second priority, the byte granularity counter and the packet granularity counter are respectively allocated to the first Sketch storage space and the second Sketch storage space; wherein the byte granularity counter is used to count the data statistics results of the byte granularity, and the packet granularity counter is used to count the data statistics results of the packet granularity.

[0163] In another embodiment provided by the present disclosure, the first Sketch storage space and the second Sketch storage space respectively include a multi-layer structure; after allocating the byte-granularity counter to the first Sketch storage space, the apparatus further includes:

[0164] The counter allocation module is further configured to allocate the counter of the packet granularity to the first Sketch storage space based on the number of buckets of the packet granularity when it is detected that the remaining space of the first Sketch storage space is greater than a threshold and the space occupied by the counters in each layer of the first Sketch storage space is the same;

[0165] When it is detected that the remaining space of the first Sketch storage space is not greater than a threshold and there is still an uncompleted allocation counter, detecting whether the remaining storage space of the second Sketch storage space satisfies the storage space required by the uncompleted allocation counter;

[0166] When it is detected that the remaining storage space of the second Sketch storage space can satisfy the storage space required by the counter for the uncompleted allocation, the counter for the uncompleted allocation is allocated to the second Sketch storage space.

[0167] In another embodiment provided by the present disclosure, the first access part of the pipeline in the programmable switch further includes inserting a pre-processing module, and the device further includes:

[0168] A pre-insertion processing module, used for performing pre-insertion processing of data through the insertion pre-processing module; wherein the pre-insertion processing includes flow monitoring process admission processing, Sketch insertion value pre-calculation processing of the byte granularity and centralized pre-calculation processing of independent logic gate operations;

[0169] The operation processing module is used to evenly distribute the dependent logic gate operations to the first Sketch storage space and the second Sketch storage space for operation processing.

[0170] The Sketch-based traffic processing device provided by the embodiment of the present disclosure performs data statistics of packet granularity and byte granularity on the traffic to be detected, inserts the data statistics results of the packet granularity into the Sketch storage space of the programmable switch, and inserts the data statistics results of the byte granularity into the Sketch storage space of the programmable switch through compression and probability compensation. Since the compression and probability compensation methods can reduce the order of magnitude of the data statistics results of the byte granularity and can also ensure the accuracy of the data statistics results of the byte granularity, it is possible to achieve accurate data statistics of the packet granularity and the byte granularity at the same time.

[0171] In the case of dividing each functional module according to each function, the embodiment of the present disclosure provides a flow result query device based on Sketch, and the flow processing device based on Sketch can be applied to a server. Fig.11 The following is a schematic block diagram of the functional modules of a Sketch-based traffic processing device provided by an exemplary embodiment of the present disclosure. Fig.11 As shown, the Sketch-based traffic processing device includes:

[0172] The data query module 30 obtains identification information of the traffic to be queried, and performs traffic query processing on the Sketch storage space of the programmable switch based on the identification information;

[0173] A query result acquisition module 40, configured to acquire a first query result of a packet granularity of the Sketch storage space, and acquire a second query result of a byte granularity of the Sketch storage space;

[0174] The data verification module 50 is used to perform mutual verification through the first query result and the second query result to obtain the first target query result corresponding to the packet granularity and the second target query result corresponding to the byte granularity after mutual verification.

[0175] In another embodiment provided by the present disclosure, the data verification module is further used to:

[0176] Compare the magnitude relationship between the first query result and the first value, and use the smallest value between the first query result and the first value as the first target query result; wherein the first value is determined based on the second query result and the Ethernet minimum frame length;

[0177] Compare the size relationship between the second query result and the second numerical value, and use the minimum value between the second query result and the second numerical value as the second target query result; wherein the second numerical value is determined based on the first query result and the maximum transmission unit in data transmission.

[0178] The relevant device part corresponds to the above method. Please refer to the description of the method embodiment for details, which will not be repeated here.

[0179] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.

[0180] Fig.12 The structure diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown in FIG. Fig.12 As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801 , and the processor 1801 can execute corresponding steps in the above method disclosed in the embodiment of the present disclosure.

[0181] The processor 1801 may also be referred to as a central processing unit (CPU), which may be an integrated circuit chip having signal processing capabilities. Each step in the method disclosed in the embodiment of the present disclosure may be completed by an integrated logic circuit of hardware in the processor 1801 or by instructions in the form of software. The processor 1801 may be a general-purpose processor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a memory 1802, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or other mature storage media in the art. The processor 1801 reads the information in the memory 1802 and completes the steps of the method in combination with its hardware.

[0182] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, they can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Fig.13 The computer system 1900 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can perform various functions, including the functions described above. Fig.13 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure.

[0183] Computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0184] like Fig.13As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 to a random access memory (RAM) 1903. In the RAM 1903, various programs and data required for the operation of the computer system 1900 can also be stored. The computing unit 1901, the ROM 1902, and the RAM 1903 are connected to each other via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0185] A plurality of components in the computer system 1900 are connected to the I / O interface 1905, including: an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. The input unit 1906 may be any type of device capable of inputting information to the computer system 1900, and the input unit 1906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1908 may include, but is not limited to, a disk, an optical disk. The communication unit 1909 allows the computer system 1900 to exchange information / data with other devices over a network such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0186] The computing unit 1901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1908. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 1902 and / or communication unit 1909. In some embodiments, the computing unit 1901 may be configured to perform the above methods disclosed in the embodiments of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0187] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.

[0188] The computer-readable storage medium in the disclosed embodiments may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specifically, the computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0189] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0190] The embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor, the above method disclosed in the embodiments of the present disclosure is implemented.

[0191] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.

[0192] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0193] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0194] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0195] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).

[0196] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A flow processing method based on Sketch, characterized in that: Applied to a programmable switch, the method comprises: Perform data statistics at packet and byte granularity for the traffic to be inspected; The data statistics results at the packet granularity are inserted into the Sketch storage space of the programmable switch, and the data statistics results at the byte granularity are inserted into the Sketch storage space of the programmable switch through compression and probability compensation; wherein the compression and probability compensation method is used to convert the data statistics results at the byte granularity by an order of magnitude.

2. The method according to claim 1, characterized in that The method of inserting the byte-granularity data statistics result into the Sketch storage space of the programmable switch by compression and probability compensation includes: Obtaining a compression ratio, and performing data compression on the data statistical result of the byte granularity according to the compression ratio to obtain loss precision of the data compression; Probabilistic compensation is performed on the data statistical result of the byte granularity after data compression based on the loss precision, and the probability-compensated data is inserted into the Sketch storage space of the programmable switch.

3. The method according to claim 2, characterized in that The method further comprises: Based on the data statistics result of the byte granularity and the compression ratio, obtaining the increment value and compensation probability of each counter in the Sketch storage space of the programmable switch; The counting results of each counter in the Sketch storage space of the programmable switch are determined based on the increment value and the compensation probability.

4. The method according to claim 1, characterized in that: Creating a first Sketch storage space and a second Sketch storage space in the programmable switch; wherein the first Sketch storage space is located in a first access part of a pipeline in the programmable switch, and the second Sketch storage space is located in a later access part of the pipeline in the programmable switch; the method further includes: Obtain first priorities corresponding to the first Sketch storage space and the second Sketch storage space, respectively, and obtain second priorities corresponding to the byte granularity and the packet granularity, respectively; wherein the priority of the first Sketch storage space is greater than the priority of the second Sketch storage space, and the priority of the byte granularity is greater than the priority of the packet granularity; Based on the first priority and the second priority, the byte granularity counter and the packet granularity counter are respectively allocated to the first Sketch storage space and the second Sketch storage space; wherein the byte granularity counter is used to count the data statistics results of the byte granularity, and the packet granularity counter is used to count the data statistics results of the packet granularity.

5. The method according to claim 4, characterized in that The first Sketch storage space and the second Sketch storage space each include a multi-layer structure; after allocating the byte-granularity counter to the first Sketch storage space, the method further includes: When it is detected that the remaining space of the first Sketch storage space is greater than a threshold and the space occupied by the counters in each layer of the first Sketch storage space is the same, allocating the counters of the packet granularity to the first Sketch storage space based on the number of buckets of the packet granularity; When it is detected that the remaining space of the first Sketch storage space is not greater than a threshold and there is still an uncompleted allocation counter, detecting whether the remaining storage space of the second Sketch storage space satisfies the storage space required by the uncompleted allocation counter; When it is detected that the remaining storage space of the second Sketch storage space can satisfy the storage space required by the counter for the uncompleted allocation, the counter for the uncompleted allocation is allocated to the second Sketch storage space.

6. The method according to claim 4, characterized in that The first access part of the pipeline in the programmable switch also includes inserting a pre-processing module, and the method also includes: The data pre-insertion processing is performed by the insertion pre-processing module; wherein the pre-insertion processing includes flow monitoring process admission processing, Sketch insertion value pre-calculation processing of the byte granularity and centralized pre-calculation processing of independent logic gate operations; The dependent logic gate operations are evenly distributed to the first Sketch storage space and the second Sketch storage space for operation processing.

7. A flow result query method based on Sketch, characterized in that: Applied to a server, the method comprises: Obtain identification information of the traffic to be queried, and perform traffic query processing on the Sketch storage space of the programmable switch based on the identification information; Obtain a first query result of a packet granularity of the Sketch storage space, and obtain a second query result of a byte granularity of the Sketch storage space; The first query result and the second query result are mutually verified to obtain a first target query result corresponding to the packet granularity and a second target query result corresponding to the byte granularity after mutual verification.

8. The method according to claim 7, characterized in that The mutual verification by the first query result and the second query result includes: Compare the magnitude relationship between the first query result and the first value, and use the smallest value between the first query result and the first value as the first target query result; wherein the first value is determined based on the second query result and the Ethernet minimum frame length; Compare the size relationship between the second query result and the second numerical value, and use the minimum value between the second query result and the second numerical value as the second target query result; wherein the second numerical value is determined based on the first query result and the maximum transmission unit in data transmission.

9. A flow processing device based on Sketch, characterized in that: Applied to a programmable switch, the device comprises: A data statistics module is used to perform data statistics at packet and byte granularities on the traffic to be detected; A data insertion module is used to insert the data statistics results of the packet granularity into the Sketch storage space of the programmable switch, and insert the data statistics results of the byte granularity into the Sketch storage space of the programmable switch through compression and probability compensation; wherein the compression and probability compensation method is used to convert the data statistics results of the byte granularity into an order of magnitude.

10. A flow result query device based on Sketch, characterized in that: Applied to a server, the device comprises: A data query module, which obtains identification information of the traffic to be queried, and performs traffic query processing on the Sketch storage space of the programmable switch based on the identification information; A query result acquisition module, used to acquire a first query result of a packet granularity of the Sketch storage space, and acquire a second query result of a byte granularity of the Sketch storage space; A data verification module is used to perform mutual verification through the first query result and the second query result to obtain the first target query result corresponding to the packet granularity and the second target query result corresponding to the byte granularity after mutual verification.

11. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 8.