Distributed AI training low-entropy flow measurement method and device based on Sketch, equipment and medium

Abstract

The invention discloses a Sketch-based distributed AI training low-entropy flow measurement method, apparatus and device, and a medium, sliding of a window in a low-quality flow screening mechanism is simulated through reduction of a life value, and whether a data flow still exists in the window is represented through resetting of the life value. Whenever a data packet arrives, firstly checking whether a corresponding data stream exists in a temporary storage queue or not; and if not, recording the data stream frequency and initializing the life value for the data stream. Otherwise, updating the data stream information and resetting the life value of the data stream information. And meanwhile, life values of other streams are reduced. And when the frequency of a certain data stream in the temporary storage queue is greater than a high threshold value, the possibility that the data stream becomes a frequent stream is extremely high, and at the moment, the data stream is moved from the queue to a potential layer. And when the life value of a certain data stream is reduced to 0, whether the data stream is inserted into a potential layer or a low-quality layer is determined according to whether the frequency is greater than a low threshold. According to the method, the low-entropy flow can be measured more accurately.

Description

Technical Field

[0001] The present invention relates to the field of network traffic measurement, and in particular to a Sketch-based distributed AI training low-entropy traffic measurement method, device, equipment and medium. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, large-scale distributed AI training has become a key means of improving model performance. However, while this distributed architecture significantly improves the efficiency of large-model training, it also presents numerous challenges. Low entropy in traffic patterns is particularly prominent, as it not only reduces network utilization but also leads to severe link congestion and network latency, hindering parameter synchronization and reducing training efficiency. Measuring and managing network traffic is crucial in this regard.

[0003] Sketch is a probabilistic data structure that can efficiently process large amounts of network traffic data within limited storage space. It uses hash mapping and probability estimation to approximate traffic characteristics, significantly improving measurement accuracy without sacrificing too much precision.

[0004] Existing Sketch measurement methods can be divided into two categories: Sketch methods based on screening separation and Sketch methods based on hierarchical counters. The core idea of ​​existing methods based on screening separation is "selecting the best." Taking ElasticSketch as an example, it consists of two parts: a heavy flow layer and a small flow layer. Through its "ostracism method", it votes to expel small flows to the small flow layer and stores large flows in the heavy flow layer, thereby reducing the error impact of large flows on small flows. The core idea of ​​the Sketch method based on hierarchical counters is to use counters of different sizes at different levels to record flows of different sizes. Taking Pyramid Sketch as an example, it adopts a pyramid-shaped structure and dynamically allocates memory resources for traffic through counter overflow, which not only improves measurement accuracy but also fully utilizes storage space.

[0005] However, in the low-entropy traffic scenario of distributed AI training, the Sketch method, based on screening and separation, cannot predict whether traffic is a significant flow. Therefore, the counters in the heavy flow layer of Sketch must be large enough to accommodate heavy flows. Larger counters mean fewer counters, which also leads to more hash collisions, which undoubtedly reduces measurement accuracy. Small flows also cannot fully utilize the storage bits of large counters, resulting in a waste of storage resources.

[0006] Compared to the skewed traffic with a heavy-tail distribution, low-entropy traffic has less diversity. For the same number of packets, there are fewer low-entropy flows, making the overall traffic more concentrated. For the Sketch method based on hierarchical counters, this can easily lead to frequent overflow of certain counters in Sketch, ultimately affecting the accuracy of traffic detection. Summary of the Invention

[0007] In order to solve the technical problems of inaccurate measurement and performance degradation of existing Sketch-based traffic detection algorithms when dealing with distributed AI training low-entropy traffic, the present invention provides a Sketch-based distributed AI training low-entropy traffic measurement method, device, equipment and medium that can effectively distinguish low-entropy traffic of different sizes to achieve accurate measurement.

[0008] In order to achieve the above technical purpose, the technical solution of the present invention is:

[0009] A Sketch-based distributed AI training low-entropy traffic measurement method performs the following steps whenever a new data packet arrives:

[0010] Step 1: Check the information of the arriving data packet to determine whether the data flow to which the data packet belongs exists in the temporary queue;

[0011] If it does not exist, the flow ID of the data flow to which the data packet belongs is recorded in the temporary queue, and the initial frequency and initial life value are assigned to the data flow, and then the life values ​​of other data flows in the temporary queue are reduced by a predetermined value;

[0012] If it exists, the frequency of the data flow to which the data packet belongs is increased by a preset value, and the life value of other data flows in the temporary queue is reduced by a predetermined value;

[0013] Step 2: Check whether there is a data flow with a frequency greater than a preset high threshold in the temporary queue. If so, move the data flow from the temporary queue to the potential layer of Sketch for recording;

[0014] Step 3: Check whether there is a data stream in the temporary queue whose health value has returned to 0. If so, further check the frequency of the corresponding data stream:

[0015] When the frequency of the data stream is greater than the preset low threshold, the data stream is moved from the temporary queue to the potential layer of Sketch for recording;

[0016] When the frequency of the data stream is less than or equal to the preset low threshold, the data stream is moved from the temporary queue to the low-quality layer of Sketch for recording;

[0017] When a new data packet arrives, return to step 1 and execute the loop.

[0018] Furthermore, before the first data packet arrives, the following initialization steps are also included:

[0019] Allocate specific proportions of storage space for the temporary queue, Sketch's potential layer, and low-quality layer and clear the storage space; then set the values ​​of the high threshold, low threshold, initial frequency, and initial health; and set a preset value to increase the frequency of the data stream and a predetermined value to reduce the health of other data streams each time the corresponding data packet arrives.

[0020] Furthermore, the step of dynamically adjusting the low threshold is also included:

[0021] At the end of each cycle, all data flows in the current temporary queue are sorted according to their frequencies, and then, starting from the data flow with the lowest frequency, the highest frequency value among the data flows that account for a predetermined proportion of the total number is taken as the low threshold of the next cycle, and then the dynamic adjustment of the low threshold is cyclically performed in the next cycle; wherein the cycle is a preset initial life value, that is, starting from initialization, each time a data packet with the same number as the initial life value is reached is considered a cycle.

[0022] Furthermore, in step 1, whether the data flow to which the data packet belongs exists in the temporary queue is determined by comparing the flow ID in the newly arrived data packet with the flow IDs of other data flows in the temporary queue; if the same flow ID exists, it is considered that the data flow already exists in the temporary queue; otherwise, it is considered that it does not exist.

[0023] The present invention also provides a Sketch-based distributed AI training low-entropy flow measurement device, comprising:

[0024] A temporary queue processing module is used to check the information of the arriving data packet and determine whether the data flow to which the data packet belongs exists in the temporary queue;

[0025] If it does not exist, the flow ID of the data flow to which the data packet belongs is recorded in the temporary queue, and the initial frequency and initial life value are assigned to the data flow, and then the life values ​​of other data flows in the temporary queue are reduced by a predetermined value;

[0026] If it exists, the frequency of the data flow to which the data packet belongs is increased by a preset value, and the life value of other data flows in the temporary queue is reduced by a predetermined value;

[0027] The high-potential data flow detection module is used to check whether there is a data flow with a frequency greater than a preset high threshold in the temporary queue. If so, the data flow is moved from the temporary queue to the potential layer of Sketch for recording;

[0028] The health value detection module is used to check whether there is a data stream with a health value of 0 in the temporary queue. If so, it further checks the frequency of the data stream:

[0029] When the frequency of the data stream is greater than the preset low threshold, the data stream is moved from the temporary queue to the potential layer of Sketch for recording;

[0030] When the frequency of the data stream is less than or equal to the preset low threshold, the data stream is moved from the temporary queue to the low-quality layer of Sketch for recording.

[0031] Furthermore, it also includes an initialization module, which is used to divide the specific proportions of storage space occupied by the temporary queue, Sketch's potential layer and low-quality layer and clear the storage space before the first data packet arrives; then set the values ​​of the high threshold, low threshold, initial frequency and initial life value; and set a preset value to increase the frequency of the data stream and a predetermined value to reduce the life value of other data streams each time the corresponding data packet of the data stream arrives.

[0032] Furthermore, it also includes a low threshold dynamic adjustment module, which is used to sort all data streams in the current temporary queue according to frequency at the end of each cycle, and then, starting from the data stream with the lowest frequency, take the highest frequency value among the data streams that account for a predetermined proportion of the total number as the low threshold of the next cycle, and then continue to cyclically perform dynamic adjustment of the low threshold in the next cycle; wherein the cycle is a preset initial life value, that is, starting from initialization, each time a data packet with the same number as the initial life value is reached is considered a cycle.

[0033] Furthermore, in the temporary queue processing module, whether the data flow to which the data packet belongs exists in the temporary queue is determined by comparing the flow ID in the newly arrived data packet with the flow IDs of other data flows in the temporary queue. If the same flow ID exists, the data flow is considered to already exist in the temporary queue; otherwise, it is considered not to exist.

[0034] The present invention also provides an electronic device, comprising:

[0035] one or more processors;

[0036] a storage device for storing one or more programs,

[0037] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0038] The present invention also provides a computer-readable medium storing a computer program, wherein the computer program implements the aforementioned method when executed by a processor.

[0039] The technical effect of the present invention is that the present invention uses a temporary queue to conduct advance observation and predictive analysis of traffic, and combines the present invention to set the frequency and life value of traffic to analyze the possibility of traffic becoming a frequent flow, and reasonably distinguishes traffic of different sizes in low entropy traffic, reducing the mutual influence between traffic of different sizes, thereby achieving more accurate measurement of low entropy traffic. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the process of the present invention;

[0041] Figure 2 Schematic diagram of the full-flow estimation error for the CAIDA2018 dataset in an embodiment of the present invention; (a) is a schematic diagram of the full-flow mean absolute error, and (b) is a schematic diagram of the full-flow mean relative error;

[0042] Figure 3 Schematic diagram of full-flow estimation error for a low-entropy dataset in an embodiment of the present invention; (a) is a schematic diagram of the full-flow mean absolute error, and (b) is a schematic diagram of the full-flow mean relative error;

[0043] Figure 4 Schematic diagram of the insertion and query throughput of the CAIDA2018 dataset in an embodiment of the present invention; (a) is a schematic diagram of the insertion throughput, and (b) is a schematic diagram of the query throughput;

[0044] Figure 5 Schematic diagram of the insertion and query throughput of a low-entropy dataset in an embodiment of the present invention; (a) is a schematic diagram of the insertion throughput, and (b) is a schematic diagram of the query throughput. DETAILED DESCRIPTION

[0045] This paper specifically designs a measurement method based on the unique characteristics of low-entropy traffic in distributed AI training. Since the fundamental cause of low-entropy traffic is the sparse logical topology of distributed AI training and the scheduling of training tasks, and due to the execution and scheduling of training tasks, some traffic will appear repeatedly over long periods of time due to frequent communication between GPUs, forming large flows, while some traffic may appear briefly and infrequently due to long periods of no communication between GPUs, eventually becoming small flows.

[0046] Therefore, the present invention first defines the items in the traffic as long-term non-appearance and recent appearance based on the intervals between the appearance of their data packets. At the same time, based on the possibility of the traffic becoming a frequent flow, it is defined as low quality, with certain potential, and very potential. As for the interval of appearance, a longer sliding window is set. When the item slides out of the sliding window and does not reappear in this longer sliding window, it can be defined as an item that has not appeared for a long time. On the contrary, if it reappears in the window, it is defined as a recently appeared item. When the frequency of an item that has not appeared for a long time is lower than a very small low threshold, it is considered that this traffic is unlikely to become a frequent flow and can be defined as a low-quality flow. If the frequency of appearance of this item that has not appeared for a long time is higher than this low threshold, it is considered that this item still has certain potential. Finally, when the frequency of appearance of a recently appeared flow is higher than a high threshold, it is considered that it has a very high possibility of becoming a frequent flow and has great potential.

[0047] However, it is not wise to directly use a sliding window in the actual Sketch structure because the sliding window consumes memory resources and cannot be used directly to record frequency. Recording the frequency of each data stream separately will waste additional memory resources. In order to better combine the traffic prediction mechanism and make full use of memory resources, this paper proposes a lifecycle mechanism:

[0048] When a data flow packet is received, it is first placed in a temporary queue. The temporary queue stores three pieces of data flow information: flow ID, frequency, and assigned health value. When a data flow packet arrives, the system first checks whether the corresponding data flow exists in the temporary queue. If not, the data flow ID and frequency (the key-value pair for the data flow) are recorded, and an initial health value is assigned to the data flow. If it exists, the frequency of the data flow is updated and its health value is reset. Furthermore, each time a data flow packet arrives, the health values ​​of all other flows in the queue are reduced by a predetermined value. A high threshold and a low threshold are also set for the data flows in the temporary queue. If the health value of a data flow in the temporary queue has not yet reached zero, but the frequency of data packets arriving exceeds the high threshold, the data flow is considered to have a high probability of becoming a frequent flow. In this case, the data flow is removed from the temporary queue and inserted into the Sketch's potential layer. The data flow is then removed from the temporary queue, moving it to the potential layer. When a data stream's health value drops to 0, its insertion into Sketch's potential layer or Sketch's low-quality layer is determined based on whether its frequency is greater than a low threshold. If the frequency of the data stream whose health value drops to 0 is greater than the low threshold, it is inserted into the potential layer and removed from the temporary queue. If the frequency of the data stream whose health value drops to 0 is less than or equal to the low threshold, it is inserted into the low-quality layer and removed from the temporary queue.

[0049] Therefore, in the lifecycle mechanism proposed in this invention, the window sliding in the low-quality stream screening mechanism is simulated by decreasing the life value, and the reset of the life value is used to simulate whether the data stream still exists in the window. This avoids the extra memory resource waste that may be caused by using a sliding window.

[0050] The present invention will be further described below with reference to the embodiments.

[0051] See also Figure 1 , the specific steps of an embodiment provided by the present invention are as follows:

[0052] Step 1: Initialization. Set the storage space proportions for the temporary queue, the potential layer of the Sketch structure, and the low-quality layer, respectively. In this embodiment, the memory space of the temporary queue, the potential layer of the Sketch structure, and the low-quality layer accounts for 15%, 60%, and 25% of the total storage space, respectively. Set the high threshold to 50, the low threshold to 5, the initial frequency to 1, and the initial life span to 5000. Each time a new data packet arrives, the frequency of the data stream corresponding to this packet is increased by a preset value of 1, while the life spans of other data streams are reduced by a preset value of 1. Also, set the period for recording the frequencies of all data streams to dynamically adjust the low threshold to the initial life span. That is, starting from initialization, each time a number of data packets equal to the initial life span arrives as a period.

[0053] It should be noted that the above percentages and specific values ​​are the values ​​set in this embodiment for the purpose of illustrating the present invention. In actual implementation, they can be adjusted as needed to meet the implementation needs in different situations. Here, the counters of the temporary queue and Sketch are also reset to 0. The counter here refers to the container for recording frequency. The purpose of resetting it to 0 is to reset and clear the information of the entire Sketch to achieve initialization. In addition, in the temporary queue, the three information of the data flow recorded in this embodiment are the life value assigned to each data flow, and the key-value pair information, where the key-value pair information is the flow ID and frequency of the data flow.

[0054] Step 2: When a new data packet arrives, read the traffic key-value pair information in the data packet, then insert it into the temporary queue and go to step 3.

[0055] Step 3: Determine whether the data flow corresponding to the packet exists in the temporary queue. If so, reset the health value of the flow to 5000 (the initial health value), increase the frequency of the flow by 1, and decrement the health values ​​of other flows in the temporary queue by 1. Then, proceed to step 4. Otherwise, record the flow ID of the data flow in the temporary queue and assign an initial frequency of 1, thereby recording the key-value pair information of the flow, assigning the flow an initial health value of 5000, and decrementing the health values ​​of other flows in the temporary queue by 1. Then, proceed to step 4.

[0056] Step 4: Check if there is any traffic in the temporary queue whose health value has not decreased to 0 and whose frequency is greater than the high threshold of 50. If there is, it is considered that this traffic has the potential to become a frequent flow. At this time, this traffic is inserted into the Sketch potential layer and recorded. The key-value pair information of this traffic is deleted from the temporary queue, that is, the traffic is moved to the Sketch potential layer, and then go to step 5. If there is no such traffic, go directly to step 5.

[0057] Step 5: Check if there is any traffic in the temporary queue whose health value has decreased to 0 and whose frequency is greater than the low threshold. If so, it is considered that this traffic still has a certain potential to become a frequent flow. At this time, this traffic is also moved to the potential layer of Sketch and recorded, and then go to step 6. If not, go directly to step 6.

[0058] Step 6: Check if there is any traffic in the temporary queue whose health value has decreased to 0 and whose frequency is less than or equal to the low threshold. If there is, it is considered unlikely to become a frequent flow, and the traffic is determined to be low-quality, moved to the low-quality layer of Sketch, and then go to step 7. If there is no such traffic, it means that none of the three conditions exist in the temporary queue, so go directly to step 7.

[0059] Step seven is to determine whether the current cycle has ended. If so, all data flows in the temporary queue are sorted by frequency, and then, starting from the data flow with the lowest frequency, the highest frequency value among the data flows that account for a predetermined proportion of the total is taken as the lower threshold value for the next cycle. In this embodiment, the predetermined proportion is 15%, that is, among the flows sorted in the last 15%, the highest frequency among these flows is taken as the lower threshold value for the next cycle, and then the process returns to step two and loops. If the cycle has not ended, the process directly returns to step two and loops. Determining whether a cycle has ended here means counting the number of data packets that have arrived. When the number of data packets that is equal to the initial life value, that is, 5,000 data packets, is reached, a cycle is considered to have ended. It should also be noted here that the predetermined proportion of the total number and the cycle length can be adjusted as needed in actual implementation to meet the implementation needs in different situations.

[0060] It should also be noted that when implementing dynamic adjustment of the low threshold, a Count-Min Sketch can be used to count the frequency of traffic. At the end of each cycle, the Count-Min Sketch can be cleared. The reason for performing dynamic adjustment of the low threshold is that the network environment is complex and changeable, and fixed threshold settings are often difficult to cope with various emergencies. By dynamically adjusting the low threshold, this embodiment can update the low threshold in real time based on the actual distribution of traffic, making classification more accurate and reducing the possibility of misjudgment and missed judgment. The high threshold is not dynamically adjusted because, compared to the low threshold, the high threshold is more of an optimization parameter, and its greatest role is to improve the processing efficiency of the temporary queue for high-frequency flows. That is, when the frequency of a flow is higher than the high threshold, the flow can be stored in the potential layer more quickly, thereby making storage space for the temporary queue, thereby optimizing processing efficiency. Because the gap between the high and low thresholds is large, even if a flow's frequency is always slightly lower than the high threshold, preventing it from entering the potential layer before its life value reaches zero, when the life value of this flow reaches zero, its frequency will still be greater than the low threshold, and it will eventually be identified as a flow with certain potential and moved to the potential layer. Therefore, the fixed setting of the high threshold does not have a decisive impact on the present invention's prediction and classification of traffic. After comprehensively considering the resource consumption required to dynamically adjust the high threshold, the present invention no longer performs dynamic adjustment of the high threshold.

[0061] The present invention also provides a Sketch-based distributed AI training low-entropy flow measurement device, comprising:

[0062] The initialization module allocates specific storage space proportions for the temporary queue, Sketch's potential layer, and low-quality layer before the first packet arrives, and clears the storage space. It then sets the high and low thresholds, initial frequency, and initial health. It also sets a preset value to increase the frequency of a data stream and a predetermined value to decrease the health of other data streams each time a packet from that stream arrives.

[0063] The temporary queue processing module is used to check the information of the arrived data packet and determine whether the data flow to which the data packet belongs exists in the temporary queue.

[0064] If it does not exist, the flow ID of the data flow to which the data packet belongs is recorded in the temporary queue, and the initial frequency and initial life value are assigned to the data flow, and then the life values ​​of other data flows in the temporary queue are reduced by a predetermined value.

[0065] If it exists, the frequency of the data flow to which the data packet belongs is increased by a preset value, and the life values ​​of other data flows in the temporary queue are reduced by a predetermined value.

[0066] The high potential data flow detection module is used to check whether there is a data flow with a frequency greater than a preset high threshold in the temporary queue. If so, the data flow is moved from the temporary queue to the potential layer of Sketch for recording.

[0067] The health detection module is used to check whether there are data streams in the temporary queue whose health value has returned to 0. If so, it further checks the frequency of the data stream. If the frequency of the data stream is greater than the preset low threshold, the data stream is moved from the temporary queue to the potential layer of Sketch for recording. If the frequency of the data stream is less than or equal to the preset low threshold, the data stream is moved from the temporary queue to the low-quality layer of Sketch for recording.

[0068] The low threshold dynamic adjustment module is used to sort all data flows in the current temporary queue by frequency at the end of each cycle. Starting with the lowest-frequency data flow, it takes the highest frequency value among the data flows that account for a predetermined proportion of the total number of data flows as the low threshold for the next cycle. Dynamic adjustment of the low threshold is then repeated cyclically in the next cycle. The cycle is the preset initial life value. That is, from initialization, each time a number of data packets equal to the initial life value arrives, it counts as a cycle.

[0069] According to an embodiment of the present invention, the present invention also provides an electronic device and a computer-readable medium.

[0070] Electronic equipment includes:

[0071] one or more processors;

[0072] a storage device for storing one or more programs,

[0073] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0074] In practice, users can use electronic devices acting as terminal devices to interact with servers, also serving as electronic devices, over a network to receive or send messages. Terminal devices are generally various electronic devices equipped with a display and operated through a human-computer interface, including but not limited to smartphones, tablets, laptops, and desktop computers. Various specific application software can be installed on terminal devices as needed, including but not limited to web browsers, instant messaging software, social networking platforms, and shopping apps.

[0075] The server is a network service end used to provide various services. The distributed AI training low-entropy traffic measurement method provided in this embodiment is generally executed by the server. In actual application, under the necessary conditions, the terminal device can also directly execute the distributed AI training low-entropy traffic measurement method. Accordingly, the distributed AI training low-entropy traffic measurement device can be set on the server. Similarly, under the necessary conditions, the distributed AI training low-entropy traffic measurement device can also be set in the terminal device.

[0076] Similarly, the computer-readable medium of the present invention stores a computer program thereon, and when the computer program is executed by a processor, a distributed AI training low-entropy traffic measurement method of an embodiment of the present invention is implemented.

[0077] The following is a comparative experimental analysis of the method provided by the present invention and other existing solutions:

[0078] The experimental setup is as follows:

[0079] Platform: This comparative experiment was conducted on a virtual machine equipped with an eight-core CPU (Intel(R) Core(TM) i7-10875H CPU @ 2.30GHz). Each core has three levels of cache: 64KB of L1 data cache, 48KB of L1 instruction cache, 512KB of L2 cache, and 16MB of shared L3 cache. The server ran Ubuntu 20.04.4LTS.

[0080] Dataset: This comparative experiment uses an anonymized real-world dataset collected by CAIDA in 2018. The CAIDA2018 dataset contains 2.4 million packets and 50,000 flows. This paper also uses a low-entropy dataset, a synthetic dataset designed to simulate the low-entropy traffic scenarios encountered during distributed AI training. Compared to the maximum possible entropy value of 16.61, this dataset has an entropy of only 7.81.

[0081] Experimental Methods: In the flow estimation task, the proposed solution is compared with CM Sketch, CUSketch, Count Sketch, Elastic Sketch, Sting Sketch, and Heavy Guardian. In the throughput task, the proposed solution is compared with CM Sketch, CU Sketch, Elastic Sketch, Heavy Guardian, and Pyramid Sketch. The proposed solution uses the full-flow estimation's AAE and ARE to evaluate measurement accuracy, and throughput to evaluate measurement speed.

[0082] Figure 2 and Figure 3The scheme of this comparative experiment (called FP-Sketch), CM Sketch, CU Sketch, Count Sketch, Elastic Sketch, Sting Sketch and Heavy Guardian are shown for the CAIDA2018 dataset and low entropy dataset, when using different memory spaces, the AAE and ARE of the full flow estimation are presented. Figure 2 Schematic diagram of the full flow estimation error for the CAIDA2018 dataset, (a) is the schematic diagram of the full flow mean absolute error, and (b) is the schematic diagram of the full flow mean relative error. Figure 3 Schematic diagram of the full-flow estimation error for the low-entropy data set; (a) is a schematic diagram of the full-flow mean absolute error, and (b) is a schematic diagram of the full-flow mean relative error.

[0083] Figure 4 and Figure 5 The insertion and query throughput of the scheme (FP-Sketch), CM Sketch, CU Sketch, ElasticSketch, Heavy Guardian and Pyramid Sketch in this comparative experiment are shown for the CAIDA2018 dataset and the low entropy dataset under different memory space sizes. Figure 4 Schematic diagram of the insertion and query throughput of the CAIDA2018 dataset; (a) is the insertion throughput diagram, and (b) is the query throughput diagram. Figure 5 Schematic diagram of insertion and query throughput for low entropy datasets; (a) is a schematic diagram of insertion throughput, and (b) is a schematic diagram of query throughput.

[0084] Looking at full-flow AAE and ARE, existing methods have significantly improved accuracy compared to classic Sketch methods such as CM Sketch, CU Sketch, and CountSketch. Separation-based Sketch methods such as ElasticSketch and Heavy Guardian reduce the severe impact of large flows on small flows by separating large and small flows. Hierarchical counter-based Sketch methods such as Sting Sketch also achieve higher accuracy by rationally allocating memory space. The solution in this comparative experiment (FP-Sketch) achieves higher measurement accuracy in AAE and ARE than existing methods by specifically analyzing and predicting flows of different sizes in low-entropy traffic, with an average accuracy improvement of 38.6%.

[0085] In terms of throughput, the throughput of existing solutions varies little with memory usage. In terms of insertion throughput, CMSketch and CU Sketch achieve higher throughput than existing Sketch methods, as their insertion process only requires a simple hashing process and is simple to operate. Pyramid Sketch uses a pyramid-shaped hierarchical structure to dynamically store large and small streams, but frequent overflow operations make it difficult to achieve high throughput. Compared to ElasticSketch and Heavy Guardian, which require multiple comparisons of key-value information in the heavy stream layer, FP-Sketch's hashing operation can store stream information records more quickly, achieving a high level of throughput, with an average insertion throughput improvement of 49.4%.

[0086] The simple operations of CM and CU still make their query throughput higher than other existing solutions. Elastic Sketch mainly uses hash functions to determine the location of stream storage during queries, and the number of stream key information comparisons is relatively small, so the query is relatively fast. Because Pyramid Sketch needs to calculate multiple counter overflows during queries, its query throughput is not optimistic. Heavy Guardian needs to repeatedly compare multiple key-value pairs of information in the heavy flow layer, which also causes its throughput to decrease. FP-Sketch uses a temporary queue to determine the type of flow and then queries the traffic size through hashing, achieving a higher throughput than Pyramid Sketch and Heavy Guardian.

Claims

1. A distributed AI training low entropy flow measurement method based on Sketch, characterized in that: Whenever a new packet arrives, the following steps are performed: Step 1: Check the information of the arriving data packet to determine whether the data flow to which the data packet belongs exists in the temporary queue; If it does not exist, the flow ID of the data flow to which the data packet belongs is recorded in the temporary queue, and the initial frequency and initial life value are assigned to the data flow, and then the life values ​​of other data flows in the temporary queue are reduced by a predetermined value; If it exists, the frequency of the data flow to which the data packet belongs is increased by a preset value, and the life value of other data flows in the temporary queue is reduced by a predetermined value; Step 2: Check whether there is a data flow with a frequency greater than a preset high threshold in the temporary queue. If so, move the data flow from the temporary queue to the potential layer of Sketch for recording; Step 3: Check whether there is a data stream in the temporary queue whose health value has returned to 0. If so, further check the frequency of the corresponding data stream: When the frequency of the data stream is greater than the preset low threshold, the data stream is moved from the temporary queue to the potential layer of Sketch for recording; When the frequency of the data stream is not greater than the preset low threshold, the data stream is moved from the temporary queue to the low-quality layer of Sketch for recording; When a new data packet arrives, return to step 1 and execute the loop.

2. The method according to claim 1, characterized in that Before the first data packet arrives, the following initialization steps are also performed: Allocate specific proportions of storage space for the temporary queue, Sketch's potential layer, and low-quality layer and clear the storage space; then set the values ​​of the high threshold, low threshold, initial frequency, and initial health; and set a preset value to increase the frequency of the data stream and a predetermined value to reduce the health of other data streams each time the corresponding data packet arrives.

3. The method according to claim 1, characterized in that Also includes steps for dynamically adjusting the low threshold: At the end of each cycle, all data flows in the current temporary queue are sorted according to their frequencies, and then, starting from the data flow with the lowest frequency, the highest frequency value among the data flows that account for a predetermined proportion of the total number is taken as the low threshold of the next cycle, and then the dynamic adjustment of the low threshold is cyclically performed in the next cycle; wherein the cycle is a preset initial life value, that is, starting from initialization, each time a data packet with the same number as the initial life value is reached is considered a cycle.

4. The method according to claim 1, wherein In step 1, whether the data flow to which the data packet belongs exists in the temporary queue is determined by comparing the flow ID in the newly arrived data packet with the flow IDs of other data flows in the temporary queue. If the same flow ID exists, the data flow is considered to already exist in the temporary queue; otherwise, it is considered not to exist.

5. A distributed AI training low entropy flow measurement device based on Sketch, characterized in that: include: A temporary queue processing module is used to check the information of the arriving data packet and determine whether the data flow to which the data packet belongs exists in the temporary queue; If it does not exist, the flow ID of the data flow to which the data packet belongs is recorded in the temporary queue, and the initial frequency and initial life value are assigned to the data flow, and then the life values ​​of other data flows in the temporary queue are reduced by a predetermined value; If it exists, the frequency of the data flow to which the data packet belongs is increased by a preset value, and the life value of other data flows in the temporary queue is reduced by a predetermined value; The high-potential data flow detection module is used to check whether there is a data flow with a frequency greater than a preset high threshold in the temporary queue. If so, the data flow is moved from the temporary queue to the potential layer of Sketch for recording; The health value detection module is used to check whether there is a data stream with a health value of 0 in the temporary queue. If so, it further checks the frequency of the data stream: When the frequency of the data stream is greater than the preset low threshold, the data stream is moved from the temporary queue to the potential layer of Sketch for recording; When the frequency of the data stream is not greater than the preset low threshold, the data stream is moved from the temporary queue to the low-quality layer of Sketch for recording.

6. The device according to claim 5, characterized in that It also includes an initialization module, which is used to divide the storage space occupied by the temporary queue, the potential layer of Sketch and the low-quality layer into specific proportions and clear the storage space before the first data packet arrives; then set the values ​​of the high threshold, the low threshold, the initial frequency and the initial life value; and set a preset value to increase the frequency of the data stream and a predetermined value to reduce the life value of other data streams each time the corresponding data packet of the data stream arrives.

7. The device according to claim 5, characterized in that It also includes a low threshold dynamic adjustment module, which is used to sort all data streams in the current temporary queue according to frequency at the end of each cycle, and then, starting from the data stream with the lowest frequency, take the highest frequency value among the data streams that account for a predetermined proportion of the total number as the low threshold of the next cycle, and then continue to cyclically perform dynamic adjustment of the low threshold in the next cycle; wherein the cycle is a preset initial life value, that is, starting from initialization, each time a data packet with the same number as the initial life value is reached is considered a cycle.

8. The device according to claim 5, characterized in that In the temporary queue processing module, whether the data flow to which the data packet belongs exists in the temporary queue is determined by comparing the flow ID in the newly arrived data packet with the flow IDs of other data flows in the temporary queue; if the same flow ID exists, the data flow is considered to already exist in the temporary queue; otherwise, it is considered not to exist.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

10. A computer-readable medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images