Two-stage learning-enhanced sketch aggregation method and system for distributed stream frequency estimation

Abstract

The application discloses a two-stage learning-enhanced Sketch aggregation method and system for distributed stream frequency estimation, and the method comprises the following steps: each measurement node extracts local counter features, load features and structure features from a basic Sketch for a to-be-queried element, and outputs a local additive correction term and a compact node feature vector through a local correction module to correct the residual of a local basic estimation value; the corrected local frequency estimation value and the compact node feature vector of each measurement node are sent to an aggregation point for aggregation, so that an aggregated Sketch feature and an aggregated basic estimation value are obtained; and through a merging perception correction module, a permutation-invariant statistical aggregation is carried out, and in combination with the number of nodes, normalized local stream length, load estimation and the aggregated Sketch feature, an aggregated additive correction term and a skip gate value are generated, so that a global frequency estimation value of the to-be-queried element is obtained. The application reduces error accumulation and improves the global frequency estimation precision.

Description

Technical Field

[0001] This invention relates to the fields of data stream processing, network measurement, approximate query processing, and learning-enhanced data structure technology, specifically to a two-level learning-enhanced Sketch aggregation method and system for distributed stream frequency estimation. Background Technology

[0002] Frequency estimation is a fundamental task in data stream processing, aiming to estimate the number of times an element appears in a data stream within a limited storage space. This task is widely used in scenarios such as network measurement, database approximate queries, anomaly detection, machine learning feature statistics, and streaming log analysis. Traditional methods typically employ Sketch structures such as Count-Min Sketch, Tower Sketch, HashPipe, FlowRadar, and ElasticSketch, maintaining a stream summary with sublinear space and constant-time update, and returning approximate values ​​of element frequencies during queries.

[0003] In large-scale distributed systems, the same global data stream is often observed in fragments by multiple measurement nodes. For example, in network measurement, different switches or edge nodes observe local traffic separately; in database or stream processing systems, different fragment nodes process local data separately. To obtain a global frequency view, a common practice is to maintain a local sketch on each node and perform counter-level merging of multiple sketches at an aggregation point, such as performing counter-by-counter incrementing on a Count-Min sketch.

[0004] While the aforementioned distributed aggregation method avoids centralized transmission of the original data, it introduces new error problems. The error in the basic Sketch primarily stems from hash collisions. When Sketches from K nodes are aggregated, the collision errors of each node accumulate during the aggregation process, causing the aggregated error distribution to differ from the error distribution of any single node. With a fixed total memory budget, the error of traditional aggregated Sketches may significantly amplify as the number of nodes increases.

[0005] Therefore, there is an urgent need for a two-level learning-enhanced Sketch aggregation method for distributed flow frequency estimation to solve the above problems. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a two-stage learning-enhanced Sketch aggregation method and system for distributed flow frequency estimation. This invention splits the learning-enhanced correction into two stages: local correction and merging-aware correction. The local correction module is deployed at each measurement node to reduce single-node hash collision residuals before aggregation. The merging-aware correction module is deployed at the aggregation point to further correct the global residuals after aggregation by incorporating cross-node statistical information, thereby reducing the global frequency estimation bias caused by the accumulation of hash collision errors during distributed Sketch aggregation.

[0007] The objective of this invention is achieved through the following technical solution: The first aspect of this invention provides a two-level learning-enhanced Sketch aggregation method for distributed flow frequency estimation, comprising the following steps: (1) Receive local data streams at K measurement nodes respectively, and maintain a basic Sketch that supports counter-level merging at each measurement node; (2) At each measurement node, local counter features, load features and structural features are extracted from the basic Sketch for the element to be queried, and local additive correction terms and compact node feature vectors are output through the local correction module; the local basic estimate of each measurement node is residually corrected according to the local additive correction terms so that each measurement node forms a corrected local frequency estimate. (3) Send the corrected local frequency estimates and compact node feature vectors of each measurement node to the aggregation point. At the aggregation point, perform counter-level aggregation on the Sketch counters or local frequency estimates of each measurement node to obtain aggregated Sketch features and aggregated basic estimates. At the aggregation point, perform permutation-invariant statistical aggregation on the compact node feature vectors through the merging sensing correction module. Combine the number of measurement nodes, the normalized local flow length and load estimate of each measurement node and the aggregated Sketch features to generate an aggregated additive correction term and a skip gate value. Based on the aggregated basic estimate, the aggregated additive correction term and the skip gate value, obtain the global frequency estimate of the element to be queried.

[0008] Furthermore, step (1) specifically includes: Each measurement node k receives a local data stream. Count the length of the local stream And map the elements in the local data stream to the corresponding counters according to the hash function of the basic Sketch; Each measurement node maintains a basic Sketch with the same structure, which includes a d-row, w-column counter matrix and d hash functions; When the base Sketch is Count-Min Sketch, each measurement node uses the same number of hash functions, the same counter matrix width, and the same hash random seed to support the aggregation point to merge counters at the same location item by item; For the element to be queried, the measurement node obtains the counter query values ​​corresponding to multiple hash rows from the base Sketch, and obtains the local base estimate value from the counter query values.

[0009] Furthermore, step (2) specifically includes the following sub-steps: (2.1) Extract the counter query value of the element to be queried in d hash rows, the local basic estimate obtained from the counter query value, the ratio feature, the collision intensity feature and the statistical summary feature from the basic Sketch as local counter features; (2.2) The local correction module includes a global encoder, a local encoder, a load decoder, and a correction decoder. The global encoder extracts a portion of the counters from the counter matrix of the basic Sketch to obtain node-level structural features. The load decoder estimates the node load based on the node-level structural features to obtain the load estimate. The local encoder processes the local counter features of the query element to obtain the local counter feature vector. The correction decoder fuses the node-level structural features and the local counter feature vector to output the local additive correction term. (2.3) Using the additive residual correction form, the corrected local frequency estimate is expressed as the sum of the local basic estimate and the local additive correction term; (2.4) Generate compact node feature vectors based on node-level structural features, load estimates and normalized local flow lengths.

[0010] Furthermore, the ratio feature is used to characterize the offset of each row counter query value relative to the base estimate, the collision intensity feature is used to characterize the intensity of the offset relative to the node average load, and the statistical summary features include mean, variance, maximum value, and median.

[0011] Furthermore, step (3) specifically includes the following sub-steps: (3.1) The aggregation point receives the basic Sketch counter, the corrected local frequency estimate and the compact node feature vector uploaded by each measurement node. At the aggregation point, the Sketch counter or local frequency estimate of each measurement node is aggregated at the counter level to obtain the aggregated Sketch feature. (3.2) The aggregation point extracts the local features, structural features, and basic estimates of the aggregation Sketch based on the aggregation Sketch features; (3.3) The merged sensing correction module includes a cross-node statistical pooling unit, a conditional modulation unit, an aggregation residual decoder, and a skip gate unit. The cross-node statistical pooling unit calculates the mean, variance, and maximum value of the compact node feature vectors uploaded by each measurement node and concatenates them to obtain the cross-node statistical features. The conditional modulation unit modulates the cross-node statistical features according to the number of measurement nodes, the normalized node flow length sequence, and the node load estimate sequence. The aggregation residual decoder receives the aggregation Sketch local features, the aggregation Sketch structural features, and the modulated cross-node statistical features, and outputs the aggregation additive correction term. The skip gate unit receives the aggregation Sketch local features and the aggregation basic estimate, and outputs the skip gate value used to control the effectiveness of the aggregation additive correction term. (3.4) Calculate the global frequency estimate of the element to be queried according to the formula “Global frequency estimate = aggregate basic estimate + skip gate value × aggregate additive correction term”.

[0012] Furthermore, the local correction module and the merged perception correction module adopt a two-stage training method. First, they are pre-trained on the synthetic data stream to cover different skewness, number of elements, and number of nodes. Then, they are fine-tuned on the real data stream or target business traffic. During both the pre-training and fine-tuning processes, the optimization objective is to minimize the total loss function and adjust the parameters of the local correction module and the merged perception correction module.

[0013] Furthermore, the total loss function includes local correction loss, aggregate correction loss, and skip gate regularization term, wherein the local correction loss is used to constrain the local additive correction term to fit the local basic estimate residual; the aggregate correction loss is used to constrain the aggregate additive correction term to fit the aggregate basic estimate residual; and the skip gate regularization term is used to suppress unnecessary secondary corrections when the aggregate basic estimate error is small.

[0014] A second aspect of this invention provides a system for implementing the above-described two-level learning-enhanced Sketch aggregation method for distributed flow frequency estimation, comprising: Multiple measurement nodes are used to receive local data streams, maintain basic Sketch, execute local calibration modules, and upload calibrated local frequency estimates and compact node feature vectors, respectively. The aggregation point is used to receive the basic Sketch counters, corrected local frequency estimates, and compact node feature vectors from multiple measurement nodes, perform counter-level aggregation on the Sketch counters or local frequency estimates, and execute the merge sensing correction module to output a global frequency estimate. The training module is used to train the local correction module and the merging-aware correction module based on synthetic and real data streams; The query interface is used to receive the identifier of the element to be queried and return the global frequency estimate of the element to be queried.

[0015] Furthermore, the local calibration module includes: The global encoder is used to extract a portion of the counters from the counter matrix of the base Sketch to obtain node-level structural features; The local encoder is used to process the counter query value of the element to be queried, the local base estimate obtained from the counter query value, the ratio feature, the collision intensity feature, and the statistical summary feature; The load decoder is used to estimate the node load based on the node-level structural characteristics and obtain the load estimate. The correction decoder is used to fuse node-level structural features and the output of the local encoder to generate a local additive correction term.

[0016] Furthermore, the merging sensing correction module includes: The cross-node statistical pooling unit is used to calculate the mean, variance, and maximum value of the compact node feature vectors uploaded by each measurement node, and then concatenate them to obtain the cross-node statistical features. The conditional modulation unit is used to modulate cross-node statistical characteristics based on the number of measured nodes, the normalized node flow length sequence, and the node load estimate sequence; The aggregated residual decoder is used to generate an aggregated additive correction term based on the aggregated Sketch local features, aggregated Sketch structural features, and modulated cross-node statistical features. Skip gating unit, used to generate skip gating values ​​that control the extent to which the aggregate additive correction term is effective.

[0017] Compared with the prior art, the beneficial effects of the present invention are: First, the present invention suppresses the hash collision error of each node by a local correction module before aggregation, so that the correction residuals of each node tend to be low bias and independent of each other, thereby reducing the risk of linear accumulation of errors during aggregation.

[0018] Second, after aggregation, the present invention uses the cross-node statistical features of the merged perception correction module to continue to correct the residuals, thus solving the problem that the local correction module cannot observe cross-node information.

[0019] Third, this invention employs additive residual correction and skip gating mechanisms to enhance correction when the aggregation error is large and suppress overcorrection when the aggregation basis estimate is close to the true value, thereby improving stability.

[0020] Fourth, this invention only requires uploading compact node feature vectors and mergeable Sketch information, without the need for centralized transmission of raw data streams, making it suitable for network measurement, edge computing, database sharding, and stream processing systems.

[0021] Fifth, the two correction modules of this invention can be designed as small neural networks to perform forward computation during the query phase without affecting the constant-level update path of the basic Sketch, thereby improving estimation accuracy with a lower number of model parameters and computational overhead.

[0022] Sixth, this invention reduces the error accumulation generated by existing single-layer learning-enhanced Sketch during distributed aggregation by splitting the correction process in distributed Sketch frequency estimation into local error suppression before aggregation and cross-node residual correction after aggregation, thereby improving the accuracy of global frequency estimation.

[0023] Seventh, this invention has higher estimation accuracy, lower query correction overhead, and smaller model parameter size. The method is simple and convenient, and has strong practicality. Attached Figure Description

[0024] Figure 1 This is a system architecture diagram of a two-level learning-enhanced Sketch aggregation method for distributed flow frequency estimation provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a local calibration module deployed on a measurement node according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a merging sensing correction module deployed at an aggregation point according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the training and inference process provided in an embodiment of the present invention. Detailed Implementation

[0025] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0027] It should be understood that although the terms first, second, third, etc., may be used in this invention to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of this invention, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0028] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0029] Example 1: System Overall Architecture like Figure 1 As shown, the two-level learning-enhanced Sketch aggregation system for distributed flow frequency estimation provided by this invention includes K measurement nodes (K≥2, preferably K=10~1000), one or more aggregation points (multiple aggregation points constitute a hierarchical aggregation structure), a training module, and a query interface.

[0030] Each measurement node is deployed at the network edge or data source, responsible for receiving local data streams and maintaining a base Sketch that supports counter-level merging. For a query element, the measurement node reads the counter query values ​​corresponding to multiple hash rows from the base Sketch and calculates the local base estimate. The base Sketch consists of a d-row, w-column counter matrix and d hash functions, where d represents the number of hash functions in the base Sketch and w represents the width of each row of counters in the base Sketch. Each measurement node uses the same hash function and random seed to ensure that the aggregation point can perform item-by-item merging of counters at the same row and column positions. The random seed is an additional input value used to initialize or perturb the hash calculation process; it randomly generates a set of values ​​used in the hash function calculation; for example, 100 values ​​are generated, but at the same position, different random seeds will generate different numbers.

[0031] Furthermore, the base sketch can be a Count-Min Sketch, a Tower Sketch, or other mergeable sketch structures. The preferred base sketch is a Count-Min Sketch. For Count-Min Sketches, the aggregation operation can be adding counters one by one. For other mergeable sketches, their corresponding merging rules can be used. For example, for sketches composed of neural networks such as MetaSketch and LegoSketch, the merging rules mainly achieve merging through methods such as averaging weights or querying multiple nodes and taking the maximum / minimum average.

[0032] Each measurement node, in addition to deploying a basic Sketch, also deploys a local calibration module. This local calibration module reads relevant counters and structural features from the basic Sketch during queries, outputs a local additive calibration term, and generates a compact node feature vector. Structural features refer to the d×w shape of the counter matrix, i.e., d rows × w columns. The compact node feature vector is used to transmit node-level load, structure, and error pattern information to the aggregation point. Its dimension is significantly smaller than the full Sketch counter matrix, for example, 32~128 dimensions, thereby reducing inter-node communication overhead. Error pattern information refers to the tendency of different types of basic Sketch errors. For example, in a Count-Min Sketch, the error pattern will inevitably lead to an overestimation of the output. In this case, the output can be reduced based on the error pattern information to approximate the true result.

[0033] The aggregation point is responsible for receiving Sketch information (i.e., counter matrix or corrected local frequency estimate) and compact node feature vectors uploaded by each measurement node. It performs counter-level aggregation on the Sketch counters or estimation results to obtain the aggregated basic estimate. A merging sensing correction module is deployed on the aggregation point. The merging sensing correction module uses the aggregated Sketch features and cross-node statistical features to perform secondary residual correction on the aggregated basic estimate.

[0034] The training module is used to train the local calibration module and the merging sensing calibration module offline or periodically based on synthetic and real data streams. The query interface is used to receive the query element, trigger each measurement node and aggregation point to execute a two-level calibration process, and return the global frequency estimation result.

[0035] Example 2: Two-level learning reinforcement Sketch aggregation method See Figure 1 The two-level learning-enhanced Sketch aggregation method for distributed flow frequency estimation of the present invention specifically includes the following steps: (1) Receive local data streams at K measurement nodes respectively, and maintain a basic Sketch that supports counter-level merging at each measurement node.

[0036] (1.1) Each measurement node k receives a local data stream. Count the length of the local stream And it maps the elements in the local data stream to the corresponding counters according to the basic Sketch hash function.

[0037] (1.2) Each measurement node maintains a basic Sketch with the same structure, which includes a d-row, w-column counter matrix and d hash functions.

[0038] (1.3) When the base Sketch is Count-Min Sketch, each measurement node uses the same number of hash functions, the same counter matrix width, and the same hash random seed to support the aggregation point to merge counters at the same location item by item.

[0039] (1.4) For the element to be queried, the measurement node obtains the counter query values ​​corresponding to multiple hash rows from the base Sketch, and obtains the local base estimate value from the counter query values.

[0040] (2) At each measurement node, local counter features, load features and structural features are extracted from the basic Sketch for the element to be queried, and local additive correction terms and compact node feature vectors are output through the local correction module. The local basic estimate of each measurement node is residually corrected according to the local additive correction terms so that each measurement node forms a corrected local frequency estimate.

[0041] (2.1) Extract the counter query value of the element to be queried in d hash rows, the local basic estimate value obtained from the counter query value, the ratio feature, the collision intensity feature and the statistical summary feature from the basic Sketch as local counter features; among them, the ratio feature is used to characterize the offset of the counter query value of each row relative to the basic estimate value, the collision intensity feature is used to characterize the strength of the offset relative to the average load of the node, and the statistical summary features include mean, variance, maximum value, median, etc.

[0042] (2.2) The local correction module includes a global encoder, a local encoder, a load decoder, and a correction decoder. The global encoder extracts a portion of the counters from the counter matrix of the basic Sketch to obtain node-level structural features. The load decoder estimates the node load based on the node-level structural features to obtain the load estimate. The local encoder processes the local counter features of the query element to obtain the local counter feature vector. The correction decoder fuses the node-level structural features and the local counter feature vector to output the local additive correction term.

[0043] (2.3) The additive residual correction form is adopted, and the corrected local frequency estimate is expressed as the sum of the local basic estimate and the local additive correction term, which is used to directly fit the hash collision residuals that have not been eliminated by the basic Sketch.

[0044] (2.4) Generate compact node feature vectors based on node-level structural features, load estimates and normalized local flow lengths, and use the compact node feature vectors as inputs for cross-node statistical aggregation.

[0045] (3) Send the corrected local frequency estimates and compact node feature vectors of each measurement node to the aggregation point. At the aggregation point, perform counter-level aggregation on the Sketch counters or local frequency estimates of each measurement node to obtain aggregated Sketch features and aggregated basic estimates. At the aggregation point, perform permutation-invariant statistical aggregation on the compact node feature vectors through the merging sensing correction module. Combine the number of measurement nodes, the normalized local flow length and load estimate of each measurement node and the aggregated Sketch features to generate an aggregated additive correction term and a skip gate value. Based on the aggregated basic estimate, the aggregated additive correction term and the skip gate value, obtain the global frequency estimate of the element to be queried.

[0046] (3.1) The aggregation point receives the basic Sketch counter, the corrected local frequency estimate and the compact node feature vector uploaded by each measurement node. The aggregation point performs counter-level aggregation on the Sketch counter or local frequency estimate of each measurement node to obtain the aggregated Sketch feature.

[0047] (3.2) The aggregation point extracts the local features, structural features and basic estimates of the aggregation Sketch based on the aggregation Sketch features.

[0048] (3.3) The merged sensing correction module includes a cross-node statistical pooling unit, a conditional modulation unit, an aggregation residual decoder, and a skip gate unit. The cross-node statistical pooling unit calculates the mean, variance, and maximum value of the compact node feature vectors uploaded by each measurement node, and concatenates them to obtain cross-node statistical features that are independent of the node input order. The conditional modulation unit modulates the cross-node statistical features according to the number of measurement nodes, the normalized node flow length sequence, and the node load estimate sequence. The aggregation residual decoder receives the aggregation Sketch local features, the aggregation Sketch structural features, and the modulated cross-node statistical features, and outputs the aggregation additive correction term. The skip gate unit receives the aggregation Sketch local features and the aggregation basic estimate, and outputs a skip gate value used to control the effectiveness of the aggregation additive correction term.

[0049] (3.4) Calculate the global frequency estimate of the element to be queried according to the formula “Global frequency estimate = aggregate basic estimate + skip gate value × aggregate additive correction term”.

[0050] Furthermore, the aforementioned local correction module and merging perception correction module require a two-stage training approach. First, they are pre-trained on the synthetic data stream to cover different skewnesses, element counts, and node counts. Then, they are fine-tuned on the real data stream or target business traffic. During both pre-training and fine-tuning, the optimization objective is to minimize the total loss function, adjusting the parameters of the local correction module and the merging perception correction module accordingly. Finally, the trained local correction module and merging perception correction module are used for feature extraction and forward computation, executing steps (2) and (3), without updating the model parameters during the inference stage.

[0051] Furthermore, the total loss function includes local correction loss, aggregate correction loss, and skip gate regularization term. The local correction loss is used to constrain the local additive correction term to fit the local basic estimate residual; the aggregate correction loss is used to constrain the aggregate additive correction term to fit the aggregate basic estimate residual; and the skip gate regularization term is used to suppress unnecessary secondary corrections when the error of the aggregate basic estimate is small.

[0052] Example 3: Local Sketch Maintenance Suppose that the global data stream is observed jointly by K measurement nodes. The k-th measurement node processes the local data stream. Its local stream length is ,in This represents the local data stream received by the k-th measurement node. This represents the local flow length of the k-th measurement node. (Element to be queried) The true frequency at the k-th measurement node is denoted as Then the global true frequency is: In the formula, Indicates the element to be queried The true frequency at the k-th measurement node, where K represents the total number of measurement nodes. Indicates the element to be queried The global true frequency.

[0053] Node k maintains a basic Sketch. When a data stream element arrives, for the element to be queried... The counter lookup value corresponding to the hash of row j can be represented as: In the formula, This represents the non-negative error introduced by hash collisions in the j-th hash row of the k-th measurement node. This represents the element to be queried in the k-th measurement node. Query the counter value corresponding to the j-th hash row.

[0054] When the base Sketch is Count-Min Sketch, the local base estimate can be the minimum of all hash row counter query values: In the formula, This represents the element to be queried in the k-th measurement node. The local baseline estimate, This means taking the minimum value among all corresponding values ​​in all hash rows.

[0055] In traditional distributed aggregation, the aggregation point directly sums the query values ​​of the counters of each measurement node to obtain the aggregation sketch feature: In the formula, This represents the aggregated Sketch feature obtained after merging aggregation points. As can be seen from the above formula, the aggregation error is the sum of the hash collision errors of multiple nodes, and its distribution and magnitude differ from the single-node error. Therefore, to address this issue, local correction and post-aggregation correction are introduced.

[0056] Example 4: Local calibration module like Figure 2 As shown, the local correction module includes a global encoder, a local encoder, a load decoder, and a correction decoder.

[0057] The global encoder extracts a portion of the counters from the counter matrix of the base Sketch, such as the first few counters of each row, to obtain node-level structural features. These node-level structural features reflect the load, collision level, and counter distribution of the base Sketch at the current measurement node.

[0058] The load decoder estimates the node load based on the node-level structural features output by the global encoder, obtaining a load estimate, and further estimates the number of different elements in the node, where an element refers to a data stream. This load estimate is used to normalize the collision intensity features.

[0059] The local encoder receives the local counter features of the element to be queried and obtains a local counter feature vector through its encoding process. This local counter feature vector includes the following features: ① d counter query values. ② Local baseline estimate ③ Ratio characteristics , used to describe Compared to ④ Collision intensity characteristics , used to describe offset The intensity relative to the average load of the node; ⑤ Statistical summary characteristics of the counter query values, such as mean, variance, maximum, and median; ⑥ Node local flow length. And their normalized forms. In one implementation, the ratio characteristic and the collision intensity characteristic can be calculated using the following formula: In the formula, To prevent division by zero constants, This is an estimate of the number of distinct elements in a node.

[0060] The correction decoder fuses node-level structural features and local counter feature vectors to output a local additive correction term. .

[0061] Further residual correction is performed on the local baseline estimate of each measurement node before aggregation. The corrected local frequency estimate is as follows: In the formula, This represents the local frequency estimate after correction at the k-th measurement node. This represents the local additive correction term output by the local correction module of the k-th measurement node. This invention uses additive correction instead of multiplicative correction because the basic Sketch error is mainly manifested as hash collision residuals, and additive correction can directly fit the residual term; while multiplicative correction binds the error scale to the basic estimate, which can easily produce unstable estimates when low-frequency elements are submerged in noise.

[0062] The node-level structural features, load estimates, and normalized local flow lengths are used to generate compact node feature vectors via linear mapping or a multilayer perceptron. , to be used to construct cross-node statistical features at the aggregation point, where This represents the compact node feature vector of the k-th measurement node.

[0063] Example 5: Combined Sensing Correction Module like Figure 3 As shown, the merged sensing correction module includes a cross-node statistical pooling unit, a conditional modulation unit, an aggregated residual decoder, and a skip gating unit.

[0064] The aggregation point is responsible for receiving the basic Sketch counter query values ​​or corrected local frequency estimates and compact node feature vectors uploaded by each measurement node. It then performs counter-level item-by-item merging on the Sketch counter query values ​​or corrected local frequency estimates to obtain the aggregated Sketch features. For example, for Count-Min Sketch, its aggregated Sketch features are: In the formula, Let represent the aggregated Sketch feature of the j-th hash row. Then, based on the aggregated Sketch feature, extract the aggregated Sketch local features, aggregated Sketch structural features, and aggregated basic estimate. The aggregated Sketch feature with the smallest value among all hash rows is taken as the aggregated basic estimate, which can be expressed as: In the formula, This represents the aggregate basis estimate obtained from the aggregation point.

[0065] The cross-node statistical pooling unit processes the compact node feature vectors uploaded by each measurement node. Permutation-invariant aggregation is performed. Since the input order of the measurement nodes should not affect the final estimation result, the mean, variance, and maximum value can be concatenated to form cross-node statistical features: In the formula, Represents cross-node statistical characteristics. This indicates a splicing operation. , , Let each represent a compact node feature vector for all K measurement nodes. The mean, variance, and maximum value are calculated element by element. This aggregation method ensures the model's robustness to the input order of measurement nodes.

[0066] The conditional modulation unit determines the measurement node length based on the number of measurement nodes K and the normalized node flow length sequence (i.e., the normalized local flow length of each measurement node). The sorted sequence of data and the sequence of node load estimates (i.e., the sorted sequence of load estimates for each measurement node) are used to generate modulation parameters to modulate (i.e., scale and translate) the cross-node statistical features; the modulation parameters include scaling factors. Translation factor In one embodiment, the sorting sequence is processed by a conditional modulation unit to generate a scaling factor. Translation factor Then, statistical features across nodes. By performing scaling and translation transformations, the modulated cross-node statistical features are obtained: In the formula, This represents the cross-node statistical features after scaling and translation transformations. This indicates element-wise multiplication. This design enables the merged sensing correction module to adapt to different numbers of nodes, different traffic partitions, and different load distributions.

[0067] The aggregation residual decoder receives aggregated Sketch local features, aggregated Sketch structural features, and conditionally modulated cross-node statistical features, and outputs an aggregation additive correction term. .

[0068] The skip-gating unit shares some low-level features with the aggregated residual decoder, and outputs the skip-gating value through the Sigmoid activation function. , The value range is (0,1). When the aggregate basic estimate is... When the residual is large Approaching 1 to allow for stronger correction; when the aggregate basic estimate When it is close to the true frequency, Approaching 0 to suppress overcorrection. The final global frequency estimate is then: In the formula, This is the final global frequency estimate.

[0069] Example 6: Training Method like Figure 4 As shown, a two-stage training method is adopted.

[0070] Phase 1: Pre-training Phase. The training module generates or collects synthetic data streams covering different distribution skewnesses (e.g., Zipf distribution parameters from 0.5 to 2.5), different numbers of elements (e.g., 1 to 10 million), different stream lengths (e.g., 100,000 to 100 million), and different numbers of nodes (e.g., K from 2 to 100), constructing various distributed Sketch aggregation scenarios. For each synthetic scenario, the local true frequency and global true frequency of each measurement node are recorded as labels. The local correction module and the merging-aware correction module learn general hash collision error patterns and cross-node aggregation error patterns during this phase. The preferred number of pre-training rounds is 100-500, and the preferred batch size is 64-512.

[0071] In this stage, the local correction module and the merged perception correction module go through steps (2)-(3) of Example 2. First, the synthetic data stream is processed on each distributed node using a standard Sketch data structure (such as Count-MinSketch) to construct a local Sketch and record the local stream length. Then, the system extracts raw features (such as hash row counter values, basic estimates, etc.) from the local Sketch of each node for the selected query element, and combines them with node information (such as normalized stream length) to form a local feature vector; at the same time, a global encoder analyzes the entire local Sketch matrix to extract global structural features. Furthermore, the sketches of all nodes are aggregated (element-by-element addition) to generate aggregated sketch local features and aggregated sketch structural features for the query element. These are combined with conditionally modulated cross-node statistical features. Subsequently, the conditional modulation unit modulates the cross-node statistical features based on the number of measured nodes, the normalized node flow length sequence, and the node load estimate sequence. The aggregated residual decoder receives the aggregated sketch local features, aggregated sketch structural features, and modulated cross-node statistical features, and outputs an aggregated additive correction term. The skip gate unit receives the aggregated sketch local features and the aggregated basic estimate, and outputs a skip gate value to control the effectiveness of the aggregated additive correction term. Finally, the global frequency estimate of the query element is calculated according to the formula "global frequency estimate = aggregated basic estimate + skip gate value × aggregated additive correction term".

[0072] During the pre-training phase, the parameters of the local correction module and the merged perception correction module are adjusted with the goal of minimizing the total loss function until the preset training rounds are reached, resulting in pre-trained local correction module and merged perception correction module.

[0073] Phase Two: Fine-tuning Phase. The training module uses real traffic or query load from the target business scenario to fine-tune the parameters of the pre-trained local calibration module and the merged perception calibration module to adapt to the actual data distribution. The optimal number of fine-tuning rounds is 10-50, with the learning rate set to 0.1 times that of the pre-training phase.

[0074] In this stage, real traffic or real query load from the target business scenario is used. The local correction module and the merged perception correction module go through steps (2)-(3) of Example 2. Unlike the first stage, the real frequency of elements cannot be obtained as supervision labels in this stage. Therefore, the processing method is: first, use these real traffic to build Sketch on each node and record the local flow length; then, for the elements in the real query load, the input features (local features, global structural features, aggregate features, etc.) are extracted and input into the two correction modules. However, due to the lack of ground truth, the system cannot calculate the local correction loss and the aggregate correction loss. At this time, the training depends entirely on skipping the gate regularization term. This regularization term does not depend on the truth value; it only provides optimization signals by encouraging the gating mechanism to output sparse "skip" decisions, thereby guiding the model to adapt to the data distribution of the target scenario.

[0075] During the fine-tuning phase, the parameters of the pre-trained local correction module and the merged perception correction module are adjusted with the goal of minimizing the total loss function during training, until the preset number of training rounds is reached, resulting in the final trained local correction module and merged perception correction module.

[0076] It should be understood that the input data types and formats of the local correction module and the merged perceptual correction module are essentially the same in both stages, and both follow the same feature engineering process. The first stage description focuses on the construction process of the offline training dataset (from raw data stream to feature vectors), while the second stage description focuses on the data processing process in online application scenarios (from real data stream to feature vectors).

[0077] The input to the local correction module is always the global structural features, local feature vectors, and local baseline estimates, and the output is a local additive correction term and a local skip gate signal; the input to the merged perception correction module is always the aggregated Sketch local features, aggregated Sketch structural features, and conditionally modulated cross-node statistical features, and the output is an aggregated additive correction term and an aggregated skip gate signal.

[0078] In this embodiment, the total loss function includes local correction loss, aggregate correction loss, and skip gate regularization term.

[0079] The local correction loss is used to constrain the local correction result to be close to the local true frequency, i.e., to constrain the local additive correction term to fit the local baseline estimate residual. For measurement node k, the local correction loss can be calculated using weighted absolute error: In the formula, Indicates local correction loss. Indicates the element to be queried The true frequency at the k-th measurement node, Indicates the element to be queried The global frequency estimate at the k-th measurement node, This indicates calculating the average value. To train the weights, the weights Desirable This is used to increase the training weights of frequent elements, preventing the model from focusing too much on a large number of low-frequency elements and ignoring high-frequency elements that have a greater impact on estimation accuracy.

[0080] The aggregation correction loss is used to constrain the aggregation point output to approximate the global true frequency; that is, it is used to constrain the aggregation additive correction term to fit the residual of the aggregation fundamental estimate. The formula for calculating the aggregation correction loss is: In the formula, Indicates the aggregation correction loss. This represents the final global frequency estimate. Indicates the element to be queried The global true frequency.

[0081] Skip-gating regularization terms are used to suppress unnecessary quadratic corrections, specifically to suppress unnecessary quadratic corrections when the error in the aggregated basis estimate is small. For example: In the formula, This indicates that the gated regular expression should be skipped. Indicates the element to be queried The corresponding skip gate value, Indicates the element to be queried The corresponding additive correction term for polymerization, Indicates the element to be queried The corresponding aggregate fundamental estimate. The meaning of this skip-gating regularization term is: when the aggregate fundamental estimate is already close to the true frequency (i.e., the denominator is large), skipping the gate value is encouraged. The skip gate regularization term tends to 0 to reduce unnecessary corrections; when the aggregated basic estimate error is large, the effect of the skip gate regularization term is small, allowing the model to be fully corrected.

[0082] The total loss function is a weighted sum of the local correction loss, the aggregate correction loss, and the skip-gating regularization term, expressed as: In the formula, Represents the total loss function. , , They represent , , The weighting coefficients.

[0083] After the two-stage training is completed, the parameters of the local correction module and the merged perceptual correction module are fixed during the inference stage. Only Sketch update, feature extraction and forward inference are performed, and the model parameters are no longer updated.

[0084] Example 7: Query Process When the query interface receives the element to be queried When this happens, the following process will be executed: Step 1: Each measurement node determines the element to be queried. Read the d counters corresponding to the local basic Sketch And calculate the local baseline estimate. .

[0085] Step 2: Each measurement node extracts its local feature vector (i.e., local counter features), including d counter query values, local baseline estimates, ratio features, collision intensity features, and statistical summary features. The forward computation of the local correction module is then performed to obtain the local additive correction term. and compact node feature vectors The locally corrected frequency estimate can be optionally used for local node queries, but its primary purpose is to pass the correction information to the aggregation point.

[0086] Step 3: The aggregation point receives the basic Sketch counters (or calibrated local frequency estimates) and compact node feature vectors uploaded by each measurement node, and performs counter-level aggregation at the aggregation point to obtain the aggregated Sketch features. And calculate the aggregate basic estimate. .

[0087] Step 4: Construct cross-node statistical features from aggregation points The conditional modulation unit modulates the data based on the number of measured nodes, the normalized node flow length sequence, and the node load estimate sequence to obtain the desired result. Subsequently, the forward computation of the merged sensing correction module is performed to obtain the aggregate additive correction term. and skip gate value .

[0088] Step 5: Aggregate point outputs global frequency estimate. And return the results to the user through the query interface.

[0089] In this query process, the basic Sketch update is still completed locally on each measurement node, and the update path does not require neural network inference, thus preserving the low-overhead update characteristics of the basic Sketch. Neural network inference mainly occurs during the query phase, making it suitable for network measurement and stream processing scenarios where the query frequency is lower than the update frequency, such as backbone network traffic measurement and data center network monitoring.

[0090] Example 8: Two-Level Learning Enhancement Sketch Aggregation System The system includes multiple measurement nodes, an aggregation point, a training module, and a query interface. The measurement nodes are used to receive local data streams, maintain a basic Sketch counter, execute a local correction module, and upload corrected local frequency estimates and compact node feature vectors. The aggregation point receives basic Sketch counters, corrected local frequency estimates, and compact node feature vectors from multiple measurement nodes, performs counter-level aggregation on the Sketch counters or local frequency estimates, and executes a merged sensing correction module to output a global frequency estimate. The training module trains the local correction module and the merged sensing correction module based on synthetic and real data streams. The query interface receives the identifier of the element to be queried and returns the global frequency estimate of the element.

[0091] In this embodiment, the local correction module includes a global encoder, a local encoder, a load decoder, and a correction decoder. The global encoder is used to extract a portion of the counters from the counter matrix of the basic Sketch to obtain node-level structural features. The local encoder is used to process the counter query value of the element to be queried, the local basic estimate value obtained from the counter query value, the ratio feature, the collision intensity feature, and the statistical summary feature. The load decoder is used to estimate the node load based on the node-level structural features to obtain the load estimate value. The correction decoder is used to fuse the node-level structural features and the output of the local encoder to generate a local additive correction term.

[0092] In this embodiment, the merged sensing correction module includes a cross-node statistical pooling unit, a conditional modulation unit, an aggregated residual decoder, and a skip gate unit. The cross-node statistical pooling unit calculates the mean, variance, and maximum value of the compact node feature vectors uploaded by each measurement node, and concatenates them to obtain the cross-node statistical features. The conditional modulation unit modulates the cross-node statistical features based on the number of measurement nodes, the normalized node flow length sequence, and the node load estimate sequence. The aggregated residual decoder generates an aggregated additive correction term based on the aggregated Sketch local features, the aggregated Sketch structural features, and the modulated cross-node statistical features. The skip gate unit generates a skip gate value that controls the effectiveness of the aggregated additive correction term.

[0093] Example 9: Experimental Verification and Results To verify the effectiveness of this invention, a real network traffic dataset (e.g., the CAIDA network traffic dataset) was used for testing. The experiment was set up with K=10 measurement nodes, each receiving approximately 1 / 10 of the traffic fragment. The basic sketch used was Count-MinSketch, d=5, W=2048. Both the local correction module and the merging-perception correction module employed a 3-layer fully connected network with hidden layer dimensions of 128, 64, and 32, respectively.

[0094] Experimental results show that compared to direct aggregation Count-MinSketch (ARE, Average Relative Error is approximately 0.35), this invention reduces ARE to approximately 0.08, an improvement of approximately 77%. Compared to single-layer learning-enhanced Sketch (e.g., LearnedSketch direct aggregation), this invention reduces ARE by approximately 35%, with only a 5% increase in communication overhead (due to the compact node feature vector dimension being only 64 dimensions). The skip-gating mechanism makes the gating value less than 0.3 in approximately 60% of queries, indicating that the aggregation basis estimation is already quite accurate in most cases, and the skip-gating effectively suppresses the additional bias caused by overcorrection.

[0095] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A two-level learning-enhanced Sketch aggregation method for distributed flow frequency estimation, characterized in that, Includes the following steps: (1) Receive local data streams at K measurement nodes respectively, and maintain a basic Sketch that supports counter-level merging at each measurement node; (2) At each measurement node, local counter features, load features and structural features are extracted from the basic Sketch for the element to be queried, and local additive correction terms and compact node feature vectors are output through the local correction module; the local basic estimate of each measurement node is residually corrected according to the local additive correction terms so that each measurement node forms a corrected local frequency estimate. (3) Send the corrected local frequency estimate and compact node feature vector of each measurement node to the aggregation point. At the aggregation point, perform counter-level aggregation on the Sketch counter or local frequency estimate of each measurement node to obtain the aggregated Sketch feature and aggregated basic estimate. At the aggregation point, perform permutation-invariant statistical aggregation on the compact node feature vector through the merging sensing correction module. Combine the number of measurement nodes, the normalized local flow length and load estimate of each measurement node and the aggregated Sketch feature to generate the aggregated additive correction term and skip gate value. Based on the aggregated basic estimate, the aggregated additive correction term, and the skip gate value, the global frequency estimate of the element to be queried is obtained.

2. The two-level learning-enhanced Sketch aggregation method for distributed flow frequency estimation according to claim 1, characterized in that, Step (1) specifically includes: Each measurement node k receives a local data stream. Count the length of the local stream And map the elements in the local data stream to the corresponding counters according to the hash function of the basic Sketch; Each measurement node maintains a basic Sketch with the same structure, which includes a d-row, w-column counter matrix and d hash functions; When the base Sketch is Count-Min Sketch, each measurement node uses the same number of hash functions, the same counter matrix width, and the same hash random seed to support the aggregation point to merge counters at the same location item by item; For the element to be queried, the measurement node obtains the counter query values ​​corresponding to multiple hash rows from the base Sketch, and obtains the local base estimate value from the counter query values.

3. The two-level learning-enhanced Sketch aggregation method for distributed flow frequency estimation according to claim 1, characterized in that, Step (2) specifically includes the following sub-steps: (2.1) Extract the counter query value of the element to be queried in d hash rows, the local basic estimate obtained from the counter query value, the ratio feature, the collision intensity feature and the statistical summary feature from the basic Sketch as local counter features; (2.2) The local correction module includes a global encoder, a local encoder, a load decoder, and a correction decoder; the global encoder extracts a portion of the counters from the counter matrix of the basic Sketch to obtain node-level structural features; The load estimate is obtained by estimating the node load based on the node-level structural features using a load decoder; the local counter features of the query element are processed by a local encoder to obtain a local counter feature vector; and the node-level structural features and the local counter feature vector are fused by a correction decoder to output a local additive correction term. (2.3) Using the additive residual correction form, the corrected local frequency estimate is expressed as the sum of the local basic estimate and the local additive correction term; (2.4) Generate compact node feature vectors based on node-level structural features, load estimates and normalized local flow lengths.

4. The two-level learning-enhanced Sketch aggregation method for distributed flow frequency estimation according to claim 3, characterized in that, The ratio feature is used to characterize the offset of each row counter query value relative to the base estimate, the collision intensity feature is used to characterize the intensity of the offset relative to the node average load, and the statistical summary features include mean, variance, maximum value, and median.

5. The two-level learning-enhanced Sketch aggregation method for distributed flow frequency estimation according to claim 1, characterized in that, Step (3) specifically includes the following sub-steps: (3.1) The aggregation point receives the basic Sketch counter, the corrected local frequency estimate and the compact node feature vector uploaded by each measurement node. At the aggregation point, the Sketch counter or local frequency estimate of each measurement node is aggregated at the counter level to obtain the aggregated Sketch feature. (3.2) The aggregation point extracts the local features, structural features, and basic estimates of the aggregation Sketch based on the aggregation Sketch features; (3.3) The merged sensing correction module includes a cross-node statistical pooling unit, a conditional modulation unit, an aggregated residual decoder, and a skip gate unit; The mean, variance and maximum value of the compact node feature vectors uploaded by each measurement node are calculated by the cross-node statistical pooling unit, and then concatenated to obtain the cross-node statistical features. The conditional modulation unit modulates the cross-node statistical features based on the number of measured nodes, the normalized node flow length sequence, and the node load estimate sequence; the aggregation residual decoder receives the aggregation Sketch local features, the aggregation Sketch structural features, and the modulated cross-node statistical features, and outputs the aggregation additive correction term; the skip gate unit receives the aggregation Sketch local features and the aggregation basic estimate, and outputs the skip gate value used to control the effectiveness of the aggregation additive correction term. (3.4) Calculate the global frequency estimate of the element to be queried according to the formula "global frequency estimate = aggregate basic estimate + skip gate value × aggregate additive correction term".

6. The two-level learning-enhanced Sketch aggregation method for distributed flow frequency estimation according to claim 1, characterized in that, The local correction module and the merged perception correction module adopt a two-stage training method. First, they are pre-trained on the synthetic data stream to cover different skewness, number of elements, and number of nodes. Then, they are fine-tuned on the real data stream or target business traffic. During both the pre-training and fine-tuning processes, the optimization objective is to minimize the total loss function and adjust the parameters of the local correction module and the merged perception correction module.

7. The two-level learning-enhanced Sketch aggregation method for distributed flow frequency estimation according to claim 6, characterized in that, The total loss function includes local correction loss, aggregate correction loss, and skip gate regularization term. The local correction loss is used to constrain the local additive correction term to fit the local basic estimate residual; the aggregate correction loss is used to constrain the aggregate additive correction term to fit the aggregate basic estimate residual; and the skip gate regularization term is used to suppress unnecessary secondary corrections when the error of the aggregate basic estimate is small.

8. A system for implementing the two-level learning-enhanced Sketch aggregation method for distributed flow frequency estimation as described in any one of claims 1-7, characterized in that, include: Multiple measurement nodes are used to receive local data streams, maintain basic Sketch, execute local calibration modules, and upload calibrated local frequency estimates and compact node feature vectors, respectively. The aggregation point is used to receive the basic Sketch counters, corrected local frequency estimates, and compact node feature vectors from multiple measurement nodes, perform counter-level aggregation on the Sketch counters or local frequency estimates, and execute the merge sensing correction module to output a global frequency estimate. The training module is used to train the local correction module and the merging-aware correction module based on synthetic and real data streams; The query interface is used to receive the identifier of the element to be queried and return the global frequency estimate of the element to be queried.

9. The system according to claim 8, characterized in that, The local calibration module includes: The global encoder is used to extract a portion of the counters from the counter matrix of the base Sketch to obtain node-level structural features; The local encoder is used to process the counter query value of the element to be queried, the local base estimate obtained from the counter query value, the ratio feature, the collision intensity feature, and the statistical summary feature; The load decoder is used to estimate the node load based on the node-level structural characteristics and obtain the load estimate. The correction decoder is used to fuse node-level structural features and the output of the local encoder to generate a local additive correction term.

10. The system according to claim 8, characterized in that, The merging sensing correction module includes: The cross-node statistical pooling unit is used to calculate the mean, variance, and maximum value of the compact node feature vectors uploaded by each measurement node, and then concatenate them to obtain the cross-node statistical features. The conditional modulation unit is used to modulate cross-node statistical characteristics based on the number of measured nodes, the normalized node flow length sequence, and the node load estimate sequence; The aggregated residual decoder is used to generate an aggregated additive correction term based on the aggregated Sketch local features, aggregated Sketch structural features, and modulated cross-node statistical features. Skip gating unit, used to generate skip gating values ​​that control the extent to which the aggregate additive correction term is effective.

Images