Data stream processing method and system supporting set-increment hybrid updating

Abstract

The invention belongs to the technical field of information, and relates to a data stream processing method and system supporting set-increment hybrid updating. The method comprises the following steps: receiving an SIM updating operation, and judging an operation type; performing double hash correspondence, and mapping the keys of the new item into two balance buckets through two independent hash functions; carrying out SIM updating by using the two balance buckets correspondingly obtained by the double Hash; and if the SIM updating fails, a cascade overflow mechanism is started, and the SIM updating is carried out through the cooperation of the multi-stage balance buckets. According to the method, SIM data stream processing is completely supported, high-precision and low-time processing backup are both considered, optimal combination can be selected by itself, and global error accumulation is reduced. The method is suitable for actual SIM data stream processing scenes such as network monitoring, sensor data analysis, real-time network tracking, cloud computing memory monitoring and the like, and high precision and resource limitation requirements are considered.

Description

Technical Field

[0001] This invention belongs to the field of information technology, specifically relating to a data stream processing method and system that supports set-increment hybrid updates. Background Technology

[0002] Sketch is an approximation algorithm that processes large data streams and provides them with a compact data structure to approximate certain characteristics of the original data. The goal of Sketch algorithms is to process large-scale data streams with limited memory and computational resources and to quickly and approximately answer queries about the data. In the field of big data stream processing, Sketch algorithms are widely used in databases, data mining, network monitoring, and web services due to their small memory usage, tunability, and efficient visualization. Traditional Sketch algorithms (such as Count-Min Sketch) primarily support incremental updates but struggle to effectively handle set updates (i.e., full rewrite operations).

[0003] In scenarios such as sensor data collection and real-time memory monitoring, data streams typically contain both incremental updates and set updates, known as SIM (Set-Increment Mixed) data streams. However, current Sketch support for SIM data streams is limited: it cannot effectively handle full rewrites, leading to operator-based back products; in high-concurrency environments, maintaining independent counters incurs significant storage and computation costs; and errors accumulate, affecting query accuracy.

[0004] Existing technical solutions:

[0005] Counting Sketch: Maps keys to multiple counters using multiple hash functions and updates each counter incrementally. Suitable for scenarios that only support incremental operations; cannot handle collection updates.

[0006] Key-Value Sketch: Uses a small heap / key-value store, such as SpaceSaving and Elastic Sketch. It is suitable for small incremental updates, but it is difficult to accurately handle updates to a large collection of values.

[0007] Cuckoo hash table: Works using double hashing and can handle SIM data streams, but its performance drops sharply under high load.

[0008] Disadvantages of existing technology:

[0009] Counting Sketch cannot perform collection updates due to type incompatibility; Key-Value Sketch suffers from severe errors when handling SET updates and cannot guarantee accuracy; Cuckoo tables experience multiple commit failures and memory expansion when near high load. Summary of the Invention

[0010] This invention solves the problem that traditional Sketch cannot support collection updates, can maintain high-precision operators in the SIM data stream, and can maintain low-time processing within limited storage.

[0011] The technical solution adopted in this invention is as follows:

[0012] A data stream processing method supporting set-increment hybrid updates includes the following steps:

[0013] Receive SIM update requests and determine the request type;

[0014] Perform double hash mapping, using two independent hash functions to map the key of the new item to two balanced buckets;

[0015] SIM updates are performed using the two balanced buckets obtained from the double hashing.

[0016] If the SIM update fails, a cascading overflow mechanism is activated, which uses the cooperation of multiple level balance buckets to update the SIM.

[0017] Furthermore, the receiving of SIM update operations and determination of operation type include:

[0018] enter<e,op,v> , where e is the key, op is the operator := or +=; v is the value to be updated;

[0019] Determine the operation type, which includes the following two:

[0020] Collection Update:<e,:=,v> This means replacing the value of key e with v;

[0021] Incremental update:<e,+=,v> This indicates that the value of key e is incremented by v.

[0022] Furthermore, the double hash correspondence includes:

[0023] Enter the key 'e' to enter the new item;

[0024] Use the hash function h1(e) to obtain the number i1 of the first balanced bucket;

[0025] Use the hash function h2(e) to obtain the number i2 of the second balanced bucket, ensuring that i1≠i2;

[0026] Two balanced buckets, B[i1] and B[i2], are obtained and used as two target buckets.

[0027] Furthermore, the size of the balanced bucket is d, meaning it contains d items, each of which is a key-value pair.<ei,v> The items are sorted in descending order of the absolute value of vi; when a new item is added...<e,v> Upon arrival, the balancing bucket follows the following operational path:

[0028] 1) When the absolute value of the new term v, |v|, is less than or equal to the second smallest value v d-1 At that time, the new item<e,v> With minterm <e d ,v d Execute the merge, with the following merge rules:

[0029] With |v| / (|v d The probability of |+|v| is retained as e, and the estimated value is set as sgn(v)(|v) d |+|v|);

[0030] With |v d | / (|v d The probability of |+|v| is preserved in e. d And set the valuation as sgn(v d )(|v d |+|v|);

[0031] The merge result is written to the last item in the balanced bucket;

[0032] 2) When the absolute value of the new term v is greater than v d-1 At that time, first of all <e d-1 ,v d-1 >and <e d ,v d The items will be merged into one item, following the rules:

[0033] With |v d-1 | / (|v d-1 |+|v d The probability of |) is preserved in e. d-1 And set the valuation as sgn(v d-1 )(|v d-1 |+|v d |);

[0034] With |v d | / (|v d-1 |+|v d The probability of |) is preserved in e. d And set the valuation as sgn(v d )(|v d-1 |+|v d |);

[0035] The merged result is stored in the second-to-last position of the balanced bucket, and then a new item is written to the last position of the balanced bucket.<e,v> .

[0036] Furthermore, the step of using the two balanced buckets obtained from the double hash to perform SIM updates includes:

[0037] Check if the new item 'e' exists in B[i1] and B[i2]; if it exists, the set update operation is to directly overwrite; the incremental update operation is to accumulate the current value.

[0038] If e does not exist in B[i1] or B[i2], but there is an empty space in B[i1] or B[i2], then e is directly inserted into the empty space, and the value is v;

[0039] If there are no empty slots, then perform a merge operation: if |v|≤|v d-1 If |v|, then merge the new term with the minterm in B[i1] or B[i2]; otherwise, first merge the minterm and the second smallest term in B[i1] or B[i2], then insert e; where |v| d-1 | represents the value of the (d-1)th item in the balanced bucket, i.e., the second smallest value.

[0040] Furthermore, the cascading overflow mechanism searches for the minimum cost path and performs merging at the optimal position. The cascading overflow mechanism includes the following steps:

[0041] Search phase: Calculate the merge cost and perform a bucket search along the target path based on the merge cost;

[0042] Stopping strategy: Stop if a better merging position is found or the threshold is reached;

[0043] The "kick bucket" phase involves removing the smallest item from the recursive search process and moving on to the next bucket until insertion is complete.

[0044] Further, the query operation is performed using the following steps:

[0045] Input the query key e, and use two hash functions h1(e) and h2(e) to map it to two balanced buckets B[i1] and B[i2];

[0046] Scan each item sequentially within B[i1] and B[i2] to check if the item exists.<e,v> ;

[0047] If it exists, return its corresponding value v; if neither of the two balanced buckets is hit, return the default value of 0.

[0048] A data stream processing system supporting set-increment hybrid updates, comprising:

[0049] The SIM update operation receiving module is used to receive SIM update operations and determine the operation type.

[0050] The double hash mapping module is used to perform double hash mapping, which maps the key of a new item to two balanced buckets through two independent hash functions;

[0051] The trial update module is used to perform SIM updates using the two balanced buckets obtained from the double hash.

[0052] The cascading overflow module is used to initiate a cascading overflow mechanism if the SIM update fails, and to perform the SIM update through the cooperation of multiple level balance buckets.

[0053] Key technical points and beneficial effects of this invention:

[0054] This invention fully supports SIM data stream processing; it balances high precision with low-time processing backup; it can automatically select the optimal merge, reducing global error accumulation. Compared with existing technologies, this invention reduces MSE by 1-2 orders of magnitude; both edit / query vectors can reach over 20 Mops; and it speeds up time processing without compromising accuracy when memory is reduced, combining timeliness and flexibility.

[0055] This invention is applicable to practical SIM data stream processing scenarios such as network monitoring, sensor data analysis, real-time network tracking, and cloud computing memory monitoring, taking into account both high precision and resource constraints. Attached Figure Description

[0056] Figure 1 This is a flowchart of the steps of a data stream processing method that supports set-incremental hybrid updates according to the present invention.

[0057] Figure 2 This is a schematic diagram of the structure of the balance bucket of the present invention.

[0058] Figure 3 This is an example diagram of the cascading overflow mechanism of the present invention.

[0059] Figure 4 This is a schematic diagram of the constituent modules of a data stream processing system that supports set-increment hybrid updates according to the present invention. Detailed Implementation

[0060] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0061] The main inventive points of this invention are as follows:

[0062] 1) PPS sampling was promoted to achieve unbiased merging across the entire real number field.

[0063] This invention achieves unbiased merging across the entire real number field by introducing an extended merging operation (MERGE± operation). This operation supports unbiased merging of arbitrary real numbers, and its processing logic is as follows:

[0064] For two terms<e1,v1> and<e2,v2> Let e1 and e2 be keys, and v1 and v2 be values. With probability |v1| / (|v1|+|v2|), e1 is retained and its value is updated to sgn(v1)(|v1|+|v2|); with probability |v2| / (|v1|+|v2|), e2 is retained and its value is updated to sgn(v2)(|v1|+|v2|). Here, sgn represents the sign function, returning the sign of the value: +1 for positive numbers, -1 for negative numbers, and 0 for zero.

[0065] The method can be theoretically proven to be an unbiased estimate and has the minimum merging variance. It is repeatedly invoked in the merging strategy of BalanceBucket in this invention, thereby achieving an accurate approximation of the updated values ​​of any set.

[0066] 2) Design a Balance Bucket structure and select the optimal merging strategy.

[0067] The self-selected optimal merging strategy refers to the balancing of buckets based on the value of the new item |v| and the value of the current minimum item in the bucket |v| when facing set updates or incremental updates. d |、Value of the second smallest term|v d-1 The comparison between | will automatically select one of the following two optimal strategies:

[0068] If |v|≤|v d-1 |: Directly merge the new item with the minimum item in the balanced bucket;

[0069] If |v|>|v d-1 |: First, merge the smallest and second smallest items in the balanced bucket, then insert the new item.

[0070] The merge cost is calculated for each merge, and the operation path with the minimum merge cost is selected, thereby achieving local optimum error control.

[0071] 3) Introduce the Cascading Overflow mechanism to achieve global optimal balance across buckets.

[0072] The present invention provides a data stream processing method supporting set-increment hybrid updates, the process of which is as follows: Figure 1 As shown, it includes the following steps:

[0073] S101: Receive SIM update operation and determine the operation type.

[0074] SIM update operation refers to the ability to update received data items using one of the following two methods:

[0075] Set update: in the form of<e,:=,v> This means replacing the value of key e with v;

[0076] Incremental update: (e.g., ...)<e,+=,v> This indicates that the value of key e is incremented by v.

[0077] The inputs, processing methods, and outputs for this step are as follows:

[0078] enter:<e,op,v> , where e is the key (e.g., sensor ID, network stream tag, etc.); op is the operator (:= or +=); v is the value to be updated.

[0079] Processing method: Determine the operation type, that is, determine whether it is a set update or an incremental update based on the op, and send it to the corresponding update process.

[0080] Output: Mark the operation type and proceed to the next step of hash mapping.

[0081] S102: Perform double hash mapping, using two independent hash functions to map the key of the new item to two balanced buckets.

[0082] Double hash mapping refers to using two independent hash functions to map the key 'e' of a new item to two different balanced buckets, ensuring that update operations have two insertion / query opportunities and reducing hash collisions. This step initially calls the balanced bucket as the storage target for the bucket structure.

[0083] The specific steps for performing double hashing in this invention are as follows:

[0084] a) Enter the key 'e' for the new item;

[0085] b) Use the hash function h1(e) to obtain the number i1 of the first balanced bucket;

[0086] c) Use the hash function h2(e) to obtain the number i2 of the second balanced bucket, ensuring that i1≠i2; thus obtaining two balanced buckets B[i1] and B[i2], which serve as two target buckets;

[0087] d) Read all items in the balance buckets B[i1] and B[i2] to prepare for the next stage of matching and processing operations (i.e., step 3).

[0088] The Balance Bucket of this invention automatically merges or writes data based on the size of the input values. The Balance Bucket proposed in this invention is an optimized data structure for efficiently handling set updates and incremental updates within a limited space, ensuring unbiased estimation and minimum variance. The structure of the Balance Bucket is as follows: Figure 2 As shown. Figure 2 The left side represents the initial state of a balanced bucket. The size of the balanced bucket is d = 4, meaning it contains 4 items (also called "entries"). Each item is a key-value pair. <e i ,v i The symbol '>' represents the key 'ei' and the estimated value 'vi'. The sorting convention is as follows: items are sorted in descending order of the absolute value of 'vi', meaning the items with the largest absolute values ​​of the estimated values ​​are stored at the beginning, and the items with the smallest absolute values ​​are stored at the end. In practical applications, the size 'd' of the balancing bucket can be set to any positive integer according to system resources and error control requirements, typically between 3 and 8, to achieve a balance between accuracy and computational overhead.

[0089] When a new item<e,v> Upon arrival, the balancing bucket follows the following operational path:

[0090] 1) When the absolute value of the new term v, |v|, is less than or equal to the second smallest value v d-1 At that time, the new item<e,v> With minterm <e d ,v d Execute the merge, with the following merge rules:

[0091] With |v| / (|v d The probability of |+|v| is retained as e, and the estimated value is set as sgn(v)(|v) d |+|v|);

[0092] With |v d | / (|v d The probability of |+|v| is preserved in e. d And set the valuation as sgn(v d )(|v d |+|v|);

[0093] The merge result is written to the last item in the balanced bucket;

[0094] 2) When the absolute value of the new term v is greater than v d-1 At that time, first of all <e d-1 ,v d-1 >and <e d ,v d The items will be merged into one item, following the rules:

[0095] With |v d-1 | / (|v d-1 |+|vd The probability of |) is preserved in e. d-1 And set the valuation as sgn(v d-1 )(|v d-1 |+|v d |);

[0096] With |v d | / (|v d-1 |+|v d The probability of |) is preserved in e. d And set the valuation as sgn(v d )(|v d-1 |+|v d |);

[0097] The merged result is stored in the second-to-last position of the balanced bucket, and then a new item is written to the last position of the balanced bucket.<e,v> .

[0098] For example, the size of the balanced bucket can be d = 4, meaning it contains 4 items. Each item is a key-value pair. <e i ,v i > represents the key ei and the estimated value vi. The sorting convention is: |v1|≥|v2|≥|v3|≥|v4|, that is, the first position stores the item with the larger absolute value of the estimated value, and the last position stores the item with the smallest absolute value of the estimated value. Figure 2 The middle and right sections show when a new update item is added.<e,v> Upon arrival, the balancing bucket follows the following operational path:

[0099] Scenario 1: |v|≤v3

[0100] When the absolute value of the new term v, |v|, is less than or equal to the second smallest value v3, the new term will be...<e,v> With minterm<e4,v4> Perform the merge, with the following merge rules: retain e with a probability of |v| / (|v4|+|v|), and set the valuation to sgn(v)(|v4|+|v|);

[0101] The probability of |v4| / (|v4|+|v|) is retained as e4, and the estimated value is set to sgn(v4)(|v4|+|v|); the new estimated value is ±(|v4+v|), and the sign is determined by the merged item. The merged result is written to the last item in the bucket.

[0102] Scenario 2: |v|>v3

[0103] When the absolute value of the new term v is greater than v3, first...<e3,v3> and<e4,v4> The items are merged into one item according to the following rules: e3 is retained with a probability of |v3| / (|v3|+|v4|), and its estimated value is set to sgn(v3)(|v3|+|v4|); e4 is retained with a probability of |v4| / (|v3|+|v4|), and its estimated value is set to sgn(v4)(|v3|+|v4|); the merged result is stored in the second-to-last position in the balanced bucket (i.e., the position where v3 is replaced), and then the new item is written at the very end of the balanced bucket.<e,v> .

[0104] The above-described merging operation uses unbiased estimation to ensure that the expected value of each item can still be accurately estimated after merging.

[0105] S103: Perform a trial write-back / merge, i.e., use the two balanced buckets obtained from the double hash to perform SIM updates.

[0106] This invention proposes a data stream processing system framework optimized for set-increment mix (SIM) updates, aiming to support low-error, high-throughput valuation updates. The framework consists of multiple balanced buckets, and each key is assigned to two target buckets through a dual hash mapping strategy, thereby performing local updates, valuation merging, and local control within the bucket.

[0107] Trial write / merge refers to performing the following logic in two balanced buckets B[i1] and B[i2] to attempt to complete the update as early as possible without cross-bucket search:

[0108] a) Check if the new item 'e' exists in B[i1] and B[i2]; if it exists, the Set operation will directly overwrite it; the Increment operation will accumulate the current value.

[0109] b) If a match is not found (i.e. e does not exist in B[i1] or B[i2]), but there is an empty space in B[i1] or B[i2], then e is directly inserted into the empty space with a value of v.

[0110] c) If there are no empty slots, perform a merge operation: if |v| ≤ |v d-1 If |v|, then merge the new term with the minterm in B[i1] or B[i2]; otherwise, first merge the minterm and the second smallest term in B[i1] or B[i2], then insert e. Where |v| d-1 | represents the value of the (d-1)th item (the second to last item) in the balanced bucket, i.e., the second smallest value.

[0111] The inputs, processing, and outputs of this step are as follows:

[0112] Input: key e, value v, operation type (Set or Increment), buckets B[i1] and B[i2].

[0113] Processing: Find, Overwrite, Insert, or Merge.

[0114] Output: Update successful or failed (mark whether an overflow is needed).

[0115] This step is the main execution stage of the balanced bucket. Its core structure is used for key-value matching, space determination, and the execution of the MERGE± merging strategy.

[0116] S104: If the SIM update operation in step S103 fails, the CascadingOverflow mechanism is activated to update the SIM through the cooperation of multiple level balance buckets.

[0117] The input to a cascading overflow mechanism: the item that triggers the overflow.<e,v> The target bucket is either B[i1] or B[i2]; the mark update was unsuccessful.

[0118] The output of the cascading overflow mechanism is: find a new bucket B* that minimizes the merge cost, and insert or merge a new item there.

[0119] The cascading overflow mechanism's processing flow involves searching for the minimum-cost path and merging at the optimal position, specifically including:

[0120] a) Search phase: Based on the merging cost, perform a bucket search along the target path;

[0121] b) Stopping strategy: Stop if a better merging position is found or the threshold is reached;

[0122] c) Kicking phase: Kick the min item out of the recursive search process and move on to the next bucket until insertion is complete.

[0123] This step uses the minimum value comparison and merging cost calculation logic of the balanced bucket in each jump and target insertion point selection.

[0124] The cascading overflow mechanism proposed in this invention is an overflow handling mechanism that, through the cooperation of multiple buckets, automatically stops based on the globally minimum merging cost, achieving a globally optimal merging strategy. The implementation steps of cascading overflow are as follows:

[0125] 1) Initialization phase

[0126] If the primary bucket (i.e., the two balanced buckets B[i1] and B[i2] mentioned above) cannot accommodate the new update item (i.e., insertion fails or the merge cost is too high), a cascading overflow is triggered; the current update item is then moved to the next bucket.<e,v> Marked as "candidate", it is ready to be spread to the outer bucket.

[0127] 2) Cascading

[0128] Candidates are searched for the next level bucket (candidate bucket) through the system's built-in extended mapping table (such as alternative hash or path graph);

[0129] Within each candidate bucket, calculate the minimum merge cost for merging with that candidate:

[0130] Cost = 2·∣v∣·∣v target |

[0131] Among them, v target This represents the valuation of the item with the smallest absolute value in the candidate bucket, i.e., the valuation of the item that might be merged at this point. The lower the merger cost, the more suitable the position is for receiving this candidate.

[0132] Record the current minimum cost path and continue recursively to the next lower-level bucket until the set maximum level is reached or the stopping condition is met. The maximum level is a parameter that limits the depth of the recursive search, preventing excessive searching that could lead to wasted computational resources or inefficiency.

[0133] 3) Self-selected stopping strategy

[0134] If the merging cost within a certain bucket is significantly less than that of the bucket above it, then the search is terminated.

[0135] Or randomly select a probability p = min(1, c) prev / c cur Stop jumping to control depth; where c prev c represents the cost of merging the bucket at the previous level. cur This represents the merging cost of the current bucket, Cost = 2·|v|·|v target |;

[0136] If the jump fails (attempting to find the next bucket but the mapping table returns empty / the target bucket is full / the receiving conditions are not met) or exceeds the limit (the deepest level has been reached and can not go any deeper or the consecutive failure limit has been exceeded), the search is eventually forced to terminate, and the candidate is inserted into the current bucket.

[0137] 4) Update execution and write-back

[0138] Perform a merge or replace operation within the target bucket; if the original item is removed, continue to use it as a new candidate for Overflow.

[0139] Figure 3 This is an example of the cascading overflow mechanism of the present invention, which is described in detail below:

[0140] 1) Upper layer: Initial data and conditional judgment (Case classification)

[0141] Figure 3 The upper and middle layers are the raw data blocks, and each frame contains e. n (Element identifier, i.e., key) + value, representing initial data or intermediate variables. Three types of judgment rules are defined using Case 1, Case 2.1, and Case 2.2 to simulate the program / system's "Continue" or "Stop" logic for data processing. Figure 3 In Chinese, "wp" stands for "with probability," meaning "with the probability of..."

[0142] Case 1: The core issue is whether the numerical product exceeds the limit. The formula is |a|×|b|<threshold (thresholds include 13.68, 28.08, etc., which need to be defined according to the scenario). If the condition is met, "Continue (continue the calculation)"; otherwise, it may trigger subsequent overflows.

[0143] Case 2.1 / 2.2: Based on probability p ε or (1-p) ε The branching logic is as follows: Case 2.1 terminates directly with a "Stop" (red cross) because the condition is not met, while Case 2.2 enters the "Continue" branch.

[0144] 2) Critical Triggers: From Single Exceptions to Cascading Exceptions

[0145] Initial exception:<e,7.8> e2:-3.6 is the "initial trigger point".

[0146] First judgment: Case 1: |-3.6|×|7.8|=28.08<+∞→Satisfied, "Continue", continue the calculation.

[0147] Second correlation: e2:-3.6 and e8:3.8 are calculated, |-3.6|×|3.8|=13.68<28.08→still “Continue”, but it is close to the threshold.

[0148] Cascading overflow: As associative computing advances, e 12 -0.03, e 16Data such as 4.1 were involved, and what was originally a "single anomaly" gradually triggered more branch judgments due to correlation and dependence (such as multiplication and transfer calculation), eventually causing the batch of data to exceed the expected range and forming a "chain reaction".

[0149] 3) Lower layer: Data reconstruction after overflow (B1-B8 blocks)

[0150] The lower-level blocks B1-B8 are the result blocks after the overflow, which can be understood as "the final storage / output after the exception propagation":

[0151] The original 'e' is mapped to block B1 (e:7.8), e2:-3.6 goes to block B4, and the rest of 'e'... n Due to the cascading, they were also reassigned to B1-B8.

[0152] Numerical changes (such as e11:1.603) reflect the modifications that overflow makes to the data itself (which may be the result of precision loss, truncation, overflow, or underflow).

[0153] After the merging in step S104 is completed, the valuation is updated. The valuation refers to the approximate value Q(e) of the current e. This step uses a balanced bucket as the valuation query container and directly reads the key value. The method for updating the valuation is: query the two balanced buckets B[i1] and B[i2] corresponding to e and determine whether there is a matching item; if found, directly read its valuation v; if not found, return 0 by default.

[0154] After updating the SIM using the method of this invention, a query operation is performed. The query operation quickly searches for the key in the mapping bucket and returns a calculated value or a default value of 0. Here, the mapping bucket refers to the two specific balanced buckets to which the key e is mapped through two hash functions h1 and h2. In the update phase (step S102), these two buckets are the write targets; in the query phase, these two buckets are called "mapping buckets" and are used to retrieve whether a value for key e exists. In other words, all mapping buckets are balanced buckets, but not all balanced buckets are used as mapping buckets at all times.

[0155] The specific steps for the query operation are as follows:

[0156] a) Input query key e, and use two hash functions h1(e) and h2(e) to map it to two target buckets B[i1] and B[i2];

[0157] b) Scan each item sequentially within B[i1] and B[i2] respectively to check if the item exists.<e,v> ;

[0158] c) If it exists, return its corresponding value v; if neither bucket is hit, return the default value 0.

[0159] The above query operation is characterized by: a fixed query path and O(1) complexity; a small number of items within the bucket, resulting in extremely low search time; and a low error range for approximate estimation. The above query process does not require triggering merging or overflow, making it a high-performance read path.

[0160] This invention can be used in the following scenarios:

[0161] Privacy-preserving big data analytics: For user behavior log data (such as clickstreams, access frequencies, etc.), the method of this invention can be used to process user IDs and event intensity, achieve anonymized compression representation through balanced buckets, and combine MERGE± merging to avoid precision leakage. Ultimately, global statistical features (such as hotspot user estimation) are obtained, solving privacy compliance issues without exposing the original data, achieving a balance between privacy and performance.

[0162] Buffer Management on High-Speed ​​Switches: In port buffer management of network switches, rapid fluctuations in packet traffic lead to frequent flow item replacements. This invention employs an incremental estimation of the flow ID and size for each flow, records the estimated value using a Balance Bucket in the high-speed path, and uses a cascading overflow mechanism to control flow migration. Ultimately, this alleviates the problem of sudden packet loss caused by insufficient space, achieving high-performance dynamic buffer control.

[0163] Sensor Data Collection: In urban intelligent sensing networks, a large number of sensors, such as those for temperature, humidity, and noise, periodically upload data. The method of this invention can be used to perform SIM (Sensitive Identity Module) updates on sensor numbers and measured values, reduce storage redundancy through adaptive merging within data buckets, and ultimately construct a nearly complete urban environmental map. This solves the storage pressure problem caused by uneven data uploads, achieving high precision and lightweight storage.

[0164] Real-time memory monitoring: During cloud service operation, continuous monitoring of program object memory changes is required. The method of this invention can be used to perform a hybrid update of object ID and its memory usage using a combination of aggregate and incremental updates. Overflow management prevents explosive memory growth, ultimately forming a real-time resource monitoring graph. This solves the problems of high latency and strong intrusion of traditional analysis methods, achieving the advantages of lightweight, online analysis.

[0165] Another embodiment of the present invention provides a data stream processing system that supports set-increment hybrid updates, such as... Figure 4 As shown, it includes:

[0166] The SIM update operation receiving module 201 is used to receive SIM update operations and determine the operation type;

[0167] The double hash mapping module 202 is used to perform double hash mapping, which maps the key of a new item to two balanced buckets through two independent hash functions;

[0168] Trial update module 203 is used to perform SIM update using the two balanced buckets obtained from the double hash;

[0169] The cascading overflow module 204 is used to initiate a cascading overflow mechanism if the SIM update fails, and to perform the SIM update through the cooperation of multiple level balance buckets.

[0170] The above division of modules is merely illustrative. In practical applications, the functions described above can be assigned to different functional modules as needed to complete all or part of the functions described in the aforementioned method. The specific working process of each module can be found in the corresponding process in the aforementioned method embodiments, and will not be repeated here.

[0171] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing steps of the method of the present invention.

[0172] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) that stores a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0173] Another embodiment of the present invention provides a computer program product, the computer program product including a computer program, which, when executed by a computer, implements the steps of the method of the present invention.

[0174] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A data stream processing method supporting set- delta hybrid update, characterized by, The method comprises the following steps: receiving a SIM update operation and judging the operation type; performing double hash correspondence, mapping the key of the new item to two balanced buckets through two independent hash functions; performing SIM update by using the two balanced buckets obtained through the double hash correspondence; if the SIM update fails, starting a cascading overflow mechanism to perform SIM update through the cooperation of multiple balanced buckets.

2. The method of claim 1, wherein, The receiving of the SIM update operation and the judging of the operation type comprise: inputting <e, op, v>, wherein e is a key, op is an operator : = or +=, and v is a value to be updated; judging the operation type, which comprises the following two types: set update: <e, : =, v>, which indicates replacing the value of the key e with v; increment update: <e, +=, v>, which indicates adding v to the value of the key e.

3. The method of claim 1, wherein, The double hash correspondence comprises: inputting the key e of the new item; obtaining the number i1 of the first balanced bucket by using a hash function h1(e); obtaining the number i2 of the second balanced bucket by using a hash function h2(e), and ensuring that i1 ≠ i2; obtaining two balanced buckets B[i1] and B[i2] as two target buckets.

4. The method of claim 3, wherein, The size of the balance bucket is d, that is, it contains d items, each of which is a key-value pair <e i ,v i >, and each item is sorted in descending order of the absolute value of v i ; when a new item <e,v> arrives, the operation path of the balance bucket is as follows: 1) When the absolute value |v| of the value v of the new item is less than or equal to the second-smallest value v d-1 , the new item <e, v> is merged with the second-smallest item <e d , v d > with the following merging rule: with probability |v| / (|v d |+|v|) the estimate is set to sgn(v)(|v d |+|v|); |v d |v d |v d |v d |v d |v merging the result into the last item in the balanced bucket; 2) When the absolute value of the value v of the new item is greater than v d-1 , the <e d-1 ,v d-1 > and <e d ,v d > are merged into one item, and the merging rule is as follows: |v d-1 |v d-1 |v d |v d-1 |v d-1 |v d-1 |v d |v |v d |v d-1 |v d |v d |v d |v d-1 |v d |v saving the merged result in the second last item in the balanced bucket, and then writing the new item <e, v> in the last item in the balanced bucket.

5. The method of claim 3, wherein, The SIM update by using the two balanced buckets obtained through the double hash correspondence comprises: checking whether e of the new item exists in B[i1] and B[i2]; if e exists, the set update operation is direct covering, and the increment update operation is adding the current value; if e does not exist in B[i1] or B[i2], but there is a vacancy in B[i1] or B[i2], directly inserting e into the vacancy with the value v. If there is no empty slot, then a merge operation is performed: if |v|≤|v d-1 |, then the new item is merged with the smallest item in B[i1] or B[i2]; otherwise, the smallest and second-smallest items in B[i1] or B[i2] are merged first, and then e is inserted; where |v d-1 | represents the value of the d-1th item in the balanced bucket, i.e., the second-smallest value.

6. The method of claim 1, wherein, The cascading overflow mechanism searches for a minimum cost path and performs merging at an optimal position, and the cascading overflow mechanism comprises the following steps: a search stage: calculating a merging cost, and performing bucket jumping search along a target path based on the merging cost; a stop strategy: stopping if a better merging position is found or a threshold is reached; a kicking bucket stage: kicking out the minimum item from the recursive search process and entering the next bucket until the insertion is completed.

7. The method of claim 1, wherein, The following steps are adopted to perform a query operation: inputting a query key e, and mapping e to two balanced buckets B[i1] and B[i2] by using two hash functions h1(e) and h2(e); sequentially scanning each item in B[i1] and B[i2] respectively to check whether there is an item <e, v>; if there is, returning the corresponding value v; if neither of the two balanced buckets hits, returning a default estimated value 0.

8. A dataflow processing system supporting set-delta hybrid updates, characterized in that, The method comprises the following steps: a SIM update operation receiving module, configured to receive a SIM update operation and judge the operation type; a double hash correspondence module, configured to perform double hash correspondence, mapping the key of the new item to two balanced buckets through two independent hash functions; a trial update module, configured to perform SIM update by using the two balanced buckets obtained through the double hash correspondence; a cascading overflow module, configured to start a cascading overflow mechanism to perform SIM update through the cooperation of multiple balanced buckets if the SIM update fails.

9. A computer device, comprising: comprising a memory storing a computer program configured to be executed by a processor, the computer program comprising instructions for performing the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, which, when executed by a computer, implements the method of any one of claims 1-7.

Images