Data processing method and related products

By combining the exponential histogram and count-min sketch algorithm, this data structure solves the problems of high storage cost and latency in stream processing systems when processing retracted data streams. It achieves efficient processing of unbounded length data streams and output of approximate aggregation functions, and is suitable for scenarios with retracted data streams and a large number of groups.

CN122095352APending Publication Date: 2026-05-26HUAWEI CLOUD COMPUTING TECHNOLOGIES CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
HUAWEI CLOUD COMPUTING TECHNOLOGIES CO LTD
Filing Date
2024-03-05
Publication Date
2026-05-26

AI Technical Summary

Technical Problem

Existing stream processing systems require storing the entire data stream to support aggregation function calculations when processing data streams with retractions, resulting in high storage costs and latency. Furthermore, they cannot effectively track the maximum and minimum values ​​of each group when processing a large number of groups, exceeding the limitations of addressable memory.

Method used

It employs a combined data structure of exponential histogram and count-min sketch algorithm, dynamically adjusts memory usage by setting precision and error probability, supports processing of unbounded data streams, and provides an output of an approximate aggregation function, thus avoiding the need to store the entire data stream.

Benefits of technology

It effectively reduces memory requirements, supports real-time computation on data streams with retraction, and provides approximate but accurate and error-rate-controllable aggregation function results, suitable for processing large numbers of data streams.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122095352A_ABST
    Figure CN122095352A_ABST
Patent Text Reader

Abstract

This invention provides a data processing method and related products. The method includes: consuming a first event in a data stream based on a first data structure corresponding to a first function, wherein the first function is used to find a target value of a target attribute in the data stream, and the size of the resources occupied by the first data structure in memory is determined based on the accuracy of a first estimate and a sublinear cost, wherein the first estimate is an estimate of the target value; and obtaining the first estimate as the output of the first function based on the consumption.
Need to check novelty before this filing date? Find Prior Art