Data persistence estimation method, device, electronic device, and storage medium

Abstract

The embodiment of the present application provides a data persistence estimation method, device, electronic device, and storage medium, which belongs to the field of data processing technology. The method includes: obtaining an original data stream within a preset time window, the original data stream includes multiple original data items, deduplicating each original data item through a burst filter, and obtaining multiple target data items. For each target data item, the heat category of the target data item is obtained through a cold filter, and the heat category includes cold data, warm data, or hot data. If the heat category indicates that the target data item is cold data or warm data, the persistence value of the target data item is estimated through a cold filter, and if the heat category indicates that the target data item is hot data, the persistence value of the target data item is estimated through a hot memory, which can improve the accuracy of data persistence estimation.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data persistence estimation method, device, electronic device, and storage medium. Background Art

[0002] Data persistence estimation is to estimate the frequency of occurrence of data items within a specific time window to identify key items or abnormal items. The frequency of occurrence of data items in a highly skewed data stream is uneven, and the frequency of occurrence of hot data items is much higher than that of cold data items. When processing highly skewed data streams, traditional methods often cannot effectively distinguish between cold data items and hot data items, resulting in the persistence of hot data items being underestimated and the persistence of cold data items being overestimated, resulting in low accuracy of data persistence estimation. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a data persistence estimation method, device, electronic device, and storage medium, aiming to improve the accuracy of data persistence estimation.

[0004] To achieve the above object, a first aspect of an embodiment of the present application proposes a data persistence estimation method, the method comprising:

[0005] Acquire an original data stream within a preset time window, wherein the original data stream includes a plurality of original data items;

[0006] Deduplication processing is performed on each of the original data items through a burst filter to obtain multiple target data items;

[0007] For each of the target data items, obtaining a heat category of the target data item through a cold filter; the heat category includes cold data, warm data or hot data;

[0008] If the heat category indicates that the target data item is cold data or warm data, estimating the persistence value of the target data item through the cold filter;

[0009] If the thermal category indicates that the target data item is hot data, a persistence value of the target data item is estimated by a hot memory.

[0010] In some embodiments, the cold filter includes a primary storage structure, the primary storage structure is provided with a first hash function and a first persistence threshold, and obtaining the heat category of the target data item through the cold filter includes:

[0011] Performing a hash calculation on the target data item according to the first hash function to obtain a first hash value;

[0012] Acquire a first storage location of the target data item in the primary storage structure according to the first Hash value;

[0013] Obtaining a minimum persistence value of the first storage location to obtain a first initial count;

[0014] A heat category of the target data item is obtained according to the first initial count and the first persistence threshold.

[0015] In some embodiments, the cold filter further includes a secondary storage structure, the secondary storage structure is provided with a second hash function and a second persistence threshold, and acquiring the heat category of the target data item according to the first initial count and the first persistence threshold includes:

[0016] If the first initial count is less than the first persistence threshold, determining that the heat category of the target data item is cold data;

[0017] If the first initial count is greater than or equal to the first persistence threshold, performing hash calculation on the target data item according to the second hash function to obtain a second hash value, obtaining a second storage location of the target data item in the secondary storage structure according to the second hash value, obtaining a minimum persistence value of the second storage location, and obtaining a second initial count;

[0018] If the second initial count is less than the second persistence threshold, determining that the heat category of the target data item is warm data;

[0019] If the second initial count is greater than or equal to the second persistence threshold, it is determined that the heat category of the target data item is hot data.

[0020] In some embodiments, the primary storage structure is provided with a small counter, the secondary storage structure is provided with a large counter, and if the heat category indicates that the target data item is cold data or warm data, estimating the persistence value of the target data item through the cold filter includes:

[0021] If the heat category indicates that the target data item is cold data, the first initial count is updated by the small counter to obtain a persistence value of the target data item;

[0022] If the heat category indicates that the target data item is warm data, the second initial count is updated by the large counter to obtain a persistence value of the target data item.

[0023] In some embodiments, the hot memory includes a third hash function and a plurality of storage units, and if the heat category indicates that the target data item is hot data, estimating the persistence value of the target data item through the hot memory includes:

[0024] If the heat category indicates that the target data item is hot data, performing a hash calculation on the target data item according to the third hash function to obtain a third hash value;

[0025] Screening the plurality of storage units according to the third Hash value to obtain a target unit;

[0026] A persistence value of the target data item is estimated based on the target unit.

[0027] In some embodiments, estimating the persistence value of the target data item according to the target unit includes:

[0028] If the target unit stores the target data item, then obtaining the original persistent value of the target data item, updating the original persistent value of the target data item, and obtaining the persistent value of the target data item;

[0029] If the target unit does not store the target data item and there is remaining storage space in the target unit, the target data item is stored in the target unit, and the target data item is counted to obtain a persistence value of the target data item;

[0030] If the target unit does not store the target data item and there is no remaining storage space in the target unit, the data item with the smallest original persistence value is filtered out from the target unit to obtain a candidate data item, the candidate data item is replaced with the target data item, and the original persistence value of the candidate data item is updated to obtain the persistence value of the target data item.

[0031] In some embodiments, after estimating the persistence value of the target data item by using a thermal memory if the heat category indicates that the target data item is hot data, the data persistence estimation method further includes:

[0032] Obtaining a query request, wherein the query request includes a reference data item;

[0033] In response to the query request, query the persistence value of the reference data item from the cold filter to obtain a first persistence value;

[0034] querying the hot memory for the persistence value of the reference data item to obtain a second persistence value;

[0035] The first persistent value and the second persistent value are summed to obtain a target persistent value of the reference data item.

[0036] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a data persistence estimation device, the device comprising:

[0037] A first acquisition module, used to acquire an original data stream within a preset time window, wherein the original data stream includes a plurality of original data items;

[0038] A deduplication processing module, used for performing deduplication processing on each of the original data items through a burst filter to obtain multiple target data items;

[0039] A second acquisition module is used to acquire, for each of the target data items, a heat category of the target data item through a cold filter; the heat category includes cold data, warm data or hot data;

[0040] a first estimation module, configured to estimate the persistence value of the target data item through the cold filter if the heat category indicates that the target data item is cold data or warm data;

[0041] The second estimation module is configured to estimate the persistence value of the target data item by using a thermal memory if the thermal category indicates that the target data item is hot data.

[0042] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0043] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0044] The data persistence estimation method, data persistence estimation device, electronic device, and computer-readable storage medium of the embodiment of the present application obtain the original data stream within a preset time window, and the original data stream includes multiple original data items to perform persistence estimation on each original data item. In order to improve the processing speed of the data stream, each original data item is deduplicated by a burst filter to filter the data items that appear repeatedly in a single time window, reduce the repeated invalid updates of the data items, and obtain multiple target data items. For each target data item, the heat category of the target data item is obtained by a cold filter to distinguish between cold data items and hot data items. If the heat category indicates that the target data item is cold data or warm data, the persistence value of the target data item is estimated by a cold filter to capture the low persistence characteristics of the cold data item to avoid the persistence of the cold data item being overestimated. If the heat category indicates that the target data item is hot data, the persistence value of the target data item is estimated by a hot memory to capture the high persistence characteristics of the hot data item to avoid the persistence of the hot data item being underestimated. Cold data items and hot data items have different persistence characteristics. The cold-hot separation mechanism is used to separate the cold data items and hot data items, which reduces the problem of underestimation of hot data items and overestimation of cold data items caused by hash conflicts and improves the accuracy of persistence estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a data persistence estimation method provided in an embodiment of the present application;

[0046] Figure 2 yes Figure 1 Flow chart of step S130 in FIG.

[0047] Figure 3 yes Figure 2 Flow chart of step S240 in FIG.

[0048] Figure 4 yes Figure 1 Flow chart of step S140 in FIG.

[0049] Figure 5 yes Figure 1 Flow chart of step S150 in FIG.

[0050] Figure 6 yes Figure 5 Flow chart of step S530 in FIG.

[0051] Figure 7 is another flow chart of the data persistence estimation method provided by an embodiment of the present application;

[0052] Figure 8 is a structural diagram of a data persistence estimation device provided in an embodiment of the present application;

[0053] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] Data persistence estimation is to estimate the frequency of occurrence of data items within a specific time window to identify key items or abnormal items. The frequency of occurrence of data items in a highly skewed data stream is uneven, and the frequency of occurrence of hot data items is much higher than that of cold data items. When processing highly skewed data streams, traditional methods often cannot effectively distinguish between cold data items and hot data items, resulting in the persistence of hot data items being underestimated and the persistence of cold data items being overestimated, resulting in low accuracy of data persistence estimation.

[0058] Based on this, embodiments of the present application provide a data persistence estimation method, a data persistence estimation device, an electronic device, and a computer-readable storage medium, aiming to improve the accuracy of data persistence estimation.

[0059] The data persistence estimation method, data persistence estimation device, electronic device and computer-readable storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the data persistence estimation method in the embodiments of the present application is described.

[0060] The data persistence estimation method provided in the embodiment of the present application relates to the field of data processing technology. The data persistence estimation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data persistence estimation method, etc., but is not limited to the above forms.

[0061] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0062] Figure 1 It is an optional flowchart of the data persistence estimation method provided in an embodiment of the present application, which may include but is not limited to steps S110 to S150.

[0063] Step S110, obtaining an original data stream within a preset time window, where the original data stream includes a plurality of original data items;

[0064] Step S120, performing deduplication processing on each original data item through a burst filter to obtain multiple target data items;

[0065] Step S130, for each target data item, obtaining a heat category of the target data item through a cold filter; the heat category includes cold data, warm data or hot data;

[0066] Step S140, if the heat category indicates that the target data item is cold data or warm data, estimating the persistence value of the target data item through a cold filter;

[0067] Step S150 : if the heat category indicates that the target data item is hot data, the persistence value of the target data item is estimated through the hot storage.

[0068] In step S110 of some embodiments, data persistence estimation technology has been widely used in the fields of network security, traffic analysis, advertising fraud detection, etc., and can be used to estimate the frequency of occurrence of data items within a specific time window, so as to identify key data items or abnormal data items from the data stream. When performing the data persistence estimation task, it is necessary to obtain the input data stream. In order to ensure the standardization of the persistence definition, the input data stream is divided into time windows according to a preset window length to obtain the original data stream within the preset time window, and the preset window length is the length of the preset time window. The original data stream includes multiple original data items, which are data items of the original data stream and can be set according to specific business scenarios. Each original data item has a unique identifier, and the identifier can be an ID identifier.

[0069] In network security scenarios, raw data items can be IP addresses, port numbers, packet types, etc. By estimating the persistence of raw data items, such as obtaining the frequency of IP addresses appearing in multiple time windows, persistent low-frequency attacks such as slow but persistent distributed denial of service attacks (DDos) or abnormal traffic can be detected. In traffic analysis scenarios, raw data items can be user behavior types, page requests, session durations, etc. By estimating the persistence of raw data items, high-traffic time periods and popular content can be identified. In advertising fraud detection scenarios, raw data items can be click sources, click times, click frequencies, etc. By estimating the persistence of raw data items, the behavior of automated scripts forging ad clicks multiple times can be identified.

[0070] In step S120 of some embodiments, in order to improve the data stream processing speed, each original data item is deduplicated through a burst filter to quickly filter out data items that appear repeatedly within a single preset time window, thereby obtaining multiple target data items. This solves the problem of repetitive updates of data items within a single time window, reduces the computing resources consumed in processing duplicate data items, and improves overall throughput.

[0071] In the related art, Bloom filters are used to perform deduplication operations, but Bloom filters have a certain possibility of false positives, which makes it impossible to correctly record the cold data items that actually exist, resulting in the underestimation of the persistence of the cold data items. In order to improve the accuracy of persistence estimation and avoid the delay of data processing caused by complex deduplication steps, the burst filter of the embodiment of the present application adopts a lightweight hash structure, which only records the data item when it first appears, so as to meet the real-time requirements of scenarios such as financial transaction scenarios, and reduce the false positives of Bloom filters.

[0072] Specifically, the burst filter includes multiple storage units and hash functions, each storage unit includes multiple cells, and the hash function can be MD5, SHA-1, SHA-256, SHA-3, etc. The original data item is hashed by the hash function, and the original data item is mapped to a specific storage unit. The ID identifier of the original data item is obtained, and the ID identifier of each cell in the specific storage unit is compared with the ID identifier of the original data item. If the specific storage unit does not store the ID identifier of the original data item, the ID identifier of the original data item is stored in the empty cell in the specific storage unit to obtain the target data item. If the specific storage unit already stores the ID identifier of the original data item, the original data item is not recorded.

[0073] The original data item has a status mark, which includes open or closed. The status mark of the original data item is detected by the burst filter. If the status mark is open, it means that the original data item has not been processed in the current time window, that is, the preset time window. Then the ID mark of the original data item is inserted into a specific storage unit, and its status mark is updated from open to closed, and the original data item is passed to the cold filter, and the status mark of the original data item is set to open. If the status mark is closed, it means that the original data item has been processed in the current time window, and the burst filter skips the current insertion step to avoid repeated updates. For example, when the original data item e arrives for the first time, the burst filter recognizes it as a new item, updates the mark and passes it to the cold filter. When the original data item e arrives again in the same window, the burst filter directly skips processing, thereby reducing unnecessary operations. This design not only avoids multiple updates of the counter in a single time window, fundamentally eliminates the overestimation problem, ensures the accuracy of persistence estimation in a single time window, but also reduces the computational pressure of subsequent modules.

[0074] In order to reduce frequent access to the storage structure and improve the overall throughput, the embodiment of the present application adopts a centralized update strategy. At the end of the current time window, that is, when each original data item in the preset time window has completed the deduplication operation, all target data items in the burst filter are uniformly updated to the cold filter and hot storage in batches, and all flags of these target data items are reset to on, so as to prepare for the processing of the next time window, that is, the cold filter, to ensure the statistical consistency at the window level. By adopting a lightweight burst filter and a centralized update strategy, the high computational burden brought by online updates is avoided, the throughput of the algorithm is significantly improved, and it can run in real time in a high-speed data stream environment.

[0075] As the scale of data streams continues to grow, persistence estimation techniques need to consider algorithm efficiency and memory usage. In related technologies, a fixed-size counter is assigned to each data item to accommodate the potential demand for high-persistence items. However, in highly skewed data streams, most data items have low persistence (cold items) and a few data items have high persistence (hot items), which means that only a few high-persistence items occupy the main memory resources, resulting in a large number of high-bit wastes of cold item counters, significantly reducing memory utilization efficiency. In addition, hash conflicts further exacerbate this problem, especially when there are a large number of cold items, which may cause the counters of high-persistence items to be incorrectly overwritten or repeatedly updated.

[0076] In addition, existing persistence estimation techniques also have limitations in scalability and adaptability. Most techniques are optimized for specific application scenarios and lack the flexibility to adapt to multiple scenarios. For example, under different data flow distribution and persistence requirements, the processing strategy for cold items and hot items may need to be adjusted, but it is usually impossible to dynamically adjust memory allocation and counter design. Moreover, in a distributed environment, existing techniques are difficult to scale to support large-scale parallel computing, which is particularly critical in cloud computing and IoT scenarios that need to process massive amounts of data.

[0077] In order to achieve a balance between memory utilization, accuracy and processing speed, the embodiment of the present application realizes the separation of cold items and hot items through the collaborative work of the three-layer structure of the burst filter, cold filter and hot memory, and utilizes a multi-stage processing mechanism, solving the problems of memory waste, overestimation and underestimation errors, and lack of real-time performance in the related technology in the high-skewness data flow scenario. In addition, the three-layer structure of the embodiment of the present application is flexible in design, and can dynamically adjust the memory allocation ratio of the cold filter and the hot memory. Through the layered design and dynamic adjustment mechanism to adapt to the persistent distribution characteristics of different data streams, it is easy to expand to a variety of application scenarios, such as network security, traffic analysis, advertising fraud detection, and real-time Internet of Things data processing.

[0078] Cold items and hot items in data streams have significantly different persistence characteristics. There are many cold items but low persistence, while there are few hot items but high persistence. Based on the characteristics of persistence distribution in data streams, cold filters are used to capture the low persistence characteristics of cold items, avoid cold items from occupying the memory of hot storage, and optimize memory usage efficiency. Hot storage records high-persistence hot items, providing accurate persistence statistics and fast query capabilities. Through dynamic collaboration between modules, efficient distinction and processing of cold items and hot items are achieved to support persistence estimation and efficient query.

[0079] See also Figure 2 In some embodiments, step S130 may include but is not limited to steps S210 to S240:

[0080] Step S210, performing hash calculation on the target data item according to a first hash function to obtain a first hash value;

[0081] Step S220, obtaining a first storage location of the target data item in the primary storage structure according to the first hash value;

[0082] Step S230, obtaining a minimum persistence value of the first storage location to obtain a first initial count;

[0083] Step S240: Obtain the heat category of the target data item according to the first initial count and the first persistence threshold.

[0084] In step S210 of some embodiments, the cold filter includes a primary storage structure, the primary storage structure includes a plurality of storage units, each storage unit is provided with a hash function, and the hash function is used as a first hash function. The first hash functions of each storage unit may be the same or different. The primary storage structure is also provided with a persistence threshold, and the persistence threshold is used as a first persistence threshold, and the first persistence threshold is used to judge the heat of the target data item. In order to locate the storage location of the target data item, for each target data item, the target data item is input into the first hash function of each storage unit for hash calculation to obtain the first hash value of each storage unit.

[0085] In step S220 of some embodiments, each storage unit includes a plurality of cells, and the first storage position can be represented by a position index of the cell in the storage unit. The first storage position of each storage unit in the primary storage structure of the target data item is obtained according to the first hash value of each storage unit. For example, the primary storage structure L1 includes d1 storage units, the target data item is e, the first hash function of the i-th storage unit is hash1i, and the first hash value of the i-th storage unit is hash1i(e), then the first storage position of the i-th storage unit is L1[i][hash1i(e)], i is greater than or equal to 1 and less than or equal to d1.

[0086] In step S230 of some embodiments, the original persistence value of each storage unit at the first storage position is obtained, and the minimum original persistence value is selected as the minimum persistence value to obtain a first initial count. The original persistence value is the count value of the first storage position counted by the counter, and the first initial count is an integer greater than or equal to 0.

[0087] In step S240 of some embodiments, a heat category of the target data item is obtained according to the first initial count and the first persistence threshold, where the heat category includes non-hot data or hot data, and the non-hot data includes cold data and warm data.

[0088] Through the above steps S210 to S240, the heat categories of data items can be distinguished to perform persistence estimation on the data items based on the heat categories, thereby avoiding hash conflicts caused by mapping data items of different heat to the same hash value, thereby avoiding overestimation and underestimation problems caused by hash conflicts and improving the accuracy of persistence estimation.

[0089] See also Figure 3 In some embodiments, step S240 may include but is not limited to steps S310 to S340:

[0090] Step S310, if the first initial count is less than the first persistence threshold, determining that the heat category of the target data item is cold data;

[0091] Step S320: if the first initial count is greater than or equal to the first persistence threshold, performing hash calculation on the target data item according to the second hash function to obtain a second hash value, obtaining a second storage location of the target data item in the secondary storage structure according to the second hash value, obtaining a minimum persistence value of the second storage location, and obtaining a second initial count;

[0092] Step S330: if the second initial count is less than the second persistence threshold, determining that the heat category of the target data item is warm data;

[0093] Step S340: If the second initial count is greater than or equal to the second persistence threshold, determine that the heat category of the target data item is hot data.

[0094] In step S310 of some embodiments, the cold filter also includes a secondary storage structure, which includes multiple storage units, each of which is provided with a second hash function. The second hash functions of each storage unit may be the same or different. The secondary storage structure is also provided with a second persistence threshold, which is used to assist the first persistence threshold in determining the heat of the target data item. If the first initial count is less than the first persistence threshold, the heat category of the target data item is determined to be cold data, and the persistence value of the target data item can be estimated through the primary storage structure.

[0095] In step S320 of some embodiments, if the first initial count is greater than or equal to the first persistence threshold, the target data item is input into the second hash function of each storage unit for hash calculation to obtain the second hash value of each storage unit. Each storage unit includes a plurality of cells, and the second storage position can be represented by the position index of the cell in the storage unit, and the cell is used to store the original persistent value and status mark of the data item. The second storage position of each storage unit in the secondary storage structure of the target data item is obtained according to the second hash value of each storage unit. For example, the secondary storage structure L2 includes d2 storage units, the target data item is e, the second hash function of the i-th storage unit is hash2i, and the second hash value of the i-th storage unit is hash2i(e), then the second storage position of the i-th storage unit is L2[i][hash2i(e)], i is greater than or equal to 1 and less than or equal to d2. The original persistent value of each storage unit in the second storage position is obtained, and the smallest original persistent value is selected as the minimum persistence value to obtain the second initial count.

[0096] In step S330 of some embodiments, if the second initial count is less than the second persistence threshold, the heat category of the target data item is determined to be warm data, and the persistence value of the target data item can be estimated through the secondary storage structure.

[0097] In step S340 of some embodiments, if the second initial count is greater than or equal to the second persistence threshold, the heat category of the target data item is determined to be hot data, and the persistence value of the target data item can be estimated through the hot memory.

[0098] Through the above steps S310 to S340, it is possible to determine whether the target data item is cold data, warm data, or hot data, so as to solve the problem of inaccurate persistence estimation caused by hash conflicts through the cold-hot separation mechanism, reduce the error of persistence estimation, and improve the accuracy of the overall estimation. In addition, the dynamic data transfer between the cold filter and the hot storage is controlled by the persistence threshold, and the storage strategy is adjusted according to the real-time characteristics of the data flow, ensuring the efficient allocation of memory resources.

[0099] See also Figure 4In some embodiments, step S140 may include but is not limited to step S410 or step S420:

[0100] Step S410, if the heat category indicates that the target data item is cold data, the first initial count is updated by a small counter to obtain a persistence value of the target data item;

[0101] Step S420: If the heat category indicates that the target data item is warm data, the second initial count is updated by the large counter to obtain a persistence value of the target data item.

[0102] In step S410 of some embodiments, in order to improve memory utilization and the accuracy of persistence estimation, a hierarchical storage structure is adopted, and a small counter is provided in the primary storage structure, and a large counter is provided in the secondary storage structure, thereby avoiding resource waste caused by a fixed counter design. The number of count bits of the small counter is smaller than that of the large counter, for example, the number of count bits of the small counter may be 8 bits, and the number of count bits of the large counter may be 32 bits.

[0103] If the heat category indicates that the target data item is cold data, the original persistent value and status mark of the first storage position of each storage unit in the primary storage structure are obtained, and each original persistent value and the first initial count are compared respectively. If the original persistent value is equal to the first initial count and the status mark is on, the original persistent value is increased by 1 through a small counter, and the status mark is updated to off to obtain the persistence value of the target data item.

[0104] In step S420 of some embodiments, if the heat category indicates that the target data item is warm data, the original persistent value and the status mark of the second storage position of each storage unit in the secondary storage structure are obtained, and each original persistent value is compared with the second initial count respectively. If the original persistent value is equal to the second initial count and the status mark is on, the original persistent value is increased by 1 through the large counter, and the status mark is updated to off, so as to obtain the persistence value of the target data item.

[0105] In some embodiments, the primary storage structure is represented as L1 and the secondary storage structure is represented as L2. L1 includes multiple small storage buckets, each of which is equipped with a small counter to record cold items with low persistence values. L2 includes fewer large storage buckets, each of which is equipped with a large counter to record data items with slightly higher persistence. After the data item enters the cold filter, it will be dynamically allocated to L1 or L2 according to the range of its persistence value, and the data item will be mapped to the buckets of L1 and L2 respectively using a hash function. If the persistence value of the data item is low, the corresponding counter in L1 is updated. If the count value of a counter in L1 exceeds the first persistence threshold, the data item will be transferred from L1 to L2, and its corresponding counter will be updated in L2 to continue to accumulate its persistence value. If the count value of a counter in L2 exceeds the second persistence threshold, the corresponding data item is identified as a high persistence item and is transferred to the hot storage.

[0106] For example, when the target data item e arrives for the first time, its hash value is mapped to a bucket of L1, and the counter corresponding to the target data item e in this bucket increases from 0 to 1. As the target data item e arrives multiple times, the counter gradually increases to the first persistence threshold, triggering the upgrade mechanism, and the target data item e is transferred to L2. In L2, the counter of the target data item e continues to increase until it exceeds the second persistence threshold, at which point the target data item is identified as a high-persistence item and transferred to the hot storage.

[0107] In the above steps S410 to S420, cold items with low persistence are identified through cold filters, and only persistent changes of cold items within the time window are recorded. The cold filter adopts a layered design and optimizes memory usage efficiency through a combination of different counters, avoiding resource waste caused by the use of fixed counters for cold items, while ensuring the statistical accuracy of cold items and hot items.

[0108] See also Figure 5 In some embodiments, step S150 may include but is not limited to steps S510 to S530:

[0109] Step S510, if the heat category indicates that the target data item is hot data, performing a hash calculation on the target data item according to a third hash function to obtain a third hash value;

[0110] Step S520, screening multiple storage units according to the third hash value to obtain a target unit;

[0111] Step S530: estimating the persistence value of the target data item according to the target unit.

[0112] In step S510 of some embodiments, the hot memory is responsible for storing and managing high-persistence items, and is an efficient storage structure based on entries. The hot memory includes a third hash function and multiple storage units, each storage unit includes multiple entries, and each entry is used to store an ID, a state mark, and an original persistent value of a data item. If the heat category indicates that the target data item is hot data, the target data item is input into the third hash function for hash calculation to obtain a third hash value.

[0113] In step S520 of some embodiments, after the target data item arrives at the hot storage, it is mapped to a specific storage unit by the third hash value to obtain the target unit. To reduce hash conflicts, a double-table hash calculation can be used to obtain the target unit. Specifically, the number of storage units, length, is obtained, and a modulo calculation is performed on length*(length-1) according to the third hash value to obtain an intermediate hash value, where * represents a multiplication operation. A modulo calculation is performed on length according to the intermediate hash value to obtain the target unit of the first hash table. A division calculation is performed on length-1 according to the intermediate hash value to obtain the target unit of the second hash table.

[0114] In step S530 of some embodiments, the persistence value of the target data item is estimated by a counter set by the target unit.

[0115] In the above steps S510 to S530, the cold filter and the hot memory work together to allow only hot items to enter the hot memory, which significantly reduces hash conflicts and redundant calculations and improves the accuracy and efficiency of persistence estimation.

[0116] See also Figure 6 In some embodiments, step S530 may include but is not limited to step S610, step S620 or step S630:

[0117] Step S610: if the target unit stores the target data item, the original persistent value of the target data item is obtained, and the original persistent value of the target data item is updated to obtain the persistent value of the target data item;

[0118] Step S620: If the target unit does not store the target data item and there is remaining storage space in the target unit, the target data item is stored in the target unit, and the target data item is counted to obtain a persistence value of the target data item;

[0119] Step S630, if the target unit does not store the target data item and there is no remaining storage space in the target unit, filter out the data item with the smallest original persistence value from the target unit to obtain a candidate data item, replace the candidate data item with the target data item, and update the original persistence value of the candidate data item to obtain the persistence value of the target data item.

[0120] In step S610 of some embodiments, each entry of the target unit is traversed and the status of the entry is detected. If an entry stores the ID of the target data item and the status mark is on, the original persistence value of the target data item is increased by 1, and the status mark is updated to off, to obtain the persistence value of the target data item.

[0121] In step S620 of some embodiments, if the target data item is not stored in any entry of the target unit and there is remaining storage space in the target unit, the ID identifier of the target data item is stored in the empty entry of the target unit, and the target data item is started to be counted, 1 is used as the persistence value of the target data item, and the status mark corresponding to the entry is set to off.

[0122] In step S630 of some embodiments, if each entry of the target unit does not store the ID of the target data item, and the target unit does not have any remaining storage space, that is, there is no empty entry, then the low-priority item in the entry needs to be replaced with a high-priority item. Compare the original persistent values ​​stored in each entry of the target unit, select the data item with the smallest original persistent value, and obtain the candidate data item. Obtain the original persistent value of the candidate data item, add 1 to the original persistent value to obtain the candidate persistent value, perform a modulo calculation on the candidate persistent value according to the third hash value, and if the calculation result is 0, that is, the third hash value is equal to the candidate persistent value, replace the candidate data item with the target data item, and use the candidate persistent value as the persistence value of the target data item, and set the status mark of the target data item to off.

[0123] For example, when a target data item is promoted from a cold filter to a hot memory, the storage unit corresponding to its hash value is checked for vacancies. If there is already a low-persistence item in the storage unit and its priority is lower than that of the target data item, a replacement operation is performed to replace the low-persistence item with the target data item and reinitialize the counter.

[0124] Assume that the data stream contains five data items, e1, e2, e3, e4, and e5. Data items e1 and e2 are both identified as new items by the burst filter and passed to the cold filter. The persistence value of e1 is low, and it always stays in L1, and its counter is finally recorded as 4. However, e2 continues to appear, and its counter increases from the first persistence threshold of 10 in L1 to the second persistence threshold of 50 in L2, enters the hot memory and is recorded as a high persistence item. Data items e3 and e4 are replaced in the hot memory part, and e3 is retained as a higher priority item.

[0125] Through the above steps S610 to S630, the hot items can be efficiently stored and managed, and the persistence value of the hot items can be accurately estimated.

[0126] See also Figure 7In some embodiments, after step S150, the data persistence estimation method may further include but is not limited to steps S710 to S740:

[0127] Step S710, obtaining a query request, the query request including a reference data item;

[0128] Step S720, in response to the query request, query the persistence value of the reference data item from the cold filter to obtain a first persistence value;

[0129] Step S730, querying the persistence value of the reference data item from the hot memory to obtain a second persistence value;

[0130] Step S740: sum the first persistent value and the second persistent value to obtain a target persistent value of the reference data item.

[0131] In step S710 of some embodiments, a query request is obtained, the query request including a reference data item, and is used to query a persistent value of the reference data item.

[0132] In step S720 of some embodiments, in response to the query request, a hash calculation is performed on the reference data item according to the first hash function to obtain a first query hash value, a third storage location of the storage unit in the primary storage structure is obtained according to the first query hash value, and the smallest persistence value of the third storage location is selected to obtain a first intermediate value. If the first intermediate value is less than the first persistence threshold, the first intermediate value is used as the first persistence value to obtain a target persistence value of the reference data item.

[0133] If the first intermediate value is greater than or equal to the first persistence threshold, the reference data item is hashed according to the second hash function to obtain a second query hash value, the fourth storage position of the storage unit in the secondary storage structure is obtained according to the second query hash value, and the smallest persistence value of the fourth storage position is selected to obtain a second intermediate value. If the second intermediate value is less than the second persistence threshold, the first intermediate value and the second intermediate value are summed, and the sum is used as the first persistence value to obtain a target persistence value of the reference data item.

[0134] In some embodiments, in steps S730 to S740, if the second intermediate value is greater than or equal to the second persistence threshold, a hash calculation is performed on the reference data item according to a third hash function set in the hot memory to obtain a third query hash value, and a target storage unit is obtained according to the third query hash value with reference to step S520, and if the target storage unit stores the reference data item, the persistence value of the reference data item is used as the second persistence value. The first persistence value and the second persistence value are summed to obtain a target persistence value of the reference data item.

[0135] Through the above steps S710 to S740, the persistence value of the data item can be checked layer by layer according to the specified persistence threshold to gradually return the most accurate statistical results, thereby reducing the interference of hash conflicts on the statistical results.

[0136] During the query process, hot storage can be accessed first to obtain high-persistence items to improve the query efficiency of high-persistence items. If the hot storage stores the reference data item, its count value is returned. Continue to check the secondary storage structure of the cold filter. If the minimum persistence value of the reference data item in the secondary storage structure is less than the second persistence threshold, the minimum persistence value is returned as the count value. Continue to check the primary storage structure of the cold filter. If the minimum persistence value of the reference data item in the primary storage structure is less than the first persistence threshold, the minimum persistence value is returned as the count value. Add the count values ​​returned by each layer to obtain the target persistence value of the reference data item. If the reference data item is not found in each layer, the target persistence value of the reference data item is 0.

[0137] For example, when querying data item e, an entry in the hot memory is located through data item e, and the counter value of the entry is returned. If data item e is not found in the hot memory, the returned counter value is 0. Continue to check the secondary storage structure L2 in the cold filter. If the minimum counter value located in L2 according to data item e is less than the second persistence threshold, return the minimum counter value as the counter value of L2, otherwise return the counter value 0. Continue to check the primary storage structure L1 in the cold filter. If the minimum counter value located in L1 according to data item e is less than the first persistence threshold, return the minimum counter value as the counter value of L1, otherwise return the counter value 0. Add the counter values ​​returned by the hot memory, L2, and L1 to obtain the persistence value of data item e.

[0138] The hierarchical persistence processing structure of the embodiment of the present application includes a burst filter, a cold filter and a hot memory, and the data characteristics of cold items and hot items can be efficiently distinguished through three-layer collaboration. As the first layer module, the burst filter avoids repeated updates within a single time window, which not only reduces the burden on the cold filter and the hot memory. In addition, the burst filter implements the filtering of repeated updates through the state mark, and combines the centralized update mechanism to reduce the complexity of online operations, and uniformly updates to the cold filter or hot memory at the end of the window, which greatly improves the throughput and is suitable for data stream processing scenarios with high real-time requirements. The cold filter captures low persistence values ​​and transfers high persistence items to the hot part step by step when necessary, ensuring accurate statistics while optimizing memory usage. The hot part focuses on the accurate statistics of high persistence items, supports fast query and real-time update, and is a key component of high performance and high accuracy in the entire structure. The accurate statistics and fast query capabilities of the hot memory give it a significant advantage in processing high persistence data items. Through the complete identifier storage, dynamic update and replacement strategy, the statistical errors caused by hash conflicts can be avoided, while supporting efficient query to meet real-time application requirements.

[0139] The embodiment of the present application also introduces a differentiated counter design and memory optimization mechanism. Taking full advantage of the distribution characteristics in high-skewness data streams, cold items are recorded using small counters to reduce memory usage, while hot items are recorded in detail using large counters. This dynamic optimization mechanism avoids the memory waste problem caused by the fixed counter size design, while ensuring accurate statistics of high-persistence data items.

[0140] The embodiment of the present application also introduces the hierarchical structure and distribution optimization of the cold filter. By distributing data items to a two-layer storage structure and dynamically allocating resources based on the persistence value, accurate statistics can be achieved in low-persistence data items, and the errors caused by hash conflicts can be reduced. The two-layer design of the cold filter combined with an independent hash function effectively captures the distribution characteristics of low-persistence values, and ensures that high-persistence items can seamlessly transition to hot storage through a dynamic upgrade mechanism, further optimizing the overall performance of persistence estimation.

[0141] In addition, the embodiment of the present application can also dynamically adjust the memory allocation ratio of the cold filter and the hot memory according to the ratio of cold items to hot items. This mechanism not only adapts to the typical characteristics of high-skewness data streams, but also provides more extensive support for persistence estimation in different application scenarios (such as network security monitoring, advertising fraud detection, etc.). Through three-layer structure design, memory optimization and dynamic adjustment, throughput improvement and high-skewness data stream adaptation mechanism, the overall performance of persistence estimation is improved, and the problems of memory waste, overestimation and underestimation errors and lack of real-time performance are solved.

[0142] See also Figure 8The embodiment of the present application further provides a data persistence estimation device, which can implement the above-mentioned data persistence estimation method, and the data persistence estimation device includes:

[0143] A first acquisition module 810 is used to acquire an original data stream within a preset time window, where the original data stream includes multiple original data items;

[0144] A deduplication processing module 820 is used to perform deduplication processing on each original data item through a burst filter to obtain multiple target data items;

[0145] The second acquisition module 830 is used to acquire the heat category of each target data item through a cold filter; the heat category includes cold data, warm data or hot data;

[0146] A first estimation module 840 is configured to estimate the persistence value of the target data item through a cold filter if the heat category indicates that the target data item is cold data or warm data;

[0147] The second estimation module 850 is configured to estimate the persistence value of the target data item through a thermal storage if the heat category indicates that the target data item is hot data.

[0148] The specific implementation of the data persistence estimation device is substantially the same as the specific implementation of the above-mentioned data persistence estimation method, and will not be described in detail herein.

[0149] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above data persistence estimation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0150] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0151] The processor 910 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0152] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes the data persistence estimation method of the embodiment of the present application;

[0153] Input / output interface 930, used to implement information input and output;

[0154] Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0155] bus 950 , which transmits information between various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0156] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0157] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data persistence estimation method is implemented.

[0158] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0159] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0160] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0161] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0163] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0164] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0165] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0166] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0169] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A data persistence estimation method, characterized in that: The method comprises: Acquire an original data stream within a preset time window, wherein the original data stream includes a plurality of original data items; Deduplication processing is performed on each of the original data items through a burst filter to obtain multiple target data items; For each of the target data items, obtaining a heat category of the target data item through a cold filter; the heat category includes cold data, warm data or hot data; If the heat category indicates that the target data item is cold data or warm data, estimating the persistence value of the target data item through the cold filter; If the heat category indicates that the target data item is hot data, estimating the persistence value of the target data item through a hot memory; The cold filter includes a primary storage structure, the primary storage structure is provided with a first hash function and a first persistence threshold, and the step of obtaining the heat category of the target data item through the cold filter includes: Performing hash calculation on the target data item according to the first hash function to obtain a first hash value; obtaining a first storage location of the target data item in the primary storage structure according to the first hash value; obtaining a minimum persistence value of the first storage location to obtain a first initial count; obtaining a heat category of the target data item according to the first initial count and the first persistence threshold; The cold filter further includes a secondary storage structure, the secondary storage structure is provided with a second hash function and a second persistence threshold, and acquiring the heat category of the target data item according to the first initial count and the first persistence threshold includes: If the first initial count is less than the first persistence threshold, the heat category of the target data item is determined to be cold data; if the first initial count is greater than or equal to the first persistence threshold, the target data item is hashed according to the second hash function to obtain a second hash value, a second storage location of the target data item in the secondary storage structure is obtained according to the second hash value, a minimum persistence value of the second storage location is obtained, and a second initial count is obtained; if the second initial count is less than the second persistence threshold, the heat category of the target data item is determined to be warm data; if the second initial count is greater than or equal to the second persistence threshold, the heat category of the target data item is determined to be hot data; After estimating the persistence value of the target data item by using a thermal memory if the heat category indicates that the target data item is hot data, the data persistence estimation method further includes: Obtain a query request, the query request including a reference data item; in response to the query request, query the persistence value of the reference data item from the cold filter to obtain a first persistent value; query the persistence value of the reference data item from the hot memory to obtain a second persistent value; and sum the first persistent value and the second persistent value to obtain a target persistent value of the reference data item.

2. The method according to claim 1, characterized in that: The primary storage structure is provided with a small counter, the secondary storage structure is provided with a large counter, and if the heat category indicates that the target data item is cold data or warm data, estimating the persistence value of the target data item through the cold filter includes: If the heat category indicates that the target data item is cold data, the first initial count is updated by the small counter to obtain a persistence value of the target data item; If the heat category indicates that the target data item is warm data, the second initial count is updated by the large counter to obtain a persistence value of the target data item.

3. The method according to any one of claims 1 to 2, characterized in that: The hot memory includes a third hash function and a plurality of storage units, and if the heat category indicates that the target data item is hot data, estimating the persistence value of the target data item through the hot memory includes: If the heat category indicates that the target data item is hot data, performing a hash calculation on the target data item according to the third hash function to obtain a third hash value; Screening the plurality of storage units according to the third Hash value to obtain a target unit; A persistence value of the target data item is estimated based on the target unit.

4. The method according to claim 3, characterized in that The estimating the persistence value of the target data item according to the target unit comprises: If the target unit stores the target data item, then obtaining the original persistent value of the target data item, updating the original persistent value of the target data item, and obtaining the persistent value of the target data item; If the target unit does not store the target data item and there is remaining storage space in the target unit, the target data item is stored in the target unit, and the target data item is counted to obtain a persistence value of the target data item; If the target unit does not store the target data item and there is no remaining storage space in the target unit, the data item with the smallest original persistence value is filtered out from the target unit to obtain a candidate data item, the candidate data item is replaced with the target data item, and the original persistence value of the candidate data item is updated to obtain the persistence value of the target data item.

5. A data persistence estimation device, characterized in that: The device comprises: A first acquisition module, used to acquire an original data stream within a preset time window, wherein the original data stream includes a plurality of original data items; A deduplication processing module, used for performing deduplication processing on each of the original data items through a burst filter to obtain multiple target data items; A second acquisition module is used to acquire, for each of the target data items, a heat category of the target data item through a cold filter; the heat category includes cold data, warm data or hot data; a first estimation module, configured to estimate the persistence value of the target data item through the cold filter if the heat category indicates that the target data item is cold data or warm data; a second estimation module for estimating a persistence value of the target data item through a thermal memory if the heat category indicates that the target data item is hot data; The cold filter includes a primary storage structure, the primary storage structure is provided with a first hash function and a first persistence threshold, and the step of obtaining the heat category of the target data item through the cold filter includes: Performing hash calculation on the target data item according to the first hash function to obtain a first hash value; obtaining a first storage location of the target data item in the primary storage structure according to the first hash value; obtaining a minimum persistence value of the first storage location to obtain a first initial count; obtaining a heat category of the target data item according to the first initial count and the first persistence threshold; The cold filter further includes a secondary storage structure, the secondary storage structure is provided with a second hash function and a second persistence threshold, and acquiring the heat category of the target data item according to the first initial count and the first persistence threshold includes: If the first initial count is less than the first persistence threshold, the heat category of the target data item is determined to be cold data; if the first initial count is greater than or equal to the first persistence threshold, the target data item is hashed according to the second hash function to obtain a second hash value, a second storage location of the target data item in the secondary storage structure is obtained according to the second hash value, a minimum persistence value of the second storage location is obtained, and a second initial count is obtained; if the second initial count is less than the second persistence threshold, the heat category of the target data item is determined to be warm data; if the second initial count is greater than or equal to the second persistence threshold, the heat category of the target data item is determined to be hot data; After estimating the persistence value of the target data item by using a thermal memory if the heat category indicates that the target data item is hot data, the data persistence estimation method further includes: Obtain a query request, the query request including a reference data item; in response to the query request, query the persistence value of the reference data item from the cold filter to obtain a first persistent value; query the persistence value of the reference data item from the hot memory to obtain a second persistent value; and sum the first persistent value and the second persistent value to obtain a target persistent value of the reference data item.

6. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images