Data stream cardinal number estimation method supporting element level deletion operation

Abstract

The invention relates to the technical field of computers, and particularly provides a data stream cardinal number estimation method supporting element-level deletion operation, which comprises the following steps of: in response to updating operation, updating a bucket data sequence stored in a virtual bucket; wherein the number of the virtual buckets is multiple, each virtual bucket comprises a guard unit and a plurality of conventional units, the guard unit is used for recording a minimum sampling value, and the conventional units are used for storing fingerprints; obtaining the number of non-empty units, and calculating a bucket level cardinal number of each virtual bucket according to the number of the non-empty units and the updated bucket data sequence; wherein the number of the non-empty units is the number of the conventional units, in which the fingerprints are stored, in the virtual bucket; and accumulating the bucket-level cardinal numbers of the virtual buckets, and calculating the global data cardinal number of the data stream. According to the method, the target data is dynamically updated and the global data cardinal number of the data stream is calculated based on the virtual bucket, the memory utilization rate is high, and calculation is fast and accurate.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data stream cardinality estimation method supporting element-level deletion operations. Background Art

[0002] Data stream cardinality refers to the number of different data elements in a data stream. In database management, network measurement, and security systems, data stream cardinality estimation is a fundamental task that can provide key support for data processing and resource management, and data stream cardinality estimation is widely used in real-time data analysis. For example, a database system can monitor the number of different queries, transactions, and column values ​​to dynamically adjust indexing strategies and resource allocation, thereby effectively coping with high-concurrency dynamic loads. In network traffic monitoring, cardinality estimation is used to track the uniqueness of IP addresses and connections to identify abnormal behavior. In the field of network security, cardinality estimation technology is used to analyze data packets in intrusion detection systems (IDS) in real time, detect abnormal activities in a timely manner, and ensure system stability.

[0003] Traditional data stream cardinality estimation methods include PCSA, LogLog, and HyperLogLog. Although these traditional data stream cardinality estimation methods have shown great advantages in insertion operations and memory efficiency, they are unable to cope with the need to delete data elements, which limits the adaptability of these methods in dynamic application scenarios. There is also a hash table-based "right-to-be-forgotten" data flow model (RFDS for short), which can support manual deletion operations on specific data elements, but the RFDS processing speed shows obvious deficiencies in complexity and memory utilization. Summary of the invention

[0004] An embodiment of the present invention provides a data stream cardinality estimation method that supports element-level deletion operations, which at least solves the problem that existing data stream cardinality estimation methods cannot achieve fast and accurate data management in a dynamic environment. It dynamically updates the target data based on virtual buckets and calculates the global data cardinality of the data stream, has high memory utilization, and fast and accurate calculation.

[0005] The present invention provides a data stream cardinality estimation method supporting element-level deletion operations, comprising updating a bucket data sequence stored in a virtual bucket in response to an update operation; wherein, a plurality of virtual buckets are provided, each of the virtual buckets comprises a guard unit and a plurality of regular units, the guard unit is used to record a minimum sampling value, and the regular unit is used to store a fingerprint; the number of non-empty units is obtained, and the bucket-level cardinality of each virtual bucket is calculated according to the number of non-empty units and the updated bucket data sequence; wherein, the number of non-empty units is the number of regular units storing the fingerprint in the virtual bucket; the bucket-level cardinality of each virtual bucket is accumulated to calculate the global data cardinality of the data stream.

[0006] In one embodiment of the present invention, the update operation includes an insert operation; in response to the insert operation, updating the bucket data sequence stored in the virtual bucket, including: processing the data element to be inserted according to the bucket index hash function to obtain a first bucket index; wherein the data element to be inserted is the data element to be inserted by the insert operation; confirming a first target bucket according to the first bucket index; wherein the first target bucket is the virtual bucket corresponding to the first bucket index; processing the data element to be inserted according to the fingerprint hash function to obtain a first fingerprint; judging whether the first fingerprint meets a sampling condition; wherein, when the sampling condition is met, the first fingerprint is not stored in a first regular unit, and the first fingerprint is less than a first target value; the first regular unit belongs to the first target bucket, and the first target value is the current minimum sampling value of the first target bucket; if the first fingerprint meets the sampling condition, obtaining a second target value, and judging whether the first fingerprint is less than the second target value; wherein the second target value is the current maximum fingerprint value of the first target bucket; if the first fingerprint is not less than the second target value, updating the first target value to the second target value, and terminating the current insert operation; if the first fingerprint does not meet the sampling condition, terminating the current insert operation.

[0007] In one embodiment of the present invention, when the first fingerprint satisfies the sampling condition: if the first fingerprint is less than the second target value, it is determined whether the target regular unit is a null value; wherein the target regular unit is the regular unit corresponding to the second target value, the null value is the maximum value of the data type, and when the target regular unit is the null value, the target regular unit does not store the fingerprint; if the target regular unit is the null value, the first fingerprint is inserted into the target regular unit, and the current insertion operation is terminated; if the target regular unit is not the null value, the first target value is updated to the second target value; after updating the first target value, the second target value is updated to the first fingerprint, and the current insertion operation is terminated.

[0008] In one embodiment of the present invention, before determining whether the first fingerprint satisfies the sampling conditions, it also includes preprocessing the first target bucket; preprocessing the first target bucket includes: obtaining additional attributes to be inserted; the additional attributes to be inserted are additional attributes of the data element to be inserted; verifying the second regular unit according to a preset strategy function to confirm whether the second regular unit stores an expired fingerprint; wherein the second regular unit belongs to the first target bucket; the expired fingerprint is a fingerprint that has not been verified by the preset strategy function; if the second regular unit stores the expired fingerprint, the second regular unit is set to a null value, the expired additional attribute is updated to the additional attribute to be inserted, and the current preprocessing is terminated; wherein, when the second regular unit is the null value, the second regular unit does not store the fingerprint; the expired additional attribute is the additional attribute of the second regular unit; if the second regular unit does not store the expired fingerprint, the current preprocessing is terminated.

[0009] In one embodiment of the present invention, before verifying the first target bucket according to the preset strategy function, it also includes: determining whether the additional attribute to be inserted is empty; if the additional attribute to be inserted is empty, then proceeding to determining whether the first fingerprint meets the sampling condition; if the additional attribute to be inserted is not empty, then proceeding to verifying the first target bucket according to the preset strategy function.

[0010] In one embodiment of the present invention, the additional attribute to be inserted includes a timestamp; the preset policy function is set to a time policy verification function, and the time policy verification function It is expressed as: , in, The value of Indicates that the fingerprint of the current second conventional unit has passed verification, The value of Indicates that the fingerprint of the current second conventional unit has not passed verification; is the current timestamp; the fingerprint stored for the current second conventional unit, is a historical timestamp, and the historical timestamp is the timestamp of the current second regular unit; The preset time window.

[0011] In one embodiment of the present invention, the additional attributes to be inserted include a timestamp and a spatial region; the preset strategy function is set to a spatiotemporal strategy verification function, and the spatiotemporal strategy verification function It is expressed as: , in, The value of Indicates that the fingerprint of the current second conventional unit has passed verification, The value of Indicates that the fingerprint of the current second conventional unit has not passed verification; is the current timestamp; the fingerprint stored for the current second conventional unit, is a historical timestamp, and the historical timestamp is the timestamp of the current second regular unit; is the preset time window; is a historical space area, and the historical space area is the space area of ​​the current second conventional unit; It is the preset space area.

[0012] In one embodiment of the present invention, the update operation includes a passive deletion operation; in response to the passive deletion operation, the bucket data sequence stored in the virtual bucket is updated, including: processing the data element to be deleted according to the bucket index hash function to obtain a second bucket index; wherein the data element to be deleted is the data element to be deleted by the passive deletion operation; confirming the second target bucket according to the second bucket index; wherein the second target bucket is the virtual bucket corresponding to the second bucket index; processing the data element to be deleted according to the fingerprint hash function to obtain a second fingerprint; judging whether the second fingerprint meets the deletion condition; wherein the deletion condition is that the second fingerprint is stored in a third regular unit, and the third regular unit belongs to the second target bucket; if the deletion condition is met, the third regular unit is set to a null value, and the current passive deletion operation is terminated; when the third regular unit is the null value, the third regular unit does not store the fingerprint; if the deletion condition is not met, the current passive deletion operation is terminated.

[0013] In one embodiment of the present invention, the bucket level cardinality of each virtual bucket is calculated according to the number of non-empty cells and the updated bucket data sequence. , expressed as: , in, is the bit width of the fingerprint, is the minimum sampling value of the current virtual bucket, is the number of non-empty units in the current virtual bucket, is the number of the regular units of the virtual bucket, , , is the number of the virtual buckets; The bucket-level cardinality of each virtual bucket is accumulated to calculate the global data cardinality of the data flow , expressed as: .

[0014] In one embodiment of the present invention, before responding to the update operation, it also includes: setting the number of virtual buckets, and the number of regular units of each virtual bucket; wherein the number of regular units of each virtual bucket is equal; setting each regular unit to a null value; wherein the null value is the maximum value of the data type; setting the value of each guard unit to an initial sampling value; wherein the initial sampling value is the maximum value of the data type; defining a bucket index hash function and a fingerprint hash function; wherein the bucket index hash function is used to obtain the bucket index corresponding to the data element, and the fingerprint hash function is used to obtain the fingerprint corresponding to the data element.

[0015] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0016] The data stream cardinality estimation method supporting element-level deletion operations described in the present invention calculates the bucket-level cardinality of each virtual bucket based on the bucket data sequence and the number of non-empty cells after the virtual bucket is updated, thereby calculating the global data cardinality of the data stream based on the bucket-level cardinality, with high overall memory utilization and fast and accurate calculation. Efficient data element management can be achieved under conditions of limited memory conditions. In addition, it can support data element insertion and deletion operations with O(1) time complexity, effectively meeting the data management requirements in a dynamic environment, and ensuring real-time and user-specific data management requirements. The data stream cardinality estimation method supporting element-level deletion operations described in the present invention has good scalability and adaptability, and provides an efficient and reliable solution for cardinality estimation in a dynamic data environment, and is particularly suitable for a variety of dynamic data processing scenarios such as real-time stream processing, high-concurrency database management, and security monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work. In the drawings:

[0018] Figure 1 It is one of the flow charts of the data stream cardinality estimation method supporting element-level deletion operation in the preferred embodiment of the present invention.

[0019] Figure 2 This is the second flow chart of the data stream cardinality estimation method supporting element-level deletion operations in the preferred embodiment of the present invention. DETAILED DESCRIPTION

[0020] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0021] It should be noted that traditional data stream cardinality estimation methods, such as PCSA, LogLog, and HyperLogLog, can only perform insertion operations and cannot cope with the need to delete data elements. Therefore, the adaptability of these methods in dynamic application scenarios is limited, especially in systems that require continuous updates, such as real-time streaming media monitoring and user behavior tracking. The fundamental limitation is that traditional data stream cardinality estimation methods map data elements to statistical indicators in the storage pool. Once inserted, the data element no longer has an independent identity in the shared memory pool. This design makes it difficult to distinguish and delete individual data elements.

[0022] RFDS supports manual deletion of specific data elements, allowing users to request deletion of their historical data from the model to meet privacy protection requirements. However, RFDS uses hash tables to store sampled data elements, which makes its performance unstable under large data volumes, especially in programmable hardware (such as FPGA) or systems with high concurrency requirements. The operational complexity of RFDS, O(q), is related to the size of the hash table, q. Therefore, the processing speed of RFDS shows obvious deficiencies in complexity and memory utilization.

[0023] Existing cardinality estimation methods lack the ability to support efficient data deletion with O(1) time complexity, which has become a bottleneck for achieving fast and accurate data management in dynamic environments.

[0024] To solve the above problems, refer to Figure 1 As shown, the present invention provides a data stream cardinality estimation method supporting element-level deletion operations, comprising:

[0025] In response to the update operation, the bucket data sequence stored in the virtual bucket is updated.

[0026] Get the number of non-empty cells, and calculate the bucket-level cardinality of each virtual bucket based on the number of non-empty cells and the updated bucket data sequence.

[0027] The bucket-level cardinality of each virtual bucket is accumulated to calculate the global data cardinality of the data stream.

[0028] It should be noted that in the data stream cardinality estimation method supporting element-level deletion operations described in the present invention, the virtual bucket array structure is used to store data elements when performing data stream cardinality estimation, can support user insertion and deletion operations, has sublinear memory consumption, and supports deployment through programmable hardware acceleration, ensuring high adaptability in various application scenarios.

[0029] The virtual bucket array structure includes Virtual buckets, each of which includes a guard unit and a regular unit. Preferably, in each virtual bucket, only one guard unit is provided, and one regular unit is provided. This design allows each virtual bucket to perform adaptive sampling and deletion operations independently, which can achieve efficient and flexible data management, has high space utilization, and meets the needs of data stream cardinality estimation. To facilitate the identification of each virtual bucket, the virtual bucket is recorded as , .

[0030] The guard unit is used to store fingerprints, which correspond to the virtual bucket corresponding to the guard unit. The minimum sampling value of , in order to adjust the sampling rate. To facilitate the distinction between each virtual bucket The minimum sampling value of the virtual bucket The corresponding minimum sampling value is recorded as In the process of cardinality estimation, the minimum sampling value Dynamically update and give priority to recording the fingerprints corresponding to new data elements that are smaller than the current sample value. Therefore, in the process of processing each update operation data element, the memory utilization rate is high.

[0031] Each regular unit is used to store a fingerprint and has an optional field for additional attributes to facilitate policy-driven deletion operations. Additional attributes include timestamps, spatial regions, etc. For easy identification, the fingerprint stored in the corresponding regular unit is recorded as , During the cardinality estimation process, each regular cell is responsible for tracking the state of the corresponding data element to achieve accurate cardinality estimation. The number of non-empty cells in each virtual bucket is the number of regular cells with fingerprints stored in the virtual bucket.

[0032] After the cardinality estimation system receives the update operation data sequence, it processes each update operation data element of the update operation data sequence in turn, hashes it into a virtual bucket, generates a corresponding fingerprint, and thus updates the bucket data sequence stored in the virtual bucket.

[0033] The update operation data sequence includes the update operation data element, the processing operation type and the additional type required for the update operation data element. The processing operation type includes the insertion operation and the passive deletion operation. Additional attributes such as timestamp, spatial area, etc. are optional. The bucket data sequence stored in the virtual bucket includes the minimum sampling value stored in the guard unit, the fingerprint stored in the regular unit, and the additional attributes.

[0034] After the update operation data sequence is executed, or when the user has a query requirement, the bucket-level cardinality of the corresponding virtual bucket is calculated based on the current number of non-empty units and the minimum sampling value of each virtual bucket in the bucket data sequence.

[0035] The data stream cardinality estimation method supporting element-level deletion operations described in the present invention calculates the bucket-level cardinality of each virtual bucket based on the bucket data sequence and the number of non-empty cells after the virtual bucket is updated, thereby calculating the global data cardinality of the data stream based on the bucket-level cardinality, with high overall memory utilization and fast and accurate calculation. Efficient data element management can be achieved under conditions of limited memory conditions. In addition, it can support data element insertion and deletion operations with O(1) time complexity, effectively meeting the data management requirements in a dynamic environment, and ensuring real-time and user-specific data management requirements. The data stream cardinality estimation method supporting element-level deletion operations described in the present invention has good scalability and adaptability, and provides an efficient and reliable solution for cardinality estimation in a dynamic data environment, and is particularly suitable for a variety of dynamic data processing scenarios such as real-time stream processing, high-concurrency database management, and security monitoring.

[0036] Reference Figure 2 As shown, the data stream cardinality estimation method supporting element-level deletion operation of the present invention, in some embodiments, further includes an initialization operation before responding to the update operation to provide initial conditions for subsequent update operations. Specifically, the initialization operation includes:

[0037] First, the number of virtual buckets is set, which can provide a space partitioning basis for the subsequent insertion and deletion of data elements.

[0038] Next, set up each virtual bucket The number of conventional units It should be noted that each virtual bucket The number of regular units is equal to Through this step, it is possible to facilitate the regular unit to store the inserted data elements and ensure the dispersion and uniformity of the data.

[0039] After setting the quantity, set each regular cell to an empty value. The initial value is set to , which is the maximum value of the data type, to ensure that regular cells are empty by default before data is inserted.

[0040] Similarly, the value of each guard unit is set to the initial sampling value. Specifically, the value of each regular unit is set to The initial value is set to , which is the maximum value of the data type, to ensure that new data elements meet the sampling conditions by default, so that the system can effectively capture new data in the early stage of operation. In the subsequent cardinality estimation process, the value of the guard unit is dynamically updated.

[0041] in, is the bit width of the fingerprint, that is, the bit length of the fingerprint, that is, the number of bits in the cell. Considering that there is no concept of infinity in computers, infinity is equivalent to the maximum value of the current data type. For example, for an unsigned 8-bit data type, infinity is equivalent to the binary 11111111, that is -1. The minimum sampling value is related to the adaptive sampling rate. Specifically, the adaptive sampling rate is expressed as ,exist Set to In the case of , the adaptive sampling rate is 1. As the minimum sampling value of the guard unit decreases, the adaptive sampling rate also decreases.

[0042] After completing the setup, define the bucket index hash function and fingerprint hash function , in order to process the data elements. Among them, the bucket index hash function is used to obtain the bucket index corresponding to the data element, and the fingerprint hash function is used to obtain the fingerprint corresponding to the data element.

[0043] After the initialization is completed, the update operation data element can be processed. The processing operation types of the update operation include insert operation and passive delete operation. Considering that different update operation data elements correspond to different processing operation types, the processing operations performed on them are also different. Preferably, before processing the update operation data element, first determine and select whether to perform a passive delete operation or an insert operation based on its processing operation type to meet the user's specific data management needs.

[0044] In some embodiments, the data stream cardinality estimation method of the present invention updates the bucket data sequence stored in the virtual bucket in response to the insertion operation, including:

[0045] The data element to be inserted is processed according to the bucket index hash function to obtain the first bucket index, wherein the data element to be inserted is the data element to be inserted by the insert operation, and the first bucket index can be obtained after it is processed by the bucket index hash function.

[0046] After obtaining the first bucket index, the first target bucket into which the data element to be inserted is to be inserted can be determined according to the first bucket index, wherein the first target bucket is a virtual bucket corresponding to the first bucket index.

[0047] At the same time, the additional attributes to be inserted are obtained. The additional attributes to be inserted are additional attributes of the data element to be inserted, and the additional attributes to be inserted include timestamp, spatial area, etc. In some cases, the additional attributes to be inserted may also be empty.

[0048] After obtaining the additional attribute to be inserted, it is determined whether the additional attribute to be inserted is empty or not empty, so as to go to the corresponding step to process the data element to be inserted. Specifically, if the additional attribute to be inserted is empty, go to the adaptive sampling insertion sub-operation. If the additional attribute to be inserted is not empty, go to the policy-driven deletion sub-operation first, pre-process the first target bucket, and then perform the adaptive sampling insertion sub-operation to achieve efficient processing of the data stream.

[0049] Among them, the policy-driven deletion sub-operation is based on the preset policy function to realize automatic data cleaning, and automatically remove the data when the data meets the cleaning conditions to achieve efficient processing of the data stream.

[0050] Specifically, the policy-driven deletion sub-operation is performed, and the pre-processing of the first target bucket includes:

[0051] Verify the second regular unit according to the preset policy function to confirm whether the second regular unit stores an expired fingerprint. The second regular unit belongs to the first target bucket, that is, the second regular unit is a regular unit of the first target bucket. Preferably, the second regular unit is a non-empty regular unit, that is, during verification, each non-empty regular unit of the first target bucket is verified.

[0052] An expired fingerprint is a fingerprint that has not been verified by a preset policy function. The data element corresponding to the expired fingerprint is invalid and needs to be deleted; correspondingly, the data element corresponding to the non-expired fingerprint is valid.

[0053] If the second regular unit stores an expired fingerprint, the second regular unit is set to a null value. Specifically, the fingerprint of the second regular unit is set to , i.e. the maximum value of the data type, which realizes the deletion of expired fingerprints and releases the corresponding storage space. After completion, the second regular unit does not store fingerprints.

[0054] Preferably, while deleting the expired fingerprint, the expired additional attribute is updated to the additional attribute to be inserted, so as to facilitate the subsequent insertion of the data element to be inserted. The expired additional attribute is the additional attribute of the second regular unit. After completing the operation on the second regular unit, the current preprocessing of the first target bucket is terminated, and the adaptive sampling and insertion sub-operation is performed on the data element to be inserted.

[0055] If the second regular unit does not store expired fingerprints, it does not need to be deleted, and the current preprocessing of the first target bucket is directly terminated, and the adaptive sampling and insertion sub-operation is performed on the data element to be inserted.

[0056] In some embodiments of the data stream cardinality estimation method supporting element-level deletion operation of the present invention, the additional attribute to be inserted includes a timestamp. Correspondingly, the preset strategy function is set to a time strategy verification function, and the time strategy verification function It is expressed as:

[0057] ,

[0058] in, The value of It indicates that the fingerprint of the current second regular unit has passed the verification and does not need to be deleted at this time. The value of Indicates that the fingerprint of the current second regular unit has not passed the verification and needs to be deleted. Indicates if. It is the current timestamp. How to obtain the current timestamp will not be described in detail. The fingerprint stored for the current second regular unit, It is the historical timestamp, that is, the timestamp of the current second regular unit. It is a preset time window. When the difference between the two timestamps is not less than the preset time window, it means that the current fingerprint is expired and needs to be deleted.

[0059] In some embodiments of the data stream cardinality estimation method supporting element-level deletion operations described in the present invention, the additional attributes to be inserted include timestamps and spatial regions. The preset strategy function is set to a spatiotemporal strategy verification function. It is expressed as:

[0060] ,

[0061] in, The value of It indicates that the fingerprint of the current second regular unit has passed the verification and does not need to be deleted at this time. The value of Indicates that the fingerprint of the current second regular unit has not passed the verification and needs to be deleted. It is the historical space area, that is, the space area of ​​the current second conventional unit. The current fingerprint is valid only when the difference between the two timestamps is less than the preset time window and the historical spatial area is within the preset spatial area; otherwise, it expires and needs to be deleted.

[0062] After the policy-driven deletion sub-operation is completed, the pre-processing of the first target bucket is realized, or the additional attribute to be inserted is empty, the adaptive sampling insertion sub-operation is performed. Specifically, it includes:

[0063] The data element to be inserted is processed according to the fingerprint hash function to obtain a first fingerprint. The first fingerprint corresponds to the data element to be inserted.

[0064] After obtaining the first fingerprint, it is determined whether the first fingerprint meets the sampling condition, thereby realizing efficient processing of the data stream according to the determination result. When the sampling condition is met, the first fingerprint is not stored in the first regular unit, and the first fingerprint is less than the first target value.

[0065] The first regular unit belongs to the first target bucket, and the first target value is the current minimum sampling value of the first target bucket. Due to the initialization operation, the minimum sampling value of the first target bucket is the maximum value at the beginning, so that the new data element meets the sampling condition by default, so as to facilitate the system to effectively capture new data in the early stage of operation.

[0066] If the first fingerprint does not meet the sampling condition, the current insertion operation is terminated.

[0067] If the first fingerprint meets the sampling condition, the second target value is obtained, and it is determined whether the first fingerprint is less than the second target value. According to the comparison result of the first fingerprint and the second target value, it is determined how to update the minimum sampling value of the first target bucket. Among them, the second target value is the current maximum fingerprint value of the first target bucket.

[0068] In the case where the first fingerprint satisfies the sampling condition, if the first fingerprint is less than the second target value, it is determined whether the target regular unit is a null value. The target regular unit is a regular unit corresponding to the second target value, and the null value is the maximum value of the data type. In the case where the target regular unit is a null value, the target regular unit does not store a fingerprint. If the target regular unit is a null value and does not store a fingerprint, the first fingerprint is inserted into the target regular unit and the current insertion operation is terminated.

[0069] If the target regular cell is not a null value, the current first target value needs to be updated, and the first target value is updated to the second target value. After the first target value is updated, the second target value is updated to the first fingerprint, and the current insertion operation ends.

[0070] If the first fingerprint is not less than the second target value, the current first target value needs to be updated, the first target value is updated to the second target value, and the current insertion operation is terminated.

[0071] In some embodiments, the data stream cardinality estimation method supporting element-level deletion operation of the present invention updates the bucket data sequence stored in the virtual bucket in response to the passive deletion operation, including:

[0072] The data element to be deleted is processed according to the bucket index hash function to obtain a second bucket index. At the same time, the data element to be deleted is processed according to the fingerprint hash function to obtain a second fingerprint. The data element to be deleted is a data element to be deleted by the passive deletion operation. The second bucket index can be obtained by processing it through the bucket index hash function, and the second fingerprint can be obtained by processing it through the fingerprint hash function.

[0073] After obtaining the second bucket index, the second target bucket where the data element to be deleted is located can be determined according to the second bucket index, wherein the second target bucket is a virtual bucket corresponding to the second bucket index.

[0074] Then, it is determined whether the second fingerprint satisfies a deletion condition, wherein the deletion condition is that the second fingerprint is stored in the third regular unit, and the third regular unit belongs to the second target bucket.

[0075] If the deletion condition is met, the third regular unit is set to a null value. Specifically, the fingerprint of the third regular unit is set to , i.e., the maximum value of the data type, realizes the deletion of the fingerprint and releases the corresponding storage space, thus meeting the user's specific data management needs. After completion, the third regular unit does not store the fingerprint, and the current passive deletion operation ends. Preferably, if the third regular unit stores additional attributes, it is also cleared when deleting the fingerprint.

[0076] If the deletion conditions are not met, the deletion fails and the current passive deletion operation ends.

[0077] The data stream cardinality estimation method supporting element-level deletion operation of the present invention, in some embodiments, estimates the data stream cardinality in real time after completing the passive deletion or insertion operation. By combining the virtual bucket array structure and the dynamic update of the guard unit value, the cardinality of the data stream can be calculated quickly and accurately without directly storing all data elements, meeting the requirements of limited memory conditions and high memory utilization, and ensuring efficient data analysis and real-time monitoring capabilities.

[0078] Specifically, the bucket-level cardinality of each virtual bucket is calculated based on the number of non-empty cells and the updated bucket data sequence. , expressed as:

[0079] ,

[0080] in, is the bit width of the fingerprint, is the minimum sampling value of the current virtual bucket, is the number of non-empty units in the current virtual bucket, is the number of regular units of the virtual bucket, , , is the number of virtual buckets.

[0081] Get the bucket-level cardinality of each virtual bucket After that, it is accumulated to calculate the global data cardinality of the data stream , expressed as:

[0082] .

[0083] It should be noted that the term "including" and its variations used in the embodiments of the present invention are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative and not restrictive. Those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0084] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and corresponding operation entrances are provided for the user to choose to authorize or refuse.

[0085] The various steps described in the method implementation methods provided by the embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method implementation methods may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0086] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same or similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

[0087] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A data stream cardinality estimation method supporting element-level deletion operations, characterized in that: include: In response to the update operation, the bucket data sequence stored in the virtual bucket is updated; wherein the virtual bucket is provided with a plurality of virtual buckets, each of the virtual buckets comprises a guard unit and a plurality of regular units, the guard unit is used to record the minimum sampling value, and the regular unit is used to store fingerprints; Obtaining the number of non-empty cells, and calculating the bucket-level cardinality of each of the virtual buckets according to the number of non-empty cells and the updated bucket data sequence; wherein the number of non-empty cells is the number of the regular cells storing the fingerprint in the virtual bucket; The bucket-level cardinalities of the virtual buckets are accumulated to calculate the global data cardinality of the data flow.

2. The data stream cardinality estimation method supporting element-level deletion operation according to claim 1, characterized in that: The update operation includes an insert operation; In response to the insert operation, updating the bucket data sequence stored in the virtual bucket includes: Processing the data element to be inserted according to the bucket index hash function to obtain a first bucket index; wherein the data element to be inserted is the data element to be inserted by the insertion operation; According to the first bucket index, confirm a first target bucket; wherein the first target bucket is the virtual bucket corresponding to the first bucket index; Processing the data element to be inserted according to the fingerprint hash function to obtain a first fingerprint; Determine whether the first fingerprint satisfies a sampling condition; wherein, if the sampling condition is met, the first fingerprint is not stored in a first regular unit, and the first fingerprint is less than a first target value; the first regular unit belongs to the first target bucket, and the first target value is the current minimum sampling value of the first target bucket; If the first fingerprint meets the sampling condition, a second target value is obtained, and it is determined whether the first fingerprint is less than the second target value; wherein the second target value is the current maximum fingerprint value of the first target bucket; If the first fingerprint is not less than the second target value, the first target value is updated to the second target value, and the current insertion operation is terminated; If the first fingerprint does not meet the sampling condition, the current insertion operation is terminated.

3. The data stream cardinality estimation method supporting element-level deletion operation according to claim 2, characterized in that: When the first fingerprint satisfies the sampling condition: If the first fingerprint is less than the second target value, determining whether the target regular unit is a null value; wherein the target regular unit is the regular unit corresponding to the second target value, the null value is the maximum value of the data type, and when the target regular unit is the null value, the target regular unit does not store the fingerprint; If the target regular unit is the null value, inserting the first fingerprint into the target regular unit and ending the current inserting operation; If the target regular unit is not the null value, the first target value is updated to the second target value; after the first target value is updated, the second target value is updated to the first fingerprint, and the current insertion operation is terminated.

4. The data stream cardinality estimation method supporting element-level deletion operation according to claim 2 or 3, characterized in that: Before determining whether the first fingerprint meets the sampling condition, the method further includes preprocessing the first target bucket; Preprocessing the first target bucket includes: Acquire the additional attribute to be inserted; the additional attribute to be inserted is the additional attribute of the data element to be inserted; Verify the second regular unit according to the preset policy function to confirm whether the second regular unit stores an expired fingerprint; wherein the second regular unit belongs to the first target bucket; the expired fingerprint is a fingerprint that has not been verified by the preset policy function; If the second regular unit stores the expired fingerprint, the second regular unit is set to a null value, the expired additional attribute is updated to the additional attribute to be inserted, and the current preprocessing is terminated; wherein, when the second regular unit is the null value, the second regular unit does not store the fingerprint; the expired additional attribute is the additional attribute of the second regular unit; If the second regular unit does not store the expired fingerprint, the current preprocessing is terminated.

5. The data stream cardinality estimation method supporting element-level deletion operation according to claim 4, characterized in that: Before verifying the first target bucket according to the preset strategy function, the method further includes: Determine whether the additional attribute to be inserted is empty; If the additional attribute to be inserted is empty, proceed to determining whether the first fingerprint meets the sampling condition; If the additional attribute to be inserted is not empty, the process proceeds to verifying the first target bucket according to a preset strategy function.

6. The data stream cardinality estimation method supporting element-level deletion operation according to claim 4, characterized in that: The additional attribute to be inserted includes a timestamp; The preset strategy function is set as a time strategy verification function, and the time strategy verification function It is expressed as: ， in, The value of Indicates that the fingerprint of the current second conventional unit has passed verification, The value of Indicates that the fingerprint of the current second conventional unit has not passed verification; is the current timestamp; the fingerprint stored for the current second conventional unit, is a historical timestamp, and the historical timestamp is the timestamp of the current second regular unit; The preset time window.

7. The data stream cardinality estimation method supporting element-level deletion operation according to claim 4, characterized in that: The additional attributes to be inserted include a timestamp and a spatial region; The preset strategy function is set as a spatiotemporal strategy verification function. It is expressed as: ， in, The value of Indicates that the fingerprint of the current second conventional unit has passed verification, The value of Indicates that the fingerprint of the current second conventional unit has not passed verification; is the current timestamp; the fingerprint stored for the current second conventional unit, is a historical timestamp, and the historical timestamp is the timestamp of the current second regular unit; is the preset time window; is a historical space area, and the historical space area is the space area of ​​the current second conventional unit; It is the preset space area.

8. The data stream cardinality estimation method supporting element-level deletion operation according to claim 1, characterized in that: The update operation includes a passive deletion operation; In response to the passive deletion operation, updating the bucket data sequence stored in the virtual bucket includes: Processing the data element to be deleted according to the bucket index hash function to obtain a second bucket index; wherein the data element to be deleted is the data element to be deleted by the passive deletion operation; According to the second bucket index, confirm a second target bucket; wherein the second target bucket is the virtual bucket corresponding to the second bucket index; Process the data element to be deleted according to the fingerprint hash function to obtain a second fingerprint; Determine whether the second fingerprint satisfies a deletion condition; wherein the deletion condition is that the second fingerprint is stored in a third regular unit, and the third regular unit belongs to the second target bucket; If the deletion condition is met, the third regular unit is set to a null value, and the current passive deletion operation is terminated; when the third regular unit is the null value, the third regular unit does not store the fingerprint; If the deletion condition is not met, the current passive deletion operation is terminated.

9. The data stream cardinality estimation method supporting element-level deletion operation according to claim 1, characterized in that: Calculate the bucket-level cardinality of each virtual bucket according to the number of non-empty cells and the updated bucket data sequence , expressed as: ， in, is the bit width of the fingerprint, is the minimum sampling value of the current virtual bucket, is the number of non-empty units in the current virtual bucket, is the number of the regular units of the virtual bucket, , , is the number of the virtual buckets; The bucket-level cardinality of each virtual bucket is accumulated to calculate the global data cardinality of the data flow , expressed as: 。 10. The data stream cardinality estimation method supporting element-level deletion operation according to claim 1, characterized in that: Before responding to the update operation, it also includes: Setting the number of the virtual buckets and the number of the regular units of each of the virtual buckets; wherein the number of the regular units of each of the virtual buckets is equal; Setting each of the regular cells to a null value; wherein the null value is the maximum value of the data type; Setting the value of each guard unit to an initial sampling value; wherein the initial sampling value is the maximum value of the data type; A bucket index hash function and a fingerprint hash function are defined; wherein the bucket index hash function is used to obtain the bucket index corresponding to the data element, and the fingerprint hash function is used to obtain the fingerprint corresponding to the data element.

Images