Data management method and management device, storage medium and electronic equipment

Abstract

The invention discloses a data management method and device, a storage medium and electronic equipment, and the method comprises the steps: building a Bloom filter for a target data set in an initial state, and enabling the Bloom filter to comprise a first preset number of hash functions and a first preset number of bit arrays; in response to an insertion request of an element in the target data set, under the condition that it is determined that the Bloom filter reaches the capacity limit, the Bloom filter is switched into an inactive state, and a new Bloom filter in an active state is established, the new bloom filter in the active state comprises a second preset number of hash functions and a second preset number of bit array, and the second preset number is greater than the first preset number; and mapping the to-be-inserted element to a new Bloom filter in an active state based on the unique ID of the to-be-inserted element. According to the method, the data query efficiency can be improved, the probability that the data are wrongly marked is reduced, and the false alarm rate is reduced.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data management method, a computer-readable storage medium, an electronic device, and a data management apparatus. Background Technology

[0002] Existing control data management methods include database-based storage and modulo-based storage. Database-based storage can solve some problems, but it suffers from high costs, low search efficiency, and difficulty in controlling data balance when dealing with large and rapidly growing datasets. Storage based on standard modulo operations can handle data balance issues and offers high search efficiency, but its scalability is poor.

[0003] Currently, there is also a control data representation method based on Bloom filters, which is suitable for fast element representation and lookup based on compact storage structures. However, as the amount of data in the Bloom filter increases, more bits in the bit array may be marked as 1, which increases the false alarm rate (the rate at which the Bloom filter incorrectly reports the existence of an element that is not actually in the set), thereby reducing the accuracy of the query. Summary of the Invention

[0004] This application aims to at least partially address one of the technical problems in related technologies. To this end, the first objective of this application is to propose a data management method. In an initial state, a Bloom filter is established for a target dataset. The Bloom filter includes a first preset number of hash functions and a first preset number of bit arrays. In response to an insertion request for an element in the target dataset, if it is determined that the Bloom filter has reached its capacity limit, the Bloom filter is switched to an inactive state, and a new active Bloom filter is created. The insertion request carries the unique ID of the element to be inserted. The newly created active Bloom filter includes a second preset number of hash functions and a second preset number of bit arrays, where the second preset number is greater than the first preset number. Based on the unique ID of the element to be inserted, the element to be inserted is mapped to the newly created active Bloom filter. This improves data query efficiency and reduces the probability of data being incorrectly labeled, thereby reducing the false alarm rate.

[0005] The second objective of this application is to provide a computer-readable storage medium.

[0006] The third objective of this application is to propose an electronic device.

[0007] The fourth objective of this application is to provide a data management device.

[0008] To achieve the above objectives, a first aspect of this application proposes a data management method, the method comprising: in an initial state, establishing a Bloom filter for a target dataset, wherein the Bloom filter includes a first preset number of hash functions and a first preset number of bit arrays; in response to an insertion request for an element in the target dataset, if it is determined that the Bloom filter has reached its capacity limit, transferring the Bloom filter to an inactive state and creating a new active Bloom filter, wherein the insertion request carries a unique ID of the element to be inserted, and the newly created active Bloom filter includes a second preset number of hash functions and a second preset number of bit arrays, the second preset number being greater than the first preset number; and mapping the element to be inserted to the newly created active Bloom filter based on the unique ID of the element to be inserted.

[0009] According to the data management method of this application embodiment, in the initial state, a Bloom filter is established for the target dataset. The Bloom filter includes a first preset number of hash functions and a first preset number of bit arrays. In response to an insertion request for an element in the target dataset, if it is determined that the Bloom filter has reached its capacity limit, the Bloom filter is switched to an inactive state, and a new active Bloom filter is created. The insertion request carries the unique ID of the element to be inserted. The newly created active Bloom filter includes a second preset number of hash functions and a second preset number of bit arrays, where the second preset number is greater than the first preset number. The element to be inserted is mapped to the newly created active Bloom filter based on its unique ID. Therefore, this method can improve data query efficiency and reduce the probability of data being incorrectly labeled, thereby reducing the false alarm rate.

[0010] In addition, the data management method according to the above embodiments of this application may also have the following additional technical features: According to one embodiment of this application, each bit array has the same number of bits, and the hash function maps the element to be inserted to the corresponding bit array based on the unique ID of the element to be inserted.

[0011] According to one embodiment of this application, after mapping the element to be inserted to a newly created active Bloom filter, the method further includes: in response to a query request, obtaining the unique ID of the element to be queried carried in the query request; based on the unique ID of the element to be queried, querying the corresponding number of bits of the inactive Bloom filters in order of longest to shortest duration of inactivity, and determining whether the element to be queried exists; if the element to be queried does not exist in any of the inactive Bloom filters, querying the corresponding number of bits of the newly created active Bloom filters based on the unique ID of the element to be queried; if the element to be queried does not exist in the newly created active Bloom filters, determining that the element to be queried does not exist in any of the Bloom filters.

[0012] According to one embodiment of this application, the corresponding number of bits is determined based on the first preset number of hash functions, the sequence number corresponding to the current Bloom filter, and the number of hash functions added to the newly created active Bloom filter.

[0013] According to one embodiment of this application, the method further includes: determining the hash function lookup bits of all elements based on the unique ID of the element to be queried, wherein the total number is determined based on the first preset number of hash functions, the total number of currently existing Bloom filters, and the number of hash functions added by newly created active Bloom filters; and determining the corresponding bit bit for each Bloom filter when querying the element to be queried based on the hash function lookup bits.

[0014] According to one embodiment of this application, determining whether the element to be queried exists includes: calculating the hash value corresponding to the element to be queried based on its unique ID; determining the bit value of each bit group in the Bloom filter based on the hash value of the element to be queried; determining that the element to be queried exists in the Bloom filter when all the bit values ​​of a bit group in the Bloom filter are 1; and determining that the element to be queried does not exist in the Bloom filter when the bit values ​​are not all 1.

[0015] According to one embodiment of this application, the method further includes: verifying the actual existence of the element to be queried in an auxiliary data structure, wherein the auxiliary data structure is a hash table or a database, and the auxiliary data structure is used to store the unique ID of each element in the target dataset to verify the authenticity of the query result; if the unique ID of the element to be inserted exists in the auxiliary data structure, determining that the element to be queried exists in the Bloom filter; if the unique ID of the element to be inserted does not exist in the auxiliary data structure, determining that the element to be queried does not exist in the Bloom filter.

[0016] To achieve the above objectives, a second aspect of this application provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the data management method described above.

[0017] The computer-readable storage medium according to the embodiments of this application, by implementing the above-described data management method during execution, can improve data query efficiency and reduce the probability of data being incorrectly labeled, thereby reducing the false alarm rate.

[0018] To achieve the above objectives, an electronic device is proposed in a third aspect of this application, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the data management method described above.

[0019] The electronic device according to the embodiments of this application can improve data query efficiency and reduce the probability of data being incorrectly labeled by executing the above-described data management method, thereby reducing the false alarm rate.

[0020] To achieve the above objectives, a fourth aspect of this application provides a data management apparatus, comprising: a first establishment module, configured to establish a Bloom filter for a target dataset in an initial state, wherein the Bloom filter includes a first preset number of hash functions and a first preset number of bit arrays; a second establishment module, configured to, in response to an insertion request for an element in the target dataset, if it is determined that the Bloom filter has reached its capacity limit, transfer the Bloom filter to an inactive state and create a new active Bloom filter, wherein the insertion request carries a unique ID of the element to be inserted, and the newly created active Bloom filter includes a second preset number of hash functions and a second preset number of bit arrays, the second preset number being greater than the first preset number; and a mapping module, configured to map the element to be inserted to the newly created active Bloom filter based on the unique ID of the element to be inserted.

[0021] According to the data management apparatus of this application embodiment, a first establishment module is used to establish a Bloom filter for a target dataset in an initial state. The Bloom filter includes a first preset number of hash functions and a first preset number of bit arrays. A second establishment module is used to, in response to an insertion request from an element in the target dataset, switch the Bloom filter to an inactive state and create a new active Bloom filter when it is determined that the Bloom filter has reached its capacity limit. The insertion request carries a unique ID of the element to be inserted. The newly created active Bloom filter includes a second preset number of hash functions and a second preset number of bit arrays, where the second preset number is greater than the first preset number. A mapping module is used to map the element to be inserted to the newly created active Bloom filter based on the unique ID of the element. Therefore, this apparatus can improve data query efficiency and reduce the probability of data being incorrectly labeled, thereby reducing the false alarm rate.

[0022] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0023] Figure 1 This is a flowchart of a data management method according to an embodiment of this application.

[0024] Figure 2 This is a schematic diagram illustrating the operations performed by a Bloom filter in different states according to embodiments of this application.

[0025] Figure 3 This is a flowchart illustrating a specific example of a data management method according to this application.

[0026] Figure 4 This is a block diagram of an electronic device according to an embodiment of this application.

[0027] Figure 5 This is a block diagram of a data management device according to an embodiment of this application. Detailed Implementation

[0028] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0029] The data management method, computer-readable storage medium, electronic device, and data management apparatus proposed in this application are described below with reference to the accompanying drawings.

[0030] Figure 1This is a flowchart of a data management method according to an embodiment of this application.

[0031] like Figure 1 As shown, the data management method in this application embodiment may include the following steps: S1. In the initial state, a Bloom filter is established for the target dataset. The Bloom filter includes a first preset number of hash functions and a first preset number of bit arrays. The first preset number can be determined according to the actual situation.

[0032] S2, in response to an insertion request for an element in the target dataset, if it is determined that the Bloom filter has reached its capacity limit, the Bloom filter is switched to an inactive state and a new Bloom filter in an active state is created. The insertion request carries the unique ID of the element to be inserted. The new Bloom filter in an active state includes a second preset number of hash functions and a second preset number of bit arrays, where the second preset number is greater than the first preset number.

[0033] S3 maps the element to be inserted to a newly created, active Bloom filter based on the element's unique ID.

[0034] Specifically, in the initial state, a Bloom filter is established for the target dataset. The Bloom filter consists of a first preset number of hash functions and a first preset number of bit arrays. The initial state refers to the initial configuration of the Bloom filter during creation, including the initialization of the bit array and the setting of the hash functions. At this stage, all bits in the bit array are set to 0, indicating that no element has been inserted. The target dataset can be any type of data collection, such as a microgrid control dataset. A microgrid control dataset may include various control commands, status information, sensor readings, etc., which need to be inserted and queried quickly and efficiently to ensure the stable operation and real-time control of the microgrid. A Bloom filter is established for the target dataset to efficiently store and manage data. The Bloom filter is a space-efficient probabilistic data structure used to test whether an element is in a set. It consists of a bit array and a hash function, which can efficiently handle insertion and query operations, but may generate false alarms, i.e., incorrectly reporting the existence of a non-existent element. The bit array is the core component of the Bloom filter; it is a fixed-length array where each element is a bit, and the initial value is set to 0. The size of the bit array determines how many elements a Bloom filter can store and its false alarm rate. A hash function maps an input of arbitrary length (e.g., an element's ID) to a specific index position in a bit array. In a Bloom filter, multiple hash functions can be used to calculate multiple index positions for each element, thereby improving the uniformity of data distribution and reducing the false alarm rate. The first preset number of hash functions refers to the number of hash functions set during Bloom filter initialization; for example, the first preset number can be 3. The number of hash functions directly affects the false alarm rate and storage efficiency; the more hash functions, the lower the false alarm rate. The first preset number of bits in the bit array refers to the length of the bit array during Bloom filter initialization. The length of the bit array determines the maximum number of elements a Bloom filter can store and its false alarm rate. The longer the bit array, the lower the false alarm rate; for example, the first preset number can be 3.

[0035] For example, in the initial state, the initial parameters of the Bloom filter are set, including a hash function of k0 and a bit array of k0, and the bit array is initialized with all bits initially set to 0. The k0 hash functions map data to the indices of the bit array. Thus, the initialized Bloom filter is ready to receive data, with the bit array empty, waiting for data insertion.

[0036] When the number of elements in a target dataset (such as a microgrid control dataset) grows to a point where the current Bloom filter reaches its capacity limit, a mechanism is needed to expand the Bloom filter to continue effectively managing the dataset. This process involves marking the current Bloom filter as inactive and creating a new, larger Bloom filter to take over subsequent insertion operations. Specifically, in response to an insertion request in the target dataset, if it is determined that the Bloom filter has reached its capacity limit, the Bloom filter can be put into an inactive state, and a new active Bloom filter can be created. The insertion request carries the unique ID of the element to be inserted, and the new active Bloom filter includes a second preset number of hash functions and a second preset number of bit arrays, where the second preset number is greater than the first preset number.

[0037] In other words, an insertion request is received, containing a unique ID of the element to be inserted. This ID can be any value that can uniquely identify an element in the dataset; for example, in a microgrid, it could be a specific control command code or sensor reading. During the process of receiving insertion requests, it is checked in real-time whether the currently active Bloom filter has reached its capacity limit. The capacity limit can be based on the proportion of occupied bits in the bit array or a preset number of elements. Once it is confirmed that the current Bloom filter has reached its capacity limit, it can be marked as inactive. This means that it will no longer accept new element insertions, but it can still be used for query operations to check if an element exists in the dataset. When the current Bloom filter reaches its capacity limit, if insertion requests continue to be received, a new active Bloom filter can be created, and the parameters of the new Bloom filter can be defined. This includes a second preset number of hash functions and a second preset number of bit arrays. The second preset number is greater than the first preset number, meaning that the new filter has stronger data processing capabilities than the old filter, and more hash functions help reduce the false alarm rate, which is 0.5 times that of the previous one. s s is a positive integer, and a larger bit array provides more storage space. This means creating a new, active Bloom filter with a hash function and bit array length greater than the previous one. This makes the new Bloom filter ready to receive new data, while the old Bloom filter no longer receives new data but is still used for queries.

[0038] In one embodiment of this application, the initial false alarm rate can be calculated based on the parameters of the Bloom filter, and the attenuation rate can be selected according to actual needs. For example, the formula fp = 1 - (1 - fp0)(1 - fp1)...(1 - fpn-1) ≤ fp0 + fp1 +...fpn-1 represents the total false alarm rate fp of the Bloom filter, which is the false alarm rate in a system composed of multiple Bloom filter stages. Each Bloom filter has its own false alarm rate fpi. The total false alarm rate of all these Bloom filters combined is calculated; the total false alarm rate equals 1 minus the product of the probabilities that none of the Bloom filters produce false alarms. The formula for the false alarm rate of a single Bloom filter is: fpi = fp0 * r i This formula represents the false alarm rate fpi of the i-th Bloom filter, which is the false alarm rate fp0 of the first Bloom filter multiplied by the decay r raised to the power of i. The formula for calculating the number of hash functions is: ki = k0 + i * log2r -1 The number of hash functions ki in the i-th Bloom filter is calculated, where k0 is the initial number of hash functions, and s is the number of hash functions added with each new filter. The decay rate is defined as r = 0.5. s Here, s is the number of hash functions added by each new filter compared to the previous one. The simplified formula for calculating the number of hash functions is: ki = k0 + i * s, which provides a simpler way to calculate the number of hash functions for the i-th Bloom filter by directly adding k0 and s to i. From this formula, we can derive the formula fp = 1 - (1 - fp0)(1 - fp0 * r) ... = 1 - qPochhammer(fp0, r). Here, the false alarm rate of each Bloom filter is r times the false alarm rate of the previous Bloom filter (i.e., fpi = fp0 * ri). Here, r is a decay factor used to control the false alarm rate of each newly added Bloom filter. When the false alarm rates of all Bloom filters decrease by a factor of r, the total false alarm rate can be approximated using the Pochhammer function, i.e., qPochhammer(fp0, r). The Pochhammer inequality provides a method for estimating the total false alarm rate when multiple Bloom filters are combined, based on the complement of the product of the false alarm rates of each Bloom filter.

[0039] In this way, when designing a Bloom filter, the initial Bloom filter configuration can be calculated and determined based on parameters such as the maximum acceptable false alarm rate, the space required to store this data (i.e., the size of the bit array), and the number of hash functions to be used. Furthermore, the parameters of the Bloom filter can be flexibly adjusted according to the needs and constraints of the actual application to achieve optimal performance and space utilization efficiency. That is, if the system has high requirements for query performance, a smaller r value may be needed to reduce the overall false alarm rate, thereby improving query accuracy.

[0040] For example, suppose the initial Bloom filter (BF0) has two hash functions and two bit arrays (e.g., containing 100 bits). When BF0 reaches its capacity limit, a new Bloom filter (BF1) is created with three hash functions and three bit arrays (e.g., containing 150 bits). Through this process, the Bloom filter can dynamically scale to accommodate growing datasets while maintaining efficient query performance. Adding a new Bloom filter not only provides more storage space but also reduces the false alarm rate by increasing the number of hash functions, thereby improving the overall accuracy and reliability of the system. This mechanism is particularly suitable for applications that require processing large amounts of dynamic data, such as microgrid control systems, where the real-time nature and accuracy of the data are crucial to system performance.

[0041] After creating a new active Bloom filter, the element to be inserted can be mapped to the newly created active Bloom filter based on its unique ID. This efficiently inserts new data elements into the newly created Bloom filter, allowing these elements to be quickly queried and their existence verified. The insertion request includes a unique identifier (ID) for the element to be inserted. This ID can be any form of data, such as a number, string, hash value, or other unique identifier, determining the currently active Bloom filter. In the Bloom filter array, only the most recently created filter is active, handling new insertion and query requests. The hash value of the element's ID can be calculated using the set of hash functions configured for the active Bloom filter. Each hash function generates a hash value, which is used to determine the position in the bit array. For example, taking the modulo of each hash value with the length of the bit array yields the index position in the bit array; these index positions are the unique identifiers of the element to be inserted in the Bloom filter. At each calculated index position in the bit array, the corresponding bit is set to 1. This indicates that the element has been "inserted" into the Bloom filter. After the element is inserted, the state of the Bloom filter is updated to reflect the addition of the new element.

[0042] Suppose there is an active Bloom filter BF1 with three hash functions (h1, h2, h3) and a three-bit array (e.g., containing a total of 15 bits). If an element with ID 1004 is inserted, and the hash value is calculated using the three hash functions (5, 10, 15 respectively), then the 5th, 10th, and 15th bits of the bit array are marked as 1. Through this process, the new element is effectively mapped into the active Bloom filter. This mapping method allows the Bloom filter to perform element lookups quickly while maintaining a low false alarm rate. As new elements are continuously added, the Bloom filter can dynamically expand its capacity to adapt to the growth of the dataset, which is particularly important for processing dynamic datasets such as microgrid control datasets.

[0043] This allows for effective management of dynamic datasets, while controlling false alarm rates and improving query efficiency. Each step ensures the Bloom filter is correctly updated and expanded to accommodate ever-growing data demands.

[0044] According to one embodiment of this application, each bit array has the same number of bits, and the hash function maps the element to be inserted to the corresponding bit array based on the unique ID of the element to be inserted.

[0045] Specifically, in a Bloom filter, the bit array is a fixed-length array where each element is a bit, which can only take one of two values: 0 or 1. This bit array is used to represent the presence of elements in the dataset. In other words, the bit array is the data structure used by the Bloom filter to store information; it's a one-dimensional array where each element is a bit, and all bits in the bit array are set to 0 during Bloom filter initialization. The bit array records the presence of elements in the dataset. When an element is inserted, its hash function calculates its position in the bit array and sets the bits at those positions to 1. A hash function is an algorithm that converts input data (such as an element ID) into a fixed-size output (i.e., an index value). In a Bloom filter, the hash function maps the ID of the element to be inserted to one or more positions in the bit array. Each hash function generates a hash value, which, after modulo operation, can be converted into an index in the bit array. A Bloom filter can have multiple hash functions, which reduces hash collisions and lowers the false alarm rate.

[0046] When a new element needs to be inserted into a Bloom filter, for the element to be inserted, a hash value is calculated using each hash function in the Bloom filter. Each hash value is then modulo the length of the bit array to obtain its index position in the bit array. At each calculated index position in the bit array, the corresponding bit is set to 1, indicating that the element has been "inserted" into the Bloom filter. Suppose we have a Bloom filter with a bit array length of 10 and two hash functions h1 and h2. The ID of the element to be inserted is 1002. Hash values ​​are calculated using h1 and h2, assuming h1(1002)mod10=3 and h2(1002)mod10=7. Therefore, the bits at indices 3 and 7 in the bit array are set to 1. The updated bit array might look like this: [0,0,1,0,0,0,1,0,0,0]. In this example, element ID 1002 is mapped to indices 3 and 7 of the bit array by its hash value, and the bits at these two positions are set to 1, indicating that element ID 1002 has been "inserted" into the Bloom filter.

[0047] In this way, Bloom filters can efficiently represent the existence of a large number of elements while maintaining low storage requirements. During a query, if all corresponding bits are 1, the element is considered likely to exist in the dataset, despite the possibility of false alarms. This method is particularly suitable for scenarios requiring rapid element existence determination, such as caching, database indexing, and web crawling.

[0048] According to one embodiment of this application, after mapping the element to be inserted to a newly created active Bloom filter, the method further includes: in response to a query request, obtaining the unique ID of the element to be queried carried in the query request; based on the unique ID of the element to be queried, querying the corresponding number of bits of the inactive Bloom filters in order of longest to shortest duration of inactivity, and determining whether the element to be queried exists; if the element to be queried does not exist in any of the inactive Bloom filters, querying the corresponding number of bits of the newly created active Bloom filter based on the unique ID of the element to be queried; if the element to be queried does not exist in any of the newly created active Bloom filters, determining that the element to be queried does not exist in any of the Bloom filters.

[0049] Specifically, when a new element is inserted into a Bloom filter, it is first mapped to a newly created active Bloom filter. Then, when a query request is received, the element is first searched for in the active Bloom filters. If the element is not found in the active Bloom filters, the inactive Bloom filters are queried in order of their inactivity duration, from shortest to longest, to determine if the element exists. After mapping the element to be inserted to a newly created active Bloom filter, the index position in the bit array is calculated using the hash function of the active Bloom filter based on the unique ID of the element to be inserted, and the bits at these positions are set to 1. In response to a query request, the unique ID of the queried element carried in the query request is retrieved. The system can query inactive Bloom filters. Inactive Bloom filters are sorted from longest to shortest inception time. Starting with the first inactive Bloom filter, the system sequentially queries the bit array of each inactive Bloom filter. For each inactive Bloom filter, a hash function is used to calculate the index position, and the bits at those positions are checked. If all corresponding bits are 1 in any inactive Bloom filter, the element is considered likely to exist in that filter. If none of the inactive Bloom filters have all corresponding bits set to 1, the element is determined not to exist in any of them. In this way, the presence or absence of an element in a given Bloom filter can be determined progressively and accurately.

[0050] If the element to be queried is not found in any of the inactive Bloom filters, then from the newly created active Bloom filters, the corresponding number of bits are retrieved based on the unique ID of the element to be queried. For example, the hash function of the active Bloom filter can be used to calculate the index position in the bit array based on the unique ID of the element to be queried, and the bits at these index positions can be checked to see if they are all 1. If all corresponding bits are 1, then the element is considered to possibly exist in the active Bloom filter; if any bit is not 1, then the element is determined not to exist in the active Bloom filter.

[0051] If evidence of an element's existence is found in an active Bloom filter or any inactive Bloom filter, then the element is considered likely to exist in the entire dataset. If no evidence of an element's existence is found in any Bloom filter, then it can be determined that the element does not exist in the dataset.

[0052] This allows Bloom filters to effectively process dynamic datasets, improving query accuracy and efficiency through fast queries in active Bloom filters and more detailed queries in inactive ones. Simultaneously, by considering the inactivity duration of inactive Bloom filters, historical data can be utilized more effectively, ensuring the comprehensiveness and accuracy of query results. This mechanism is particularly suitable for applications requiring the processing of large amounts of dynamically changing data, such as microgrid control datasets, where the real-time performance and accuracy of the data are crucial to system performance.

[0053] According to one embodiment of this application, the number of bits corresponding to the number of bits is determined based on a first preset number of hash functions, the sequence number corresponding to the current Bloom filter, and the number of hash functions added to newly created active Bloom filters.

[0054] Specifically, the first preset number of hash functions, denoted as k0, is the number of hash functions set during the initialization of the Bloom filter. These hash functions are used to calculate the hash value of the element to be inserted or queried. The sequence number corresponding to the current Bloom filter refers to the number of the Bloom filter in the sequence. For example, the first Bloom filter has a sequence number of 1 (such as the first Bloom filter during initialization), the second has a sequence number of 2, and so on. The sequence number is used to determine how many hash functions to use. For example, if a Bloom filter is the first in the sequence, then it will use the first k0 hash functions. When a new Bloom filter is created, its number of hash functions is more than that of the previous Bloom filter, denoted as k0+s, where s is the number of newly added hash functions. The added hash functions help reduce the false alarm rate and allow the new Bloom filter to process more data.

[0055] Therefore, the corresponding number of bits can be determined based on the first preset number of hash functions, the sequence number corresponding to the current Bloom filter, and the number of hash functions for newly created active Bloom filters. For example, the corresponding number can be calculated using the formula: Number = k0 + (i-1)s. Here, k0 is the initial number (first preset number), i is the sequence number corresponding to the current Bloom filter, and s is the number of hash functions added to newly created active Bloom filters. That is, when an element needs to be inserted into a Bloom filter or the existence of an element needs to be checked, for the element to be inserted or queried, the hash value is calculated using the hash function set of the current Bloom filter. If the current Bloom filter is the first in the sequence, then k0 hash functions are used; if it is the second, then k0+s hash functions are used, and so on. Each hash value is modulo the length of the bit array to obtain the index position in the bit array. At each calculated index position in the bit array, the corresponding bit is set to 1 (for insertion operations). During a query, it is checked whether all the bits at the calculated index positions are 1. If all positions are 1, then the element is considered to be likely to exist in the Bloom filter; if any position is 0, then the element is definitely not in the Bloom filter.

[0056] Suppose we currently have a Bloom filter (index 1) with a bit array length of 20 and an initial number of hash functions (k0=2). Now, the initial Bloom filter is full, and a new Bloom filter (index 2) is created with 3 hash functions (k1=k0+1) (an increase of 1 hash function). Inserting element ID 1004: Calculate the hash value using the two hash functions (2 + (1-1)*1) of the first Bloom filter, assuming the resulting index positions are 3 and 7. Calculate the hash value using the three hash functions (2 + (2-1)*1) of the second Bloom filter, assuming the resulting index positions are 3, 7, and 14. Set the bits at these positions to 1 in the bit arrays of both Bloom filters. Querying element ID 1004: In the first Bloom filter, check if the bits at positions 3 and 7 are both 1. In the second Bloom filter, check if the bits at positions 3, 7, and 14 are all 1. If all the bits at all these positions are 1, then element ID1004 is considered likely to exist in the Bloom filter. In this way, the Bloom filter can dynamically expand the number of its hash functions to adapt to the growth of the dataset while maintaining the accuracy and efficiency of the query.

[0057] According to one embodiment of this application, the data management method further includes: determining the hash function lookup bits of all elements based on the unique ID of the element to be queried, wherein the total number is determined based on a first preset number of hash functions, the total number of currently existing Bloom filters, and the number of hash functions added by newly created active Bloom filters; and determining the corresponding bit bit for each Bloom filter when querying the element to be queried based on the hash function lookup bits.

[0058] Specifically, based on the unique ID of the element to be queried, the possible bit array index positions of that element in all Bloom filters can be calculated. That is, the total number of hash function lookup bits is first determined. This number is based on a first preset number of hash functions, the total number of currently existing Bloom filters, and the number of hash function increments for newly created active Bloom filters. Here, the first preset number of hash functions (k0) represents the number of hash functions used in the initial Bloom filter, and the total number of currently existing Bloom filters (n) represents the total number of Bloom filters from the first Bloom filter to the currently active Bloom filter. The number of hash function increments for newly created active Bloom filters (s) represents the number of hash functions added by each new Bloom filter compared to the previous one. The calculation formula can be expressed as kmtotal = k0 + (n-1)s, where kmtotal is the total number of hash function lookup bits to be calculated for the element to be queried.

[0059] The lookup bit of the hash function determines the corresponding bit for each Bloom filter when querying the element to be queried. That is, the calculated lookup bit of the hash function is used to determine the position of the bit array corresponding to the element to be queried in each Bloom filter. For each Bloom filter, the corresponding bit array index is calculated based on the number of its hash functions. For example, for the i-th Bloom filter, if it has k0+(i-1)s hash functions, then k0+(i-1)s hash values ​​will be calculated, and k0+(i-1)s positions in the bit array will be determined. The calculated hash values ​​can be used to search for the element to be queried in parallel across all Bloom filters. That is, since each Bloom filter shares the results of the first k0 hash functions, these results can be reused, thereby reducing redundant calculations. For subsequent Bloom filters (i.e., those not the first Bloom filter), only the hash value corresponding to the newly added hash function needs to be calculated. This design allows the system to search for the element to be queried in parallel across all Bloom filters because the lookup bit of each Bloom filter has been pre-calculated.

[0060] For example, consider three Bloom filters, each with a bit array length of 20. The first Bloom filter BF0 uses k0 = 2 hash functions. The second Bloom filter BF1 uses k0 + s = 3 hash functions, where s = 1. The third Bloom filter BF2 uses k0 + 2s = 4 hash functions. The ID of the element to be queried is 1004. For instance, the first Bloom filter BF0 uses two hash functions to calculate the hash value, assuming the calculated hash value is 2 and 5. The second Bloom filter BF1 uses three hash functions; the first two are the same as BF0 (the calculated hash value may be 2 and 5), and the added hash function calculates a hash value of, say, 9. The third Bloom filter BF2 uses four hash functions; the first three hash functions may calculate hash values ​​of 2, 5, and 9 (including the results of the first two Bloom filters), and the added hash function calculates a hash value of, say, 12.

[0061] During the query process, the first Bloom filter (BF0) is checked, directly indexing positions 2 and 5 to see if they are 1. Then, the second Bloom filter (BF1) is checked, directly indexing positions 2, 5, and 9 to see if they are 1. Finally, the third Bloom filter (BF2) is checked, directly examining the 2nd, 5th, 9th, and 12th positions in the bit array to see if they are 1. If all corresponding positions in any Bloom filter are 1, the element is considered to potentially exist. If at least one position in any Bloom filter is not 1, the element is considered to definitely not exist. This improves query efficiency and ensures query accuracy.

[0062] According to one embodiment of this application, determining whether a query element exists includes: calculating the hash value corresponding to the query element based on its unique ID; determining the bit value of each bit group in the Bloom filter based on the hash value of the query element; determining that the query element exists in the Bloom filter when all the bit values ​​in a bit group of a Bloom filter are 1; and determining that the query element does not exist in the Bloom filter when the bit values ​​are not all 1.

[0063] Specifically, to determine if a query element exists, the hash value corresponding to the element's unique ID is first calculated. This involves using a hash function to convert the element's unique ID into an index position in a bit array. Each hash function generates a hash value, which is used to determine which positions in the bit array should be checked. Based on the hash value of the query element, the bit values ​​of each bit array in the Bloom filter are determined. For example, for each Bloom filter, the number of bits to be checked is determined using the number of corresponding hash functions. The specific index position in the bit array is obtained by taking the modulo of each hash value with the bit array length, and the bit values ​​at these index positions are then checked. The bit values ​​can only be 0 or 1.

[0064] The Bloom filter process involves evaluating the values ​​of specific bits. If all bits in a given Bloom filter group are 1, the element being searched for is present in the filter. If not all bits are 1, the element is not present. In other words, if all calculated index positions in any Bloom filter are 1, the element being searched for is present. This is based on the design of the Bloom filter, where the bits calculated by the hash function are set to 1 when an element is inserted. Conversely, if any calculated index position in any Bloom filter is not 1, the element is not present.

[0065] Suppose there are three Bloom filters, each with a bit array length of 20 and hash functions of 2, 3, and 4 respectively. The element ID to be queried is 1004. Calculate the hash values: For the first Bloom filter (2 hash functions), assume the calculated hash values ​​are 2 and 5. For the second Bloom filter (3 hash functions), assume the calculated hash values ​​are 2, 5, and 9. For the third Bloom filter (4 hash functions), assume the calculated hash values ​​are 2, 5, 9, and 12. Check the bit values ​​at indices 2, 5, 9, and 12 in the bit array. If all bits at these positions are 1, then the element is considered likely to exist in the Bloom filter. If any bit is 0, then the element is considered not to exist in the Bloom filter. Through this process, the Bloom filter can quickly determine whether an element exists in a set.

[0066] According to one embodiment of this application, the data management method further includes: verifying the actual existence of the element to be queried in an auxiliary data structure, wherein the auxiliary data structure is a hash table or a database, and the auxiliary data structure is used to store the unique ID of each element in the target dataset to verify the authenticity of the query result; if the unique ID of the element to be inserted exists in the auxiliary data structure, determining that the element to be queried exists in the Bloom filter; if the unique ID of the element to be inserted does not exist in the auxiliary data structure, determining that the element to be queried does not exist in the Bloom filter.

[0067] Specifically, in data management methodologies, using Bloom filters for rapid existence checks is effective, but it can produce false alarms, incorrectly reporting the existence of an element. To improve the accuracy of query results, an auxiliary data structure, such as a hash table or database, can be introduced on top of the Bloom filter to verify the actual existence of the queried element. That is, first check the Bloom filter for the element's existence; if the Bloom filter reports that the element may exist, further verify its actual existence in the auxiliary data structure. The auxiliary data structure, such as a hash table or database, stores the unique IDs of each element in the target dataset to verify the authenticity of the query results. Specifically, if the Bloom filter reports that the element may exist, query the auxiliary data structure (hash table or database) to see if the unique ID of the queried element exists. If the ID exists in the auxiliary data structure, it is determined that the Bloom filter contains the queried element; if the ID does not exist in the auxiliary data structure, it is determined that the Bloom filter does not contain the queried element.

[0068] For example, suppose there are three Bloom filters with a bit array length of 20 and hash functions of 2, 3, and 4 respectively. The element to be queried has an ID of 1004. The calculated hash values ​​might be 2 and 5 for the first filter, 2, 5, and 9 for the second filter, and 2, 5, 9, and 12 for the third filter. Checking the bits at these positions in the bit array, assuming they are all 1, the Bloom filters report that the element may exist. The system then queries an auxiliary data structure (such as a database) to see if a record with ID 1004 exists. If a record with ID 1004 exists in the database, the verification result is true, and the element does exist in the dataset. If a record with ID 1004 does not exist in the database, the verification result is false, and the element does not exist in the dataset.

[0069] Therefore, by combining a Bloom filter and an auxiliary data structure, a fast and accurate element existence check can be achieved. The Bloom filter provides a rapid initial screening, while the auxiliary data structure ensures the authenticity of the query results. This method is particularly suitable for applications requiring high query accuracy, such as microgrid control data management, caching systems, and web crawlers. In other words, it can effectively manage large-scale datasets while ensuring query accuracy, leveraging the efficiency of the Bloom filter.

[0070] In summary, as Figure 2 As shown, multiple Bloom filters (BF0, BF1, BFn-1, ..., BFn) are illustrated, each consisting of a bit array. These filters are arranged sequentially, and each filter may contain a different number of hash functions. BFn is marked as the active Bloom filter, used to handle current element insertion and query requests. The active Bloom filter is the only filter that can accept new element insertions. BF0, BF1, and BFn-1 are marked as read-only Bloom filters, used to handle query requests. These filters no longer accept new element insertions but can be used to check if an element exists. When querying data, queries can be performed sequentially from BF0 to BFn, and the hash value of the element ID to be queried can be calculated first using all the hash functions defined in BFn. Assume that BFn has k0 + (i-1)*s hash functions, where k0 is the initial number of hash functions, s is the number of hash functions added for each new Bloom filter, and i is the current Bloom filter index, which is n in this case. Based on the calculated hash value, check if the corresponding positions in the bit array of BFn are all marked as 1. If all corresponding bits in BFn are 1, then the element may exist in the dataset. Additionally, for each inactive Bloom filter, repeat the following operation: calculate the hash value of the query element ID using the hash function defined by the Bloom filter, and check if the corresponding positions in the bit array of that Bloom filter are all marked as 1. If all corresponding positions are marked as 1 in all queried Bloom filters, then it can be determined that the element exists in the dataset. If any position in any Bloom filter is not marked as 1, then it can be determined that the element does not exist in the dataset. In this way, the existence of an element in the dataset can be determined. This method utilizes the structural characteristics of Bloom filters, improving query accuracy through multiple levels of checks.

[0071] The following is combined Figure 3 The method described in this application is used to describe the method.

[0072] As a specific example, the data management method of this application may include the following steps: S101, In the initial state, a Bloom filter is established for the target dataset, wherein the Bloom filter includes a first preset number of hash functions and a first preset number of bit arrays.

[0073] S102, in response to an insertion request for an element in the target dataset, determine whether the Bloom filter has reached its capacity limit. If yes, proceed to step S103; otherwise, proceed to step S106.

[0074] S103, the Bloom filter is switched to an inactive state and a new Bloom filter is created in an active state. The insertion request carries the unique ID of the element to be inserted. The new active Bloom filter includes a second preset number of hash functions and a second preset number of bit arrays, where the second preset number is greater than the first preset number.

[0075] S104, Map the element to be inserted to a newly created, active Bloom filter based on the element's unique ID.

[0076] S105, in response to the query request, obtain the unique ID of the element to be queried carried in the query request.

[0077] S106, based on the unique ID of the element to be queried, query the corresponding number of bits of the Bloom filter in the inactive state in order from the longest duration of the inactive state to the shortest duration.

[0078] S107, Determine if the element to be queried exists. If not, proceed to step S108; if yes, proceed to step S111.

[0079] S108, from the newly created active Bloom filter, query the corresponding number of bits based on the unique ID of the element to be queried.

[0080] S109, Determine if the element to be queried exists. If not, proceed to step S110; if yes, proceed to step S111.

[0081] S110, It is determined that the element to be queried does not exist.

[0082] S111, verify the actual existence of the element to be queried in the auxiliary data structure, where the auxiliary data structure is a hash table or database. The auxiliary data structure is used to store the unique ID of each element in the target dataset to verify the authenticity of the query result.

[0083] S112, if the auxiliary data structure contains a unique ID of the element to be inserted, determine that the Bloom filter contains the element to be queried; if the auxiliary data structure does not contain a unique ID of the element to be inserted, determine that the Bloom filter does not contain the element to be queried.

[0084] In summary, the data management method according to the embodiments of this application establishes a Bloom filter for the target dataset in the initial state. The Bloom filter includes a first preset number of hash functions and a first preset number of bit arrays. In response to an insertion request for an element in the target dataset, if the Bloom filter reaches its capacity limit, the Bloom filter is switched to an inactive state, and a new active Bloom filter is created. The insertion request carries the unique ID of the element to be inserted. The newly created active Bloom filter includes a second preset number of hash functions and a second preset number of bit arrays, where the second preset number is greater than the first preset number. The element to be inserted is mapped to the newly created active Bloom filter based on its unique ID. Therefore, this method can improve data query efficiency and reduce the probability of data being incorrectly labeled, thereby reducing the false alarm rate.

[0085] Corresponding to the above embodiments, this application also proposes a computer-readable storage medium.

[0086] The computer-readable storage medium of this application embodiment stores a program thereon, which, when executed by a processor, implements the data management method described above.

[0087] According to the computer-readable storage medium of the present application embodiment, by performing the above-described data management method, the data query efficiency can be improved and the probability of data being incorrectly labeled can be reduced, thereby reducing the false alarm rate.

[0088] Corresponding to the above embodiments, this application also proposes an electronic device.

[0089] like Figure 4 As shown, the electronic device 200 of this application embodiment may include: a memory 210, a processor 220, and a program stored in the memory 210 and executable on the processor 220. When the processor 220 executes the program, it implements the above-described data management method.

[0090] The electronic device according to the embodiments of this application can improve data query efficiency and reduce the probability of data being incorrectly labeled by executing the above-described data management method, thereby reducing the false alarm rate.

[0091] Corresponding to the above embodiments, this application also proposes a data management device.

[0092] like Figure 5As shown, the data management device 100 in this application embodiment includes: a first establishment module 110, a second establishment module 120, and a mapping module 130.

[0093] The first creation module 110 is used to create a Bloom filter for the target dataset in an initial state. The Bloom filter includes a first preset number of hash functions and a first preset number of bit arrays. The second creation module 120, in response to an insertion request from an element in the target dataset, moves the Bloom filter to an inactive state and creates a new active Bloom filter if the Bloom filter has reached its capacity limit. The insertion request carries the unique ID of the element to be inserted. The newly created active Bloom filter includes a second preset number of hash functions and a second preset number of bit arrays, where the second preset number is greater than the first preset number. The mapping module 130 maps the element to be inserted to the newly created active Bloom filter based on the unique ID of the element to be inserted.

[0094] According to one embodiment of this application, each bit array has the same number of bits, and the hash function maps the element to be inserted to the corresponding bit array based on the unique ID of the element to be inserted.

[0095] According to one embodiment of this application, the second establishment module 120 is further configured to: after mapping the element to be inserted to a newly established active Bloom filter, in response to a query request, obtain the unique ID of the element to be queried carried in the query request; based on the unique ID of the element to be queried, query the corresponding number of bits of the inactive Bloom filters in order of the longest duration of inactivity from the shortest, and determine whether the element to be queried exists; if the element to be queried does not exist in any of the inactive Bloom filters, query the corresponding number of bits of the newly established active Bloom filters based on the unique ID of the element to be queried; if the element to be queried does not exist in any of the newly established active Bloom filters, determine that the element to be queried does not exist in any of the Bloom filters.

[0096] According to one embodiment of this application, the number of bits corresponding to the number of bits is determined based on a first preset number of hash functions, the sequence number corresponding to the current Bloom filter, and the number of hash functions added to newly created active Bloom filters.

[0097] According to one embodiment of this application, the second establishment module 120 is further configured to: determine the hash function lookup bits of all elements based on the unique ID of the element to be queried, wherein the total number is determined based on a first preset number of hash functions, the total number of currently existing Bloom filters, and the number of hash functions added by newly created active Bloom filters; and determine the corresponding bit bit for each Bloom filter when querying the element to be queried based on the hash function lookup bits.

[0098] According to one embodiment of this application, to determine whether the element to be queried exists, the second establishment module 120 is specifically used to: calculate the hash value corresponding to the element to be queried based on the unique ID of the element to be queried; determine the value of the bits in each bit group of the Bloom filter based on the hash value corresponding to the element to be queried; when there is a Bloom filter where all the bits in a bit group are 1, it is determined that the element to be queried exists in the Bloom filter; when the values ​​of the bits are not all 1, it is determined that the element to be queried does not exist in the Bloom filter.

[0099] According to one embodiment of this application, the second establishment module 120 is further configured to: verify the actual existence of the element to be queried in an auxiliary data structure, wherein the auxiliary data structure is a hash table or a database, and the auxiliary data structure is used to store the unique ID of each element in the target dataset to verify the authenticity of the query result; if the unique ID of the element to be inserted exists in the auxiliary data structure, determine that the element to be queried exists in the Bloom filter; if the unique ID of the element to be inserted does not exist in the auxiliary data structure, determine that the element to be queried does not exist in the Bloom filter.

[0100] It should be noted that for details not disclosed in the data management device of this application embodiment, please refer to the details disclosed in the data management method of this application embodiment, which will not be repeated here.

[0101] According to the data management apparatus of this application embodiment, a first establishment module is used to establish a Bloom filter for a target dataset in an initial state. The Bloom filter includes a first preset number of hash functions and a first preset number of bit arrays. A second establishment module is used to, in response to an insertion request from an element in the target dataset, switch the Bloom filter to an inactive state and create a new active Bloom filter when it is determined that the Bloom filter has reached its capacity limit. The insertion request carries a unique ID of the element to be inserted. The newly created active Bloom filter includes a second preset number of hash functions and a second preset number of bit arrays, where the second preset number is greater than the first preset number. A mapping module is used to map the element to be inserted to the newly created active Bloom filter based on the unique ID of the element. Therefore, this apparatus can improve data query efficiency and reduce the probability of data being incorrectly labeled, thereby reducing the false alarm rate.

[0102] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0103] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0104] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0105] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0106] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0107] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A data management method, characterized in that, The method includes: In the initial state, a Bloom filter is established for the target dataset, wherein the Bloom filter includes a first preset number of hash functions and a first preset number of bit arrays; In response to an insertion request for an element in the target dataset, if it is determined that the Bloom filter has reached its capacity limit, the Bloom filter is switched to an inactive state, and a new Bloom filter in an active state is created. The insertion request carries a unique ID of the element to be inserted, and the new active Bloom filter includes a second preset number of hash functions and a second preset number of bit arrays, wherein the second preset number is greater than the first preset number. The element to be inserted is mapped to a newly created, active Bloom filter based on its unique ID.

2. The data management method according to claim 1, characterized in that, Each bit array contains the same number of bits, and the hash function maps the element to be inserted to the corresponding bit array based on the unique ID of the element to be inserted.

3. The data management method according to claim 1, characterized in that, After mapping the element to be inserted to the newly created, active Bloom filter, the method further includes: In response to a query request, obtain the unique ID of the element to be queried carried in the query request; Based on the unique ID of the element to be queried, the corresponding number of bits are queried from the longest to the shortest duration of the inactive state to the corresponding number of bits in the Bloom filter that is in an inactive state, and it is determined whether the element to be queried exists. If the element to be queried does not exist in any of the inactive Bloom filters, retrieve the corresponding number of bits from the newly created active Bloom filters based on the unique ID of the element to be queried. If the element to be queried does not exist in the newly created active Bloom filter, then it is determined that the element to be queried does not exist in any of the Bloom filters.

4. The data management method according to claim 3, characterized in that, The corresponding number of bits is determined based on the first preset number of hash functions, the sequence number corresponding to the current Bloom filter, and the number of bits added to the hash function of the newly created active Bloom filter.

5. The data management method according to claim 3, characterized in that, The method further includes: Based on the unique ID of the element to be queried, determine the hash function lookup bit of all its elements, wherein the total number is determined based on the first preset number of hash functions, the total number of currently existing Bloom filters, and the number of hash functions added by newly created active Bloom filters; The hash function is used to determine the corresponding bit when each Bloom filter queries the element to be queried.

6. The data management method according to claim 3, characterized in that, Determining whether the element to be queried exists includes: The hash value corresponding to the element to be queried is calculated based on the unique ID of the element to be queried; Based on the hash value corresponding to the element to be queried, determine the value of the bits in each bit group of the Bloom filter; If all bits in a Bloom filter are 1, then the element to be queried exists in the Bloom filter. When the values ​​of the bits are not all 1, it is determined that the element to be queried does not exist in the Bloom filter.

7. The data management method according to claim 6, characterized in that, The method further includes: The actual existence of the element to be queried is verified in an auxiliary data structure, wherein the auxiliary data structure is a hash table or a database, and the auxiliary data structure is used to store the unique ID of each element in the target dataset to verify the authenticity of the query result; If the unique ID of the element to be inserted exists in the auxiliary data structure, it is determined that the element to be queried exists in the Bloom filter; If the unique ID of the element to be inserted does not exist in the auxiliary data structure, it is determined that the element to be queried does not exist in the Bloom filter.

8. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the data management method according to any one of claims 1-7.

9. An electronic device, characterized in that, include: A memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements the data management method according to any one of claims 1-7.

10. A data management device, characterized in that, The device includes: The first establishment module is used to establish a Bloom filter for the target dataset in the initial state, wherein the Bloom filter includes a first preset number of hash functions and a first preset number of bit arrays; The second creation module is used to respond to the insertion request of an element in the target dataset, and when it is determined that the Bloom filter has reached its capacity limit, to put the Bloom filter into an inactive state and create a new Bloom filter in an active state. The insertion request carries the unique ID of the element to be inserted, and the new Bloom filter in an active state includes a second preset number of hash functions and a second preset number of bit arrays, wherein the second preset number is greater than the first preset number. The mapping module is used to map the element to be inserted to a newly created, active Bloom filter based on the unique ID of the element to be inserted.

Images