A method and apparatus for writing data using an adaptive distributed Bloom filter

Abstract

This application provides an adaptive distributed Bloom filter data writing method and apparatus. The method is based on a Bloom filter group, which includes a first Bloom filter and a second Bloom filter. First, the state of the Bloom filter group is set according to the data written to the first Bloom filter and / or the second Bloom filter. Then, the current state of the Bloom filter group is obtained, and the first Bloom filter or the second Bloom filter is initialized according to the current state and the total number of first and / or second elements in the corresponding array set of the first and / or second Bloom filters, so as to write data to the first Bloom filter and / or the second Bloom filter, while simultaneously updating and recording the total number of first and / or second elements. This application can adaptively reset the Bloom filter and delete data during data writing, realizing automatic expansion of the Bloom filter.

Description

Technical Field

[0001] This application relates to the field of computer communication, and in particular to a data writing method and apparatus for an adaptive distributed Bloom filter. Background Technology

[0002] In recent years, with the development of product business, many business scenarios require data deduplication, such as deduplication of web page URLs, spam email identification, deduplication of exposed content, identification of duplicate elements in large collections, cache penetration, etc., and Bloom filters can effectively solve this problem.

[0003] The Bloom filter, proposed by Bloom in 1970, is actually composed of a long binary vector and a series of random mapping functions. The advantages of the Bloom filter are that its space efficiency and query time are much better than general algorithms.

[0004] However, Bloom filters cannot directly delete data. For example, in scenarios where exposures over the past 7 days are required to be unique, Bloom filters cannot be used directly because the storage capacity of the Bloom filter will keep growing if the data is not deleted. Therefore, an adaptive Bloom filter is needed to solve the problem of the Bloom filter's continuously growing capacity. Summary of the Invention

[0005] In view of this, this application provides a data writing method and apparatus for an adaptive distributed Bloom filter.

[0006] This application provides a data writing method for an adaptive distributed Bloom filter, implemented based on a Bloom filter group, wherein the Bloom filter group includes a first Bloom filter and a second Bloom filter; the method includes:

[0007] The state of the Bloom filter group is set according to the data written in the first Bloom filter and / or the second Bloom filter.

[0008] Obtain the current state of the Bloom filter group, and initialize the first Bloom filter or the second Bloom filter according to the current state and the total number of first elements and / or second elements in the corresponding array set of the first Bloom filter and / or the second Bloom filter, so as to write data into the first Bloom filter and / or the second Bloom filter, and update and record the total number of first elements and / or the total number of second elements.

[0009] Further, setting the state of the Bloom filter group based on the data written to the first Bloom filter and / or the second Bloom filter specifically includes: setting the state of the Bloom filter group to 0 when data is initially written to the first Bloom filter; setting the state of the Bloom filter group to 1 when data is being written to the first Bloom filter and the total amount of data written is less than half the capacity of the first Bloom filter; setting the state of the Bloom filter group to 2 when data is being written to both the first and second Bloom filters simultaneously; setting the state of the Bloom filter group to 3 when data is only written to the second Bloom filter and the total amount of data written is less than half the capacity of the second Bloom filter; and setting the state of the Bloom filter group to 4 when data is only written to the second Bloom filter and half of its capacity has already been written.

[0010] Furthermore, the initialization of the first Bloom filter or the second Bloom filter specifically includes: firstly, obtaining the pre-configured capacity; then, calculating the length of the array corresponding to the first Bloom filter or the second Bloom filter and the number of optimal hash functions based on the pre-configured capacity and the expected false positive rate; finally, initializing the first Bloom filter or the second Bloom filter based on the length of the array and the number of optimal hash functions to obtain the capacity of the first Bloom filter or the second Bloom filter, and recording the first start time or the second start time of writing data in the first Bloom filter or the second Bloom filter.

[0011] Furthermore, obtaining the pre-configured capacity also includes: calculating the pre-configured capacity using an expansion formula based on the first start time or second start time of the data written in the first Bloom filter or the second Bloom filter and the current time.

[0012] Furthermore, the expression for the expansion formula is:

[0013]

[0014] in, Indicates the pre-configured capacity. How long after the expected data will be deleted. For the current time, C represents the start time for writing data, and C represents the total capacity of the Bloom filter.

[0015] Furthermore, the expression for the length of the array is:

[0016]

[0017] Where m is the length of the array, C This represents the pre-configured capacity, and P represents the expected false positive rate.

[0018] The expression for the number of optimal hash functions is:

[0019]

[0020] Where k is the number of optimal hash functions, and c is the number of elements inserted into the first Bloom filter.

[0021] Furthermore, the method also includes: obtaining the state of the Bloom filter group, obtaining the corresponding Bloom filter data based on the state, and calculating based on the Bloom filter group data to query whether the element to be deleted is contained in the array set corresponding to the first Bloom filter or the second Bloom filter.

[0022] Furthermore, the Bloom filter data includes the capacity of the first or second Bloom filter, the length of the array, and the number of optimal hash functions; and / or

[0023] The specific steps of querying whether the element to be deleted is contained in the array set corresponding to the first Bloom filter or the second Bloom filter are as follows: calculate and determine whether the k points mapped to the array by the element are all equal to 1 based on the Bloom filter data. If all points are equal to 1, then the element is contained in the array set corresponding to the first Bloom filter or the second Bloom filter; otherwise, the element is not contained in the array set corresponding to the first Bloom filter or the second Bloom filter.

[0024] A second aspect of this application provides a data writing apparatus for an adaptive distributed Bloom filter, comprising one or more processors for implementing the data writing method for the adaptive distributed Bloom filter described above.

[0025] A third aspect of this application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-described adaptive distributed Bloom filter data writing method.

[0026] The adaptive distributed Bloom filter data writing method and apparatus provided in this application solves the problem of continuously growing capacity caused by the inability to delete data in the Bloom filter by using two Bloom filters, a first Bloom filter and a second Bloom filter, which can be rolled over and reset. The application resets the capacity according to the actual expected expiration time and automatically expands the future capacity according to the historical data writing rate, which has a certain degree of adaptability. In this way, the Bloom filter can be adaptively reset when the data writing volume surges, so as to write data into the Bloom filter. Attached Figure Description

[0027] Figure 1 A flowchart illustrating the data writing method for the adaptive distributed Bloom filter provided in this application;

[0028] Figure 2 A schematic diagram of the data provided in this application being written into the first Bloom filter;

[0029] Figure 3 A schematic diagram of the data dual-writing Bloom filter array provided in this application;

[0030] Figure 4 A schematic diagram of the data provided in this application being written into the second Bloom filter;

[0031] Figure 5 This is a schematic diagram of a data writing device for the adaptive distributed Bloom filter provided in this application. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0033] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0034] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0035] This application provides a data writing method and apparatus for an adaptive distributed Bloom filter, which can solve the problem of the capacity of the Bloom filter continuously increasing when used directly. At the same time, the capacity can be reset according to the actual expected expiration time, and the capacity can be automatically expanded by adapting to the writing speed.

[0036] See Figure 1The adaptive distributed Bloom filter data writing method provided in this application is based on a Bloom filter group, which includes a first Bloom filter B1 and a second Bloom filter B2. Through these two Bloom filters, data can be cyclically deleted, and the capacity can be automatically expanded according to the data writing rate. This method has high adaptability and requires no manual intervention. Specifically, it includes the following steps:

[0037] (1) Set the state of the Bloom filter group according to the data written in the first Bloom filter B1 and / or the second Bloom filter B2, wherein the state of the Bloom filter group includes five states: 0, 1, 2, 3 and 4.

[0038] In this embodiment, the Bloom filter group is assigned five states based on the data being written to the first Bloom filter B1 and / or the second Bloom filter B2: when data is initially written to the first Bloom filter B1, the state of the Bloom filter group is set to 0 (status=0); when data is being written to the first Bloom filter B1 and the total amount of data written is less than half the capacity of the first Bloom filter B1, the state of the Bloom filter group is set to 1 (status=1); when data is being written to both the first Bloom filter B1 and the second Bloom filter B2 simultaneously, the state of the Bloom filter group is set to 2 (status=2); when data is only written to the second Bloom filter B2 and the total amount of data written is less than half the capacity of the second Bloom filter B2, the state of the Bloom filter group is set to 3 (status=3); and when data is only written to the second Bloom filter B2 and half of its capacity has already been written, the state of the Bloom filter group is set to 4 (status=4).

[0039] (2) Query the current state of the Bloom filter group, and perform the corresponding operation according to the state and the total number of elements NUM1 or NUM2 to write the data into the corresponding first Bloom filter B1 and / or second Bloom filter B2. At the same time, update and record the total number of elements NUM1 and / or NUM2 in the array set corresponding to the first Bloom filter and / or second Bloom filter, and delete the element to be deleted from the array set corresponding to the first Bloom filter B1 and / or second Bloom filter B2.

[0040] It should be understood that writing data into a Bloom filter means adding elements to the array set represented by the Bloom filter.

[0041] (2.1) Query the current state of the Bloom filter group and determine whether the current state is 0. If it is 0, initialize the first Bloom filter B1 to obtain the capacity C1 and the first start time startTime1 of writing data, update the state of the Bloom filter group to 1, write the data to the first Bloom filter B1, and update and record the total number of elements NUM1 in the array set corresponding to the first Bloom filter B1; otherwise, execute step (2.2).

[0042] It should be noted that Bloom filters have a certain false positive rate. Specifically, assuming the array length is m, k hash functions are used, and c is the number of elements to be inserted into the Bloom filter, then the expression for the false positive rate is:

[0043]

[0044] Where P represents the false positive rate.

[0045] In this embodiment, the initialization of the first Bloom filter B1 is specifically as follows: First, the length of the array corresponding to the first Bloom filter B1 and the number of optimal hash functions are calculated based on the pre-configured capacity and the expected false positive rate; then, the first Bloom filter B1 is initialized based on the length of the array and the number of optimal hash functions to obtain the capacity C1 of the first Bloom filter B1, and the first start time startTime1 for writing data in the first Bloom filter B1 is recorded.

[0046] It should be understood that the pre-configured capacity and expected false positive rate can be preset according to the actual situation; the length of the array corresponding to the first Bloom filter B1 corresponds to the capacity C1 of the first Bloom filter B1.

[0047] Furthermore, the length of the array can be obtained based on the pre-configured capacity and the expected false positive rate. That is, when the number of elements in the Bloom filter is known, the expected false positive rate is P, and the pre-configured capacity is C. The formula for calculating the length of a binary bit array is:

[0048]

[0049] Furthermore, the specific method for obtaining the number of optimal hash functions is as follows: If Where c is the array length, c is the number of elements to be inserted, and k is the number of hash functions. The formula for calculating k is:

[0050]

[0051] In this embodiment, data is written into the initialized first Bloom filter B1, and the total number of elements NUM1 in the array set corresponding to the first Bloom filter B1 is updated and recorded. For example, when the first data is written into the first Bloom filter B1, the original total number of elements NUM1 is 0, and the updated total number of elements NUM1 is 1. That is, when a piece of data is written into the first Bloom filter B1, the current total number of elements NUM1 is increased by 1 based on the original total number of elements NUM1. Accordingly, the total number of elements NUM1 in the array set corresponding to the first Bloom filter B1 is represented as: NUM1 = NUM1 + 1.

[0052] (2.2) Determine if the current state is 1. If it is 1, query and determine if the total number of elements NUM1 is greater than or equal to half the capacity C1 of the first Bloom filter B1. If the total number of elements NUM1 is greater than or equal to half the capacity C1 of the first Bloom filter B1, reset the second Bloom filter B2 to delete the elements to be deleted from the array set corresponding to the second Bloom filter B2. Initialize the second Bloom filter B2 to obtain the capacity C2 of the second Bloom filter B2 and the second start time startTime2 for writing data. Update the state of the Bloom filter group to 2. Write the data into the first Bloom filter B1 and the second Bloom filter B2 at the same time. Update and record the total number of elements NUM1 in the array set corresponding to the first Bloom filter B1 and the total number of elements NUM2 in the array set corresponding to the second Bloom filter B2. Otherwise, write the data into the first Bloom filter B1 and update and record the total number of elements NUM1 in the array set corresponding to the first Bloom filter B1. If it is not 1, execute step (2.3).

[0053] In this embodiment, resetting the contents of the second Bloom filter B2 includes the capacity C2 of the second Bloom filter B2, the second start time of writing data startTime2, and the total number of elements NUM2 in the corresponding array set. It should be understood that resetting the second Bloom filter B2 will delete the elements to be deleted from the array set corresponding to the second Bloom filter B2.

[0054] In this embodiment, the initialization of the second Bloom filter B2 is specifically as follows: First, the new capacity is calculated using the expansion formula based on the first start time startTime1 of the data written in the first Bloom filter B1 and the current time; then, the length of the array corresponding to the second Bloom filter B2 and the number of optimal hash functions are calculated based on the new capacity and the expected false positive rate; finally, the second Bloom filter B2 is initialized based on the length of the array and the number of optimal hash functions to obtain the capacity C2 of the second Bloom filter B2, and the second start time startTime2 of the data written in the second Bloom filter B2 is recorded.

[0055] It should be understood that the length of the array corresponding to the second Bloom filter B2 corresponds to the capacity C2 of the second Bloom filter B2.

[0056] Furthermore, the general expression for the expansion formula is:

[0057]

[0058] Among them, C This indicates the new capacity, which is the pre-configured capacity. How long after the expected data will be deleted. The current system time. The start time for writing data. This represents the total capacity of the Bloom filter. This is a constant, representing the default minimum value.

[0059] The new capacity corresponding to the initialization of the second Bloom filter B2 can be calculated by substituting the relevant parameters into the above expansion formula. The expression is as follows:

[0060]

[0061] in, C2 represents the capacity of the second Bloom filter B2, which is the calculated new capacity. How long after the expected data will be deleted. For the current time, This indicates the first start time (startTime1) for writing data into the first Bloom filter B1. C1 represents the capacity of the first Bloom filter B1.

[0062] It should be understood that the new capacity calculated using the above expansion formula can be used as the pre-configured capacity, and together with the expected false positive rate, to calculate the length of the array and the number of optimal hash functions.

[0063] Similarly, based on the new capacity, The length of the array corresponding to the second Bloom filter B2 is calculated using the array length formula, along with the expected false positive rate. Furthermore, the number of optimal hash functions corresponding to the second Bloom filter B2 is calculated using the formula for the number of optimal hash functions. Finally, based on the length of the array... and the number of optimal hash functions Initialize the second Bloom filter B2 to obtain the capacity C2 of the second Bloom filter B2, and record the second start time startTime2 for writing data into the second Bloom filter B2.

[0064] In this embodiment, when the current state of the Bloom filter group is 1, it is necessary to first query the total number of elements NUM1; then, it is determined whether the total number of elements NUM1 is greater than or equal to half the capacity C1 of the first Bloom filter B1: if the total number of elements NUM1 is less than half the capacity C1 of the first Bloom filter B1, that is... This indicates that the total number of elements (NUM1) written to the first Bloom filter B1 is less than half the capacity (C1) of the first Bloom filter B1. In this case, it is only necessary to write data into the first Bloom filter B1, and simultaneously update and record the total number of elements (NUM1) in the corresponding array set of the first Bloom filter B1, incrementing the number of written elements by one, i.e., NUM1 = NUM1 + 1. Figure 2 As shown; if the total number of elements NUM1 is greater than or equal to half the capacity C1 of the first Bloom filter B1, that is... This indicates that the total number of elements (NUM1) written to the first Bloom filter B1 is greater than or equal to half the capacity (C1) of the first Bloom filter B1. At this point, the second Bloom filter B2 needs to be reset. This removes the elements to be deleted from the array set corresponding to the second Bloom filter B2. Simultaneously, the second Bloom filter B2 is initialized to obtain its capacity (C2) and the second start time (startTime2) for writing data. After initialization, the status of the Bloom filter group is updated to 2 (status=2), indicating that data is being written to both the first Bloom filter B1 and the second Bloom filter B2 simultaneously. Double writing is performed, writing data to both the first Bloom filter B1 and the second Bloom filter B2 at the same time. The total number of elements (NUM1) in the array set corresponding to the first Bloom filter B1 and the total number of elements (NUM2) in the array set corresponding to the second Bloom filter B2 are updated and recorded, i.e., NUM1=NUM1+1, NUM2=NUM2+1. Figure 3 As shown.

[0065] (2.3) Determine if the current state is 2. If it is 2, query and determine if the total number of elements NUM1 is greater than or equal to the capacity C1 of the first Bloom filter B1. If the total number of elements NUM1 is greater than or equal to the capacity C1 of the first Bloom filter B1, update the state of the Bloom filter group to 3 and stop writing data to the first Bloom filter B1. Only write data to the second Bloom filter B2 and update and record the total number of elements NUM2 in the array set corresponding to the second Bloom filter B2. Otherwise, write data to both the first Bloom filter B1 and the second Bloom filter B2 and update and record the total number of elements NUM1 in the array set corresponding to the first Bloom filter B1 and the total number of elements NUM2 in the array set corresponding to the second Bloom filter B2. If it is not 2, execute step (2.4).

[0066] In this embodiment, when the current state of the Bloom filter group is 2, it is necessary to first query the total number of elements NUM1; then, it is determined whether the total number of elements NUM1 is greater than or equal to the capacity C1 of the first Bloom filter B1: if the total number of elements NUM1 is less than the capacity C1 of the first Bloom filter B1, that is... This indicates that the total number of elements (NUM1) written to the first Bloom filter B1 is less than the capacity (C1) of the first Bloom filter B1. Since the capacity of the first Bloom filter B1 is not full, a double write operation is performed, writing data simultaneously to both the first Bloom filter B1 and the second Bloom filter B2. At the same time, the total number of elements (NUM1) in the array set corresponding to the first Bloom filter B1 and the total number of elements (NUM2) in the array set corresponding to the second Bloom filter B2 are updated and recorded, i.e., NUM1 = NUM1 + 1, NUM2 = NUM2 + 1. Figure 3 As shown; if the total number of elements NUM1 is greater than or equal to the capacity C1 of the first Bloom filter B1, that is... This indicates that the total number of elements NUM1 written to the first Bloom filter B1 is greater than or equal to the capacity C1 of the first Bloom filter B1. The capacity of the first Bloom filter B1 is now full. The status of the Bloom filter group is updated to 3 (status=3), and writing data to the first Bloom filter B1 stops. Data is only written to the second Bloom filter B2. Simultaneously, the total number of elements NUM2 in the array set corresponding to the second Bloom filter B2 is updated and recorded, and the number of written elements is incremented by one, i.e., NUM2 = NUM2 + 1. Figure 4 As shown.

[0067] (2.4) Determine if the current state is 3. If it is 3, query and determine if the total number of elements NUM2 is greater than or equal to half the capacity C2 of the second Bloom filter B2. If the total number of elements NUM2 is greater than or equal to half the capacity C2 of the second Bloom filter B2, reset the first Bloom filter B1 to delete the elements to be deleted from the array set corresponding to the first Bloom filter B1. Initialize the first Bloom filter B1 to obtain the capacity C1 of the first Bloom filter B1 and the first start time startTime1 for writing data. Update the state of the Bloom filter group to 4. Then write the data to the second Bloom filter B2 and update and record the total number of elements NUM2 in the array set corresponding to the second Bloom filter B2. Otherwise, directly write the data to the second Bloom filter B2 and update and record the total number of elements NUM2 in the array set corresponding to the second Bloom filter B2. If it is not 3, execute step (2.5).

[0068] In this embodiment, resetting the contents of the first Bloom filter B1 includes the capacity C1 of the first Bloom filter B1, the first start time of writing data startTime1, and the total number of elements NUM1 in the corresponding array set. It should be understood that resetting the first Bloom filter B1 can delete the elements to be deleted in the array set corresponding to the first Bloom filter B1.

[0069] In this embodiment, the initialization of the first Bloom filter B1 is specifically as follows: First, the new capacity is calculated using the expansion formula based on the second start time startTime2 of the data written in the second Bloom filter B2 and the current time; then, the length of the array corresponding to the first Bloom filter B1 and the number of optimal hash functions are calculated based on the new capacity and the expected false positive rate; finally, the first Bloom filter B1 is initialized based on the length of the array and the number of optimal hash functions to obtain the capacity C1 of the first Bloom filter B1, and the first start time startTime1 of the data written in the first Bloom filter B1 is recorded.

[0070] Furthermore, by substituting the relevant parameters into the expansion formula, the new capacity corresponding to the initialization of the first Bloom filter B1 can be calculated, and its expression is:

[0071]

[0072] in, C1 represents the capacity of the first Bloom filter B1, which is the calculated new capacity. How long after the expected data will be deleted. For the current time, This indicates the second start time (startTime2) for writing data into the second Bloom filter B2. The capacity of the second Bloom filter B2 is C2.

[0073] It should be understood that the new capacity calculated using the above expansion formula can be used as the pre-configured capacity, and together with the expected false positive rate, to calculate the length of the array and the number of optimal hash functions.

[0074] Similarly, based on the new capacity, The length of the array corresponding to the first Bloom filter B1 is calculated using the array length formula, along with the expected false positive rate. Furthermore, the number of optimal hash functions corresponding to the first Bloom filter B1 is calculated using the formula for the number of optimal hash functions. Finally, based on the length of the array... and the number of optimal hash functions The first Bloom filter B1 is initialized to obtain its capacity C1, and the first start time startTime1 for writing data into the first Bloom filter B1 is recorded.

[0075] In this embodiment, when the current state of the Bloom filter group is 3, it is necessary to first query the total number of elements NUM2; then, it is determined whether the total number of elements NUM2 is greater than or equal to half the capacity C2 of the second Bloom filter B2: if the total number of elements NUM2 is less than half the capacity C2 of the second Bloom filter B2, that is... This indicates that the total number of elements NUM2 written to the second Bloom filter B2 is less than half the capacity C2 of the second Bloom filter B2. Therefore, no redundant writing is needed; the data is directly written to the second Bloom filter B2, and the total number of elements NUM2 in the corresponding array set of the second Bloom filter B2 is updated and recorded simultaneously. That is, NUM2 = NUM2 + 1. Figure 4 As shown; if the total number of elements NUM2 is greater than or equal to half the capacity C2 of the second Bloom filter B2, that is... This indicates that the total number of elements (NUM2) written to the second Bloom filter B2 is greater than or equal to half the capacity (C2) of the second Bloom filter B2. At this point, the first Bloom filter B1 is reset, thus removing the element to be deleted from the array set corresponding to the first Bloom filter B1. Simultaneously, the first Bloom filter B1 is initialized to obtain its capacity (C1) and the first start time (startTime1) for writing data. After initialization, the status of the Bloom filter group is updated to 4 (status=4). Then, data is written to the second Bloom filter B2, and the total number of elements (NUM2) in the array set corresponding to the second Bloom filter B2 is updated and recorded (NUM2=NUM2+1). Figure 4 As shown.

[0076] (2.5) Query and determine whether the total number of elements NUM2 is greater than or equal to the capacity C2 of the second Bloom filter B2. If the total number of elements NUM2 is greater than or equal to the capacity C2 of the second Bloom filter B2, stop writing data to the second Bloom filter B2, update the state of the Bloom filter group to 1, write data to the first Bloom filter B1, and update and record the total number of elements NUM1 of the first element in the array set corresponding to the first Bloom filter B1. Otherwise, write data to the second Bloom filter B2, and update and record the total number of elements NUM2 of the second element in the array set corresponding to the second Bloom filter B2.

[0077] In this embodiment, the state of the Bloom filter group is 4, indicating that data has only been written to the second Bloom filter B2 and half of its capacity C2 has been written. First, the total number of elements NUM2 needs to be queried; then, it is determined whether the total number of elements NUM2 is greater than or equal to the capacity C2 of the second Bloom filter B2: if the total number of elements NUM2 is less than the capacity C2 of the second Bloom filter B2, i.e. This indicates that the total number of elements NUM2 written to the second Bloom filter B2 is less than the capacity C2 of the second Bloom filter B2. Therefore, the single write to the second Bloom filter B2 continues, and the total number of elements NUM2 in the corresponding array set of the second Bloom filter B2 is updated and recorded, i.e., NUM2 = NUM2 + 1. Figure 4 As shown; if the total number of elements NUM2 is greater than or equal to the capacity C2 of the second Bloom filter B2, that is... This indicates that the total number of elements NUM2 written to the second Bloom filter B2 is greater than or equal to the capacity C2 of the second Bloom filter B2. At this point, writing data to the second Bloom filter B2 needs to be stopped, and the status of the Bloom filter group should be updated to 1 (status=1). Then, only data should be written to the first Bloom filter B1, and the total number of elements NUM1 in the corresponding array set of the first Bloom filter B1 should be updated and recorded (NUM1=NUM1+1). Figure 2 As shown.

[0078] In other embodiments, the method described in this application further includes querying whether the element to be deleted is contained in the array set corresponding to the first Bloom filter B1 or the second Bloom filter B2.

[0079] (3) Obtain the status of the Bloom filter group, obtain the corresponding Bloom filter data according to the status of the Bloom filter group, and calculate based on the Bloom filter data to query whether the element to be deleted is contained in the array set corresponding to the first Bloom filter B1 or the second Bloom filter B2.

[0080] The Bloom filter data includes the capacity of the first Bloom filter B1 or the second Bloom filter B2, the length of the array, and the number of optimal hash functions.

[0081] Further, querying whether an element is contained in the array set corresponding to the first Bloom filter or the second Bloom filter specifically involves: calculating and determining whether the k points mapped to the array by the element are all equal to 1 based on the Bloom filter data mentioned above. If all points are equal to 1, then the element is contained in the array set corresponding to the first Bloom filter B1 or the second Bloom filter B2; otherwise, the element is not contained in the array set corresponding to the first Bloom filter B1 or the second Bloom filter B2.

[0082] In this embodiment, since the written data dynamically switches between the first Bloom filter B1 and the second Bloom filter B2, it is necessary to first obtain the status of the Bloom filter group, and then obtain the corresponding Bloom filter data based on the status of the Bloom filter group: when the status of the Bloom filter group is 0, it returns empty, indicating that no data is being written; when the status of the Bloom filter group is 1 or 2, it returns the relevant settings of the first Bloom filter B1 to obtain the Bloom filter data, namely the capacity of the first Bloom filter B1, the length of the array, and the number of optimal hash functions. Based on this Bloom filter data, it can be determined whether the element is included in the array set corresponding to the first Bloom filter B1; when the status of the Bloom filter group is 3 or 4, it returns the relevant settings of the second Bloom filter B2 to obtain the Bloom filter data, namely the capacity of the second Bloom filter B2, the length of the array, and the number of optimal hash functions. Based on this Bloom filter data, it can be determined whether the element is included in the array set corresponding to the second Bloom filter B2.

[0083] The adaptive distributed Bloom filter data writing method and apparatus provided in this application solves the problem of continuously growing capacity caused by the inability to delete data in the Bloom filter by using two Bloom filters, a first Bloom filter and a second Bloom filter, which can be rolled over and reset. The application resets the capacity according to the actual expected expiration time and automatically expands the future capacity according to the historical data writing rate, which has a certain degree of adaptability. In this way, the Bloom filter can be adaptively reset when the data writing volume surges, so as to write data into the Bloom filter.

[0084] Corresponding to the aforementioned embodiments of the adaptive distributed Bloom filter data writing method, the present invention also provides embodiments of an adaptive distributed Bloom filter data writing apparatus.

[0085] See Figure 5 The present invention provides an adaptive distributed Bloom filter data writing device, which includes one or more processors for implementing the adaptive distributed Bloom filter data writing method in the above embodiments.

[0086] The embodiments of the adaptive distributed Bloom filter data writing device of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device reading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 5 The diagram shown illustrates a hardware structure of any data processing-capable device containing the adaptive distributed Bloom filter capacity data writing device of this invention, except... Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0087] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0088] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0089] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the adaptive distributed Bloom filter data writing method described in the above embodiments.

[0090] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0091] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data writing method for an adaptive distributed Bloom filter, characterized in that, The method is based on a Bloom filter array, which includes a first Bloom filter and a second Bloom filter; the method includes: The state of the Bloom filter group is set according to the data written in the first Bloom filter and / or the second Bloom filter. Obtain the current state of the Bloom filter group, and initialize the first Bloom filter or the second Bloom filter according to the current state and the total number of first elements and / or the total number of second elements in the corresponding array set of the first Bloom filter and / or the second Bloom filter, so as to write data into the first Bloom filter and / or the second Bloom filter, and update and record the total number of first elements and / or the total number of second elements. Specifically, when data is initially written to the first Bloom filter, the state of the Bloom filter group is set to 0; when data is being written to the first Bloom filter and the total amount of data written is less than half the capacity of the first Bloom filter, the state of the Bloom filter group is set to 1; when data is being written to both the first and second Bloom filters simultaneously, the state of the Bloom filter group is set to 2; when data is only written to the second Bloom filter and the total amount of data written is less than half the capacity of the second Bloom filter, the state of the Bloom filter group is set to 3; and when data is only written to the second Bloom filter and half of its capacity has already been written, the state of the Bloom filter group is set to 4.

2. The data writing method for an adaptive distributed Bloom filter according to claim 1, characterized in that, The initialization of the first Bloom filter or the second Bloom filter specifically includes: first, obtaining the pre-configured capacity; then, calculating the length of the array corresponding to the first Bloom filter or the second Bloom filter and the number of optimal hash functions based on the pre-configured capacity and the expected false positive rate; finally, initializing the first Bloom filter or the second Bloom filter based on the length of the array and the number of optimal hash functions to obtain the capacity of the first Bloom filter or the second Bloom filter, and recording the first start time or the second start time of writing data in the first Bloom filter or the second Bloom filter.

3. The data writing method for an adaptive distributed Bloom filter according to claim 2, characterized in that, The process of obtaining the pre-configured capacity further includes: calculating the pre-configured capacity using an expansion formula based on the first start time or second start time of the data written in the first Bloom filter or the second Bloom filter and the current time.

4. The data writing method for an adaptive distributed Bloom filter according to claim 3, characterized in that, The expression for the expansion formula is: ， in, Indicates the pre-configured capacity. How long after the expected data will be deleted. For the current time, C represents the start time for writing data, and C represents the total capacity of the Bloom filter.

5. The data writing method for an adaptive distributed Bloom filter according to claim 2, characterized in that, The expression for the length of the array is: ， Where m is the length of the array, C' represents the pre-configured capacity, and P represents the expected false positive rate; The expression for the number of optimal hash functions is: ， Where k is the number of optimal hash functions, and c is the number of elements inserted into the first Bloom filter.

6. The data writing method for an adaptive distributed Bloom filter according to claim 1, characterized in that, The method further includes: obtaining the state of the Bloom filter group, obtaining the corresponding Bloom filter data based on the state, and calculating based on the Bloom filter group data to query whether the element to be deleted is contained in the array set corresponding to the first Bloom filter or the second Bloom filter.

7. The data writing method for an adaptive distributed Bloom filter according to claim 6, characterized in that, The Bloom filter data includes the capacity of the first or second Bloom filter, the length of the array, and the number of optimal hash functions; and / or The specific steps of querying whether the element to be deleted is contained in the array set corresponding to the first Bloom filter or the second Bloom filter are as follows: calculate and determine whether the k points mapped to the array by the element are all equal to 1 based on the Bloom filter data. If all points are equal to 1, then the element is contained in the array set corresponding to the first Bloom filter or the second Bloom filter; otherwise, the element is not contained in the array set corresponding to the first Bloom filter or the second Bloom filter.

8. A data writing device for an adaptive distributed Bloom filter, characterized in that, It includes one or more processors for implementing the data writing method of the adaptive distributed Bloom filter as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the data writing method of the adaptive distributed Bloom filter as described in any one of claims 1-7.

Images