Bloom filter-based data matching method, device and equipment, and storage medium

Abstract

The application discloses a data matching method and device based on a Bloom filter, equipment and a storage medium. The method comprises the following steps: constructing a dynamic Bloom filter according to left table data; determining a current matching mode according to the switch state of the dynamic Bloom filter for current batch right table data of right table data; performing data matching on the current batch right table data and the left table data according to the current matching mode, and updating the switch state of the dynamic Bloom filter; sequentially taking the remaining data of the right table data as new current batch right table data, and continuously performing data matching on the new current batch right table data according to the updated switch state until all the right table data is matched. The method can reasonably use the dynamic Bloom filter in the data matching process by determining the current matching mode of the current batch right table data and the left table data according to the switch state of the dynamic Bloom filter, and updating the switch state after the current batch right table data is matched.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data matching method, apparatus, device and storage medium based on Bloom filters. Background Technology

[0002] A Bloom filter (BF) is a data structure used to filter data, determining that certain data is not in a specified dataset. However, the performance and overhead of a BF are affected by factors such as the actual data volume and distribution, the performance of the BF's hash function, its number, and its length. Continuously using a single-configuration BF structure can lead to increased overhead in scenarios where Bloom filters are ineffective, and frequent use can result in unbalanced performance across various data scenarios. Therefore, how to use Bloom filters effectively is a problem that current technologies need to address. Summary of the Invention

[0003] This invention provides a data matching method, apparatus, device, and storage medium based on a Bloom filter to solve the problem that existing technologies cannot reasonably use Bloom filters in different data scenarios.

[0004] According to one aspect of the present invention, a data matching method based on a Bloom filter is provided, the method comprising:

[0005] Construct a dynamic Bloom filter based on the data in the left table;

[0006] For the current batch of right table data, determine the current matching method based on the on / off state of the dynamic Bloom filter;

[0007] The data in the right table and the left table of the current batch are matched according to the current matching method, and the on / off state of the dynamic Bloom filter is updated.

[0008] The remaining data in the right table are used as the new right table data for the current batch. Data matching is then performed on the new right table data for the current batch based on the updated switch status, until all the right table data is matched.

[0009] According to another aspect of the present invention, a data matching apparatus based on a Bloom filter is provided, the apparatus comprising:

[0010] The building module is used to construct dynamic Bloom filters based on the data in the left table;

[0011] The determination module is used to determine the current matching method for the current batch of right table data based on the on / off state of the dynamic Bloom filter.

[0012] The first matching module is used to perform data matching on the right table data and left table data of the current batch according to the current matching method, and update the on / off state of the dynamic Bloom filter.

[0013] The second matching module is used to take the remaining data of the right table as the new current batch right table data in turn, and continue to match the data of the new current batch right table data according to the updated switch status, until all the right table data is matched.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: at least one processor; and

[0015] A memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the Bloom filter-based data matching method according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the Bloom filter-based data matching method according to any embodiment of the present invention.

[0018] This invention discloses a data matching method, apparatus, device, and storage medium based on a Bloom filter. The method includes: constructing a dynamic Bloom filter based on data from a left table; determining a current matching method for the current batch of right table data based on the on / off state of the dynamic Bloom filter; performing data matching on the current batch of right table data and left table data according to the current matching method, and updating the on / off state of the dynamic Bloom filter; using the remaining data from the right table data as new current batch right table data, and continuing data matching on the new current batch right table data according to the updated on / off state, until all right table data is matched. This method, by determining the current matching method for the current batch of right table data and left table data based on the on / off state of the dynamic Bloom filter, and updating the on / off state after the current batch of right table data is matched, enables the reasonable use of the dynamic Bloom filter during the data matching process, solving the problem in the prior art that Bloom filters cannot be reasonably used in different data scenarios.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a data matching method based on a Bloom filter according to Embodiment 1 of the present invention.

[0022] Figure 2 This is a flowchart illustrating a data matching method based on a Bloom filter according to Embodiment 2 of the present invention.

[0023] Figure 3 This is a schematic diagram of a data matching device based on a Bloom filter provided in Embodiment 3 of the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention. It should be understood that the various steps described in the method embodiments of the present invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0026] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0029] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0030] Example 1

[0031] Figure 1 This is a flowchart illustrating a data matching method based on a Bloom filter according to Embodiment 1 of the present invention. This method is applicable to the case of data matching and can be executed by a data matching device based on a Bloom filter. The device can be implemented by software and / or hardware and is generally integrated into an electronic device. In this embodiment, the electronic device includes, but is not limited to, devices such as computers.

[0032] like Figure 1 As shown, Embodiment 1 of the present invention provides a data matching method based on a Bloom filter, which includes the following steps:

[0033] S110. Construct a dynamic Bloom filter based on the data in the left table.

[0034] In this embodiment, the data in the left table can be data from the left table itself. During the data matching process, the two tables can be divided into a left table and a right table. The left table can be the main table for the query, and the right table can be an additional information table. This embodiment does not limit the data types or selection methods of the left and right tables. The dynamic Bloom filter can be a Bloom filter that can be dynamically expanded. A dynamic Bloom filter includes at least one Bloom filter. A Bloom filter is a data structure that uses binary vectors and multiple hash functions to determine the existence of data.

[0035] In one embodiment, before constructing the dynamic Bloom filter, a qualitative analysis can also be performed on the usage scenario of the BF based on the original BF mechanism to determine whether the use of the BF meets the requirements.

[0036] In this embodiment, the dynamic Bloom filter is constructed only when the use of the BF can reduce the matching cost, which can exclude scenarios that are definitely not beneficial. For example, an exemplary method for qualitatively analyzing the usage benefit of the BF is as follows:

[0037] Let the cost of matching a single row of data in the right table be Cost_match, and the cost of pre-filtering a single row of data in the right table by the BF be Cost_BF. Assume that the current batch of right table data initially has M rows, the right table data after BF filtering has N rows, and the right table data that is finally successfully matched has L rows.

[0038] Then the cost before optimization by the BF is recorded as:

[0039] Cost1=Cost_match*M

[0040] The cost after optimization by the BF is recorded as:

[0041] Cost2=Cost_BF*M+Cost_match*N

[0042] Cost difference:

[0043] ΔCost=Cost1–Cost2=(M-N)*Cost_match–M*Cost_BF

[0044] When ΔCost>0, using the BF at this time can reduce the overall cost of the data matching process. However, when Cost_match<Cost_BF, that is, when the cost of pre-filtering a single row of data using the BF is significantly higher than the cost of finally matching a single row of data, ΔCost<0 is definitely established, and using the BF at this time has no benefit.

[0045] When Cost_BF>(1–N / M)*Cost_match, it is considered that ΔCost<0 is generally established, and using the BF at this time may be unbeneficial, which indicates that there is an upper limit for the cost of pre-filtering a single row of data by the BF, that is, there is an upper limit for the number of hash functions used by a single Bloom filter in the dynamic Bloom filter.

[0046] In this embodiment, the dynamic Bloom filter may be constructed according to the left table data, and the information related to the left table data is recorded in the dynamic Bloom filter.

[0047] S120: For the current batch of right table data of the right table data, determine the current matching mode according to the switch state of the dynamic Bloom filter.

[0048] Here, the right table data can be any data from the right table itself. The current batch of right table data can be the right table data that needs to be processed in the current batch. The switch status can include "on" and "off," indicating whether a dynamic Bloom filter is used to filter the data. The current matching method can include matching based on a dynamic Bloom filter and direct matching. Matching based on a dynamic Bloom filter means first filtering the right table data using a dynamic Bloom filter, and then matching the filtered right table data with the left table data, thus reducing the overall cost of the right table data matching process. Direct matching means not filtering the right table data and directly matching the right table data with the left table data.

[0049] In this embodiment, the data in the right table can be processed in batches. For the right table data in the current batch that needs to be processed, the current matching method between the right table data and the left table data in the current batch can be determined according to the on / off state of the dynamic Bloom filter.

[0050] S130. Perform data matching on the right table data and left table data of the current batch according to the current matching method, and update the on / off state of the dynamic Bloom filter.

[0051] In this embodiment, after determining the current matching method, data matching can be performed on the right table data and left table data of the current batch according to the current matching method, and the on / off state of the dynamic Bloom filter can be updated.

[0052] S140. Take the remaining data of the right table as the new current batch right table data in turn, and continue to match the data of the new current batch right table data according to the updated switch status until all the right table data is matched.

[0053] In this embodiment, after matching a batch of right table data, if all right table data has been matched, the data matching can end; otherwise, the remaining batches of data are used as the current batch of right table data, and the data matching continues until all right table data is matched.

[0054] This invention provides a data matching method based on a Bloom filter, comprising: constructing a dynamic Bloom filter based on data from a left table; determining a current matching method for the current batch of right table data based on the on / off state of the dynamic Bloom filter; performing data matching on the current batch of right table data and left table data according to the current matching method, and updating the on / off state of the dynamic Bloom filter; using the remaining data from the right table data as new current batch right table data, and continuing to perform data matching on the new current batch right table data according to the updated on / off state, until all right table data is matched. This method, by determining the current matching method for the current batch of right table data and left table data based on the on / off state of the dynamic Bloom filter, and updating the on / off state after the current batch of right table data is matched, enables the reasonable use of the dynamic Bloom filter during the data matching process, solving the problem in the prior art that Bloom filters cannot be reasonably used in different data scenarios.

[0055] Based on the above embodiments, modified embodiments of the above embodiments are proposed. It should be noted that, in order to keep the description brief, only the differences from the above embodiments are described in the modified embodiments.

[0056] In one embodiment, constructing a dynamic Bloom filter based on left table data includes: determining a current Bloom filter and a current hash function; setting the current Bloom filter based on the current hash function for the current batch of left table data; determining whether the current Bloom filter meets the filter expansion condition; if so, adding a new Bloom filter and using the new Bloom filter as the current Bloom filter; processing the remaining data of the left table data as the current batch of left table data in sequence until all left table data is processed to obtain the constructed dynamic Bloom filter.

[0057] Here, the current Bloom filter can be the currently used Bloom filter. The current hash function can be the hash function used by the current Bloom filter. The type of hash function can be set according to the actual situation, and this embodiment does not limit it. The left table data of the current batch can be the left table data that needs to be processed in the current batch. The filter expansion condition can be the condition for expanding the Bloom filter. The bit setting is the data mapping process. If you want to map a piece of data into the Bloom filter, you need to use multiple different hash functions to generate multiple hash values, and set the bit in the Bloom filter pointed to by each generated hash value to 1. This embodiment will not elaborate on this.

[0058] In this embodiment, during the construction of the dynamic Bloom filter, the left table data can be processed in batches. For each batch of left table data, the current Bloom filter can be set based on the hash function of the current Bloom filter. After a batch of data is processed, it can be determined whether the filter expansion condition is met. If so, a new Bloom filter is added. When setting the next batch of left table data, the new Bloom filter is used to set the position. The above steps are repeated until all the left table data is processed, and the constructed dynamic Bloom filter is obtained.

[0059] In this embodiment, during the construction of the Bloom filter (BF), data is only set on one BF substructure, i.e., the current Bloom filter. During subsequent data matching, when filtering the data in the right table, the first constructed Bloom filter is evaluated by combining the judgments of all Bloom filters in the dynamic Bloom filter. Only data deemed filterable by the previous Bloom filter is passed to the next Bloom filter for processing. Finally, when all Bloom filters deem the data filterable, it is finally filtered.

[0060] In one embodiment, the filter expansion conditions include: the current Bloom filter's fill rate is less than or equal to a fill rate threshold, the current Bloom filter's fill change rate is less than a fill change rate threshold, the alarm count is greater than an alarm count threshold, and the number of hash functions is less than or equal to a hash function number threshold.

[0061] The fill rate is the percentage of set bits in the Bloom filter out of the total bits. The fill rate is the fill rate of the most recently added Bloom filter; that is, when there is more than one Bloom filter in a dynamic Bloom filter, only the fill rate of the most recently added Bloom filter is counted. The fill change rate is the rate of change of the Bloom filter fill rate of the current batch of data compared to the Bloom filter fill rate of the previous batch of data. When the current batch of data is the first batch, the Bloom filter fill change rate is 0. The alarm count is the number of times a Bloom filter continuously exhibits low efficiency. The number of hash functions can be the number of hash functions used by the Bloom filter. There is an upper limit to the number of hash functions that a Bloom filter can use, i.e., a hash function threshold. The hash function threshold can be set according to actual conditions; this embodiment does not limit this. The fill rate threshold and fill change rate threshold can be set according to actual conditions; different Bloom filters can have different fill rate thresholds and fill change rate thresholds. To increase the filtering effect of the Bloom filter, the fill rate threshold of a newly added Bloom filter can be lower, and the fill change rate threshold can be higher.

[0062] In this embodiment, the filter expansion conditions can be: the current Bloom filter's fill rate is less than or equal to the fill rate threshold, the current Bloom filter's fill change rate is less than the fill change rate threshold, the alarm count is greater than the alarm count threshold, and the number of hash functions is less than or equal to the number of hash functions threshold.

[0063] For example, when constructing a BF using the left table data, the BF construction process can be monitored based on the original BF mechanism to evaluate and predict the use efficiency of the BF. The left table data is processed in batches, and the left table data is divided into M batches, with N data per batch. The initial length of the BF and the number of hash functions can be set customly, where the initial length of the BF filter is set to A bits, the initial number of hash functions is X, and the upper limit is Y. The BF fill rate threshold is P_fill, the BF fill change rate threshold is P_change, and the alert count (identifying the number of consecutive low-efficiency scenarios of the BF) threshold is MAX_ALERT.

[0064] 1) First, process the first batch of the left table data, and start constructing the BF using the first batch of data. When the construction is completed, record the current BF fill rate and BF fill change rate.

[0065] 2) When the BF fill rate is greater than P_fill, it indicates that the BF no longer meets the requirements at this time, then the BF construction is aborted, and the BF will not be used for data matching subsequently.

[0066] 3) When the BF fill rate is less than P_fill, the relationship between the BF fill change rate and the BF fill change rate threshold is further judged. When the BF fill change rate is less than P_change, the alert count is incremented by 1; otherwise, the alert count is cleared to 0.

[0067] 4) If the alert count is less than MAX_ALERT, the next batch of data is continuously processed; if the alert count is greater than MAX_ALERT, it indicates that the BF has consecutively presented low-efficiency scenarios at this time, and a new BF needs to be established; then the alert count is cleared to 0. If the number of hash functions is greater than Y, it indicates that the number of usable hash functions for the BF has reached the upper limit, the efficiency of the BF is already poor at this time, then the BF construction is aborted, and the BF will not be used for data matching subsequently. Otherwise, if the number of hash functions is less than Y, one hash function and a new Bloom filter can be added for the BF, and the next batch of data is continuously processed.

[0068] 5) When the last batch of data is processed, the BF construction is completed, and the BF can be subsequently tried for data matching.

[0069] In one embodiment, the adding a new Bloom filter further comprises: adding the number of hash functions of the new Bloom filter.

[0070] In this embodiment, the newly added Bloom filter can have more hash functions, thereby improving the filtering effect of the new Bloom filter. For example, a hash function can be added to the new Bloom filter, and the type of the hash function can be set according to the actual situation. The length of the new Bloom filter can also be set according to the actual situation; for example, to avoid wasting Bloom filters, the length of the new Bloom filter can be set to be shorter.

[0071] Example 2

[0072] Figure 2 This is a flowchart illustrating a data matching method based on a Bloom filter according to Embodiment 2 of the present invention. Embodiment 2 is an optimization based on the above embodiments. For details not covered in this embodiment, please refer to Embodiment 1.

[0073] like Figure 2 As shown, Embodiment 2 of the present invention provides a data matching method based on a Bloom filter. When the current matching method is based on a dynamic Bloom filter, the method includes the following steps:

[0074] S210. Construct a dynamic Bloom filter based on the data in the left table.

[0075] S220. For the current batch of right table data, determine the current matching method based on the on / off state of the dynamic Bloom filter.

[0076] S230. Based on the dynamic Bloom filter, perform data matching on the current batch right table data and the left table data to obtain the Bloom filter rate and the true filter rate of the current batch right table data.

[0077] The Bloom filter rate represents the percentage of rows in this batch of data that were pre-filtered by the Black-Scholes filter. The True Filter rate represents the percentage of rows in this batch of data that ultimately did not match.

[0078] In this embodiment, when the current matching method is based on a dynamic Bloom filter, the data in the right and left tables of the current batch can be matched using the dynamic Bloom filter to obtain the Bloom filter rate and the true filter rate of the right table data. For example, if the right table data of the current batch has 1000 rows, and 600 rows are filtered out using BF, the current right table data of the current batch has 400 rows. Among the 400 rows, 300 rows successfully match the data in the left table. In this case, the Bloom filter rate is 600 / 1000 = 60%, and the true filter rate is 700 / 1000 = 70%.

[0079] S240. Update the on / off state of the dynamic Bloom filter based on the Bloom filtration rate and the actual filtration rate.

[0080] In this embodiment, the on / off state of the dynamic Bloom filter can be updated based on the Bloom filtration rate and the actual filtration rate.

[0081] In one embodiment, updating the on / off state of the dynamic Bloom filter based on the Bloom filtration rate and the true filtration rate includes: determining whether the true filtration rate is less than a true filtration rate threshold, or whether the true filtration rate is greater than or equal to the true filtration rate threshold and whether the Bloom filtration rate is less than the product of the Bloom filtration rate threshold and the true filtration rate; if so, updating the on / off state of the dynamic Bloom filter to off.

[0082] The true filtration rate threshold and the Bloom filtration rate threshold can be set according to the actual situation. Different Bloom filters may have different true filtration rate thresholds and Bloom filtration rate thresholds.

[0083] In this embodiment, when the actual filtration rate is less than the actual filtration rate threshold, the on / off state of the dynamic Bloom filter can be updated to "off"; alternatively, when the actual filtration rate is greater than or equal to the actual filtration rate threshold, and the Bloom filtration rate is less than the product of the Bloom filtration rate threshold and the actual filtration rate, the on / off state of the dynamic Bloom filter can be updated to "off". This allows the Bloom filter to be turned off when its effectiveness is poor, avoiding increased costs due to frequent filter use in scenarios where filter performance is inadequate.

[0084] S250. Take the remaining data of the right table as the new current batch right table data in turn, and continue to match the data of the new current batch right table data according to the updated switch status until all the right table data is matched.

[0085] Embodiment 2 of this invention provides a data matching method based on a Bloom filter. Specifically, when the current matching method is based on a dynamic Bloom filter, the method involves matching the right and left table data of the current batch according to the current matching method and updating the on / off state of the dynamic Bloom filter. This includes: matching the right and left table data of the current batch based on the dynamic Bloom filter to obtain the Bloom filter rate and the true filter rate of the right table data of the current batch; and updating the on / off state of the dynamic Bloom filter based on the Bloom filter rate and the true filter rate. This method updates the on / off state of the dynamic Bloom filter by using the Bloom filter rate and the true filter rate, enabling the reasonable use of the dynamic Bloom filter during data matching and solving the problem in the prior art where Bloom filters cannot be used appropriately in different data scenarios.

[0086] In one embodiment, when the current matching method is direct matching, the step of performing data matching on the right table data and left table data of the current batch according to the current matching method and updating the on / off state of the dynamic Bloom filter includes: performing direct data matching on the right table data and left table data of the current batch; updating the deactivation round of the dynamic filter; and if the deactivation round has ended, updating the on / off state of the dynamic Bloom filter to on.

[0087] The shutdown cycle can be the cycle in which the dynamic Bloom filter is deactivated, and the shutdown cycle can be set according to the actual situation. The shutdown cycle can be set when the on / off state of the dynamic Bloom filter is determined to be off.

[0088] In this embodiment, when the current matching method is direct matching, after matching the data in the right table and the left table of the current batch, the deactivation round of the dynamic filter can be decremented by one. If the deactivation round = 0, the on / off state of the dynamic Bloom filter is updated to on.

[0089] In one embodiment, when the state of the dynamic Bloom filter is off for two consecutive updates, the number of deactivation rounds of the dynamic Bloom filter can be increased.

[0090] For example, if the initial deactivation round of the dynamic Bloom filter is set to 3, the filter will automatically reactivate when the current deactivation round ends (i.e., deactivation round = 0). If the dynamic Bloom filter's on / off state is updated to "off" again during the current round, meaning the filter has been deactivated twice consecutively, the deactivation round can be set to a longer duration, such as 4. This allows for improving the overall effectiveness of the BF (Browser Filter) based on the distribution characteristics of local data.

[0091] Based on the technical solutions of the above embodiments, this invention provides a specific implementation method:

[0092] During the data matching phase, based on the original BF mechanism and combined with monitoring of the BF filtering process, a mechanism for dynamically switching BF on and off is added according to the distribution characteristics of local data to improve the overall effectiveness of BF. The lower limit of the MISS rate (true filtering rate) threshold for evaluating whether BF is enabled is set as P_miss, and the pre-filtering rate threshold (Bloom filtering rate) is set as P_prefilter.

[0093] 1) When matching the first batch of data from the right table, set BF to "Open". Proceed to step 2);

[0094] 2) Check if the BF status is "temporarily open". If yes, proceed to step 3). If the BF status is "temporarily closed", proceed to step 4.

[0095] 3) Run the data matching process using BF, count the MISS rate and the BF filtering rate. Proceed to step 5);

[0096] 4) Run the data matching process without using BF, decrement the deactivation round count by one, and continue running the data matching process without using BF to process the next batch of data; automatically enable BF when the deactivation rounds are completed, and if the data processing is not finished, proceed to step 2);

[0097] 5) Check the MISS rate and BF filtering rate of the current batch of data. When the MISS rate is lower than the lower threshold limit P_miss, or the MISS rate is higher than the lower threshold limit P_miss and the BF filtering rate < MISS rate * P_prefilter, the BF state is set to "Off", otherwise it is set to "On". When the current batch of data is the last batch of data, proceed to step 6); otherwise, take the next batch of data and proceed to step 2);

[0098] 6) The matching is completed.

[0099] Third Embodiment

[0100] Figure 3 This is a schematic structural diagram of a data matching apparatus based on a Bloom filter provided by the third embodiment of the present invention. The apparatus is applicable to scenarios of data matching, wherein the apparatus can be implemented by software and / or hardware, and is generally integrated in an electronic device.

[0101] As Figure 3 shown, the apparatus includes:

[0102] A construction module 310, configured to construct a dynamic Bloom filter according to the left table data;

[0103] A determination module 320, configured to determine the current matching mode for the current batch of right table data of the right table data according to the on-off state of the dynamic Bloom filter;

[0104] A first matching module 330, configured to perform data matching on the current batch of right table data and the left table data according to the current matching mode, and update the on-off state of the dynamic Bloom filter;

[0105] A second matching module 340, configured to sequentially use the remaining data of the right table data as new current batches of right table data, and continue to perform data matching on the new current batches of right table data according to the updated on-off state until all right table data are matched.

[0106] This embodiment provides a data matching device based on a Bloom filter. The device includes: a construction module for constructing a dynamic Bloom filter based on data from a left table; a determination module for determining a current matching method for the current batch of right table data based on the on / off state of the dynamic Bloom filter; a first matching module for performing data matching on the current batch of right table data and left table data according to the current matching method, and updating the on / off state of the dynamic Bloom filter; and a second matching module for sequentially using the remaining data from the right table data as new current batch right table data, and continuing to perform data matching on the new current batch right table data according to the updated on / off state, until all right table data is matched. By determining the current matching method for the current batch of right table data and left table data based on the on / off state of the dynamic Bloom filter, and updating the on / off state after the current batch of right table data is matched, the dynamic Bloom filter can be used reasonably during the data matching process, solving the problem in the prior art that Bloom filters cannot be used reasonably in different data scenarios.

[0107] Furthermore, module 310 includes:

[0108] Determine the current Bloom filter and the current hash function;

[0109] For the current batch of left table data, set the current Bloom filter based on the current hash function;

[0110] Determine whether the current Bloom filter meets the filter expansion conditions;

[0111] If so, add a new Bloom filter and use the new Bloom filter as the current Bloom filter;

[0112] The remaining data in the left table are processed sequentially as the current batch of left table data until all the left table data is processed, resulting in the constructed dynamic Bloom filter.

[0113] Furthermore, the filter expansion conditions include: the current Bloom filter's fill rate is less than or equal to the fill rate threshold, the current Bloom filter's fill change rate is less than the fill change rate threshold, the alarm count is greater than the alarm count threshold, and the number of hash functions is less than or equal to the number of hash functions threshold.

[0114] Furthermore, the addition of the new Bloom filter also includes:

[0115] Increase the number of hash functions in the new Bloom filter.

[0116] Furthermore, when the current matching method is based on a dynamic Bloom filter, the first matching module 330 includes:

[0117] Based on the dynamic Bloom filter, the data of the right table and the data of the left table of the current batch are matched to obtain the Bloom filter rate and the true filter rate of the right table data of the current batch.

[0118] The on / off state of the dynamic Bloom filter is updated based on the Bloom filtration rate and the actual filtration rate.

[0119] Furthermore, updating the on / off state of the dynamic Bloom filter based on the Bloom filtration rate and the actual filtration rate includes:

[0120] Determine whether the true filtration rate is less than the true filtration rate threshold, or whether the true filtration rate is greater than or equal to the true filtration rate threshold and whether the Bloom filtration rate is less than the product of the Bloom filtration rate threshold and the true filtration rate.

[0121] If so, then update the on / off state of the dynamic Bloom filter to off.

[0122] Furthermore, when the current matching method is direct matching, the first matching module 330 includes:

[0123] The data in the right table and the left table of the current batch are directly matched;

[0124] Update the deactivation rounds of the dynamic filter;

[0125] If the deactivation cycle has ended, the on / off state of the dynamic Bloom filter is updated to on.

[0126] The above-described Bloom filter-based data matching device can execute the Bloom filter-based data matching method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0127] Example 4

[0128] Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0129] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0130] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0131] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the Bloom filter-based data matching method.

[0132] In some embodiments, the Bloom filter-based data matching method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the Bloom filter-based data matching method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the Bloom filter-based data matching method by any other suitable means (e.g., by means of firmware).

[0133] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0134] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0135] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0136] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0137] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0138] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0139] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0140] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data matching method based on a Bloom filter, characterized in that, The method includes: Construct a dynamic Bloom filter based on the data in the left table; For the current batch of right table data, determine the current matching method based on the on / off state of the dynamic Bloom filter; The data in the right table and the left table of the current batch are matched according to the current matching method, and the on / off state of the dynamic Bloom filter is updated. The remaining data in the right table are used as the new right table data for the current batch. The data matching of the new right table data for the current batch is continued according to the updated switch status until all the right table data is matched. When the current matching method is based on a dynamic Bloom filter, the step of matching the data in the right and left tables of the current batch according to the current matching method and updating the on / off state of the dynamic Bloom filter includes: Based on the dynamic Bloom filter, the data of the right table of the current batch and the data of the left table are matched to obtain the Bloom filter rate and the true filter rate of the right table of the current batch. The on / off state of the dynamic Bloom filter is updated based on the Bloom filtration rate and the actual filtration rate. When the current matching method is direct matching, the step of matching the data in the right table and the left table of the current batch according to the current matching method and updating the on / off state of the dynamic Bloom filter includes: The data in the right table and the left table of the current batch are directly matched; Update the deactivation cycle of the dynamic Bloom filter; If the deactivation cycle has ended, the on / off state of the dynamic Bloom filter is updated to on.

2. The method according to claim 1, characterized in that, The construction of the dynamic Bloom filter based on the data in the left table includes: Determine the current Bloom filter and the current hash function; For the current batch of left table data, set the current Bloom filter based on the current hash function; Determine whether the current Bloom filter meets the filter expansion conditions; If so, add a new Bloom filter and use the new Bloom filter as the current Bloom filter; The remaining data in the left table are processed sequentially as the current batch of left table data until all the left table data is processed, resulting in the constructed dynamic Bloom filter.

3. The method according to claim 2, characterized in that, The filter expansion conditions include: the current Bloom filter's fill rate is less than or equal to the fill rate threshold, the current Bloom filter's fill change rate is less than the fill change rate threshold, the alarm count is greater than the alarm count threshold, and the number of hash functions is less than or equal to the number of hash functions threshold.

4. The method according to claim 2, characterized in that, The addition of the new Bloom filter also includes: Increase the number of hash functions in the new Bloom filter.

5. The method according to claim 1, characterized in that, The step of updating the on / off state of the dynamic Bloom filter based on the Bloom filtration rate and the true filtration rate includes: Determine whether the true filtration rate is less than the true filtration rate threshold, or whether the true filtration rate is greater than or equal to the true filtration rate threshold and whether the Bloom filtration rate is less than the product of the Bloom filtration rate threshold and the true filtration rate. If so, then update the on / off state of the dynamic Bloom filter to off.

6. A data matching device based on a Bloom filter, characterized in that, The device includes: The building module is used to construct dynamic Bloom filters based on the data in the left table; The determination module is used to determine the current matching method for the current batch of right table data based on the on / off state of the dynamic Bloom filter. The first matching module is used to perform data matching on the right table data and left table data of the current batch according to the current matching method, and update the on / off state of the dynamic Bloom filter. The second matching module is used to take the remaining data of the right table as the new current batch right table data in turn, and continue to match the data of the new current batch right table data according to the updated switch status, until all the right table data is matched. When the current matching method is based on a dynamic Bloom filter, the first matching module includes: Based on the dynamic Bloom filter, the data of the right table of the current batch and the data of the left table are matched to obtain the Bloom filter rate and the true filter rate of the right table of the current batch. The on / off state of the dynamic Bloom filter is updated based on the Bloom filtration rate and the actual filtration rate. When the current matching method is direct matching, the first matching module includes: The data in the right table and the left table of the current batch are directly matched; Update the deactivation cycle of the dynamic Bloom filter; If the deactivation cycle has ended, the on / off state of the dynamic Bloom filter is updated to on.

7. An electronic device, characterized in that, The device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the Bloom filter-based data matching method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the data matching method based on a Bloom filter as described in any one of claims 1-5.

Images