Data matching method and device and related equipment
By using multiple hash functions to generate hash values for ordered sets and mapping them into bit arrays, the problems of high resource consumption and low efficiency in big data matching are solved, achieving efficient and secure big data matching.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-11
- Publication Date
- 2026-04-07
AI Technical Summary
Existing big data matching methods suffer from high resource consumption and low matching efficiency. In particular, they cannot achieve full set matching when dealing with large amounts of data, and data security is difficult to guarantee.
Multiple hash functions are used to perform hash operations on the data, generating hash values of ordered sets, which are then mapped into bit arrays. By matching the bit arrays with the pre-stored target correspondence, it is determined whether the target data exists in the full dataset, reducing memory resource consumption and the risk of data leakage.
It reduced hardware costs, improved matching efficiency, enabled one-time matching of all data, and enhanced data security, reducing the risk of data leakage.
Smart Images
Figure CN121808408A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data technology, and in particular to a data matching method, apparatus and related equipment. Background Technology
[0002] Big data matching refers to the process of using big data technology to process and analyze massive and diverse data to find data sets or patterns that meet specific conditions or needs. Applications of big data matching include at least: precision marketing (by collecting and analyzing consumer behavior data and preference information, businesses can more accurately target customer groups, achieve personalized recommendations and precision marketing, thereby improving marketing efficiency and conversion rates), risk assessment (big data matching can help businesses assess customer credit risk, fraud risk, etc., providing strong support for decision-making), and smart cities (in smart city construction, big data matching can assist in urban traffic management, environmental monitoring, public safety, and other aspects, improving urban governance).
[0003] Currently, big data matching methods are mainly divided into rule-based matching, statistical matching, machine learning-based matching, and semantic matching. The basic idea behind these methods is usually based on data structures such as linked lists and trees. The general approach is to process and store the entire dataset, then compare it with the input data. The matching result between the input data and the stored full dataset determines whether the input data exists within the full dataset.
[0004] Matching patterns based on data structures such as linked lists and trees require storing the data itself within the structure. This not only consumes a large amount of storage resources, leading to high hardware costs, but also compromises data security because the data needs to be physically stored. Furthermore, for large datasets, it is virtually impossible to match the entire set at once, and efficiency decreases as the data volume grows. Therefore, a low-resource-consumption, high-efficiency, and secure big data matching method is extremely important and urgent. Summary of the Invention
[0005] This application provides a data matching method, apparatus, and related equipment, which can solve the technical problems of high resource consumption and low matching efficiency in the matching of large amounts of data in related technologies.
[0006] In a first aspect, embodiments of this application provide a data matching method, the method comprising:
[0007] Get the first data to be matched;
[0008] Using M hash functions, hash operations are performed on the first data to obtain a first ordered set of the first data. The first ordered set is an ordered arrangement of M first hash values corresponding to the M hash functions, where M is a positive integer greater than 1.
[0009] When each position in the first position set of the pre-stored bit array is mapped to the first hash value, the first data is matched based on the pre-stored target correspondence and the first ordered set to obtain a matching result. The matching result is used to indicate whether there is second data in the full data that matches the first data. The target correspondence is the correspondence between each second data in the full data and the second ordered set of the second data. The second ordered set is the M second hash values that are ordered and correspond one-to-one with the M hash functions. The second ordered set corresponds to the second position set of the bit array.
[0010] Secondly, embodiments of this application provide a data matching device, the device comprising:
[0011] The acquisition module is used to acquire the first data to be matched;
[0012] The first hash processing module is used to perform hash operations on the first data using M hash functions respectively to obtain a first ordered set of the first data. The first ordered set is an ordered arrangement of M first hash values corresponding one-to-one with the M hash functions, where M is a positive integer greater than 1.
[0013] A matching module is used to match the first data based on a pre-stored target correspondence and the first ordered set, where each position in the first position set of the pre-stored bit array is mapped to the first hash value, to obtain a matching result. The matching result is used to indicate whether there is second data in the full data that matches the first data. The target correspondence is the correspondence between each second data in the full data and the second ordered set of the second data. The second ordered set is the M second hash values that are ordered and correspond one-to-one with the M hash functions. The second ordered set corresponds to the second position set of the bit array.
[0014] Thirdly, embodiments of this application provide an electronic device, including: a processor, a memory, and a program stored in the memory and executable on the processor, wherein when the program is executed by the processor, it implements the steps of the data matching method as described in the first aspect.
[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the data matching method as described in the first aspect.
[0016] Fifthly, embodiments of this application provide a computer program product including computer instructions that, when executed by a processor, implement the steps of the data matching method as described in the first aspect.
[0017] In this embodiment, multiple hash values are obtained by hashing the second data in the full dataset using multiple hash functions. These second hash values are then mapped to positions in the second position set of the bit array to determine the correspondence between the second data and the bit array positions, thus obtaining the target correspondence. During first data matching, the first data is hashed using multiple hash functions, and the resulting first ordered set of multiple first hash values is mapped to positions in the bit array. If a mapping match is successful (i.e., each position in the pre-stored first position set of the bit array is mapped to a first hash value), it is determined that there may be second data in the full dataset that matches the first data. Accordingly, based on this target correspondence and the first ordered set, the first data can be matched within the full dataset. Thus, mapping the hash values of data in the full dataset to positions in the bit array using multiple hash functions reduces memory resources and hardware costs. Furthermore, it significantly increases the data volume of the full dataset, enabling a single-step matching of the entire dataset. Moreover, the data matching process does not require actual data participation; it only involves matching ordered sets of hash values, thereby reducing the risk of data leakage during the matching process and improving matching efficiency. Attached Figure Description
[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a flowchart of a data matching method provided in an embodiment of this application;
[0020] Figure 2 This is a schematic diagram illustrating the process of hashing data.
[0021] Figure 3 This is a schematic diagram illustrating the mapping of the second ordered set to its position in the bit array;
[0022] Figure 4This is a schematic diagram illustrating the mapping and matching between the first ordered set and the positions in the bit array;
[0023] Figure 5 This is a flowchart of the processing of the original data mapping unit in the embodiments of this application;
[0024] Figure 6 This is a logic diagram of the implementation of a data matching method as provided in a specific example of an embodiment of this application;
[0025] Figure 7 This is an implementation architecture diagram of a specific example of a data matching method provided in an embodiment of this application;
[0026] Figure 8 This is a schematic diagram of the structure of a data matching device provided in an embodiment of this application;
[0027] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0029] Current mainstream data matching methods rely on data structures such as linked lists and trees for processing during both data preprocessing and matching. These solutions have significant drawbacks and limitations.
[0030] (1) High storage resource consumption and high hardware cost.
[0031] Because the data needs to be actually stored in the relevant data structures, the consumption of storage resources is huge, and in order to ensure matching efficiency, it is mainly memory resources that are consumed, resulting in high hardware costs.
[0032] (2) Data is actually stored, but security cannot be guaranteed.
[0033] To achieve data matching, the data to be matched needs to be actually loaded into memory. In order to improve the matching speed of input data, the data is generally not encrypted and stored, which can easily lead to data leakage.
[0034] (3) Matching large amounts of data is limited by resources and may be done in multiple layers, so efficiency cannot be guaranteed.
[0035] For matching large amounts of data, it is impossible to load the entire dataset into memory at once, so it is basically impossible to perform a one-time full-set matching. The matching efficiency will decrease as the amount of data increases.
[0036] Based on this, embodiments of this application provide a data matching method aimed at solving the aforementioned technical problems.
[0037] It should be noted that the data matching method provided in this application embodiment is applied in a data matching device, and mainly involves the fields of information technology (IT) support, big data and other technical fields.
[0038] See Figure 1 , Figure 1 This is a flowchart of a data matching method provided in an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:
[0039] Step 101: Obtain the first data to be matched;
[0040] Step 102: Using M hash functions, perform hash operations on the first data respectively to obtain the first ordered set of the first data. The first ordered set is the M first hash values corresponding to the M hash functions in an ordered manner, where M is a positive integer greater than 1.
[0041] Step 103: When each position in the first position set of the pre-stored bit array is mapped to the first hash value, the first data is matched based on the pre-stored target correspondence and the first ordered set to obtain a matching result. The matching result is used to indicate whether there is second data in the full data that matches the first data. The target correspondence is the correspondence between each second data in the full data and the second ordered set of the second data. The second ordered set is the M second hash values that are ordered and correspond one-to-one with the M hash functions. The second ordered set corresponds to the second position set of the bit array.
[0042] In step 101, the first data refers to the data to be matched, such as the keyword "China Eastern Airlines". The first data can be entered by the user or sent by other electronic devices, and there are no specific limitations here.
[0043] In step 102, the first data can be hashed using multiple hash functions in a preset order to obtain multiple first hash values. These multiple first hash values constitute a first ordered set of the first data. The number and order of these multiple hash functions are the same as those used to hash the second data in the full dataset.
[0044] In step 103, the multiple first hash values obtained sequentially can be mapped to the first position set in the bit array. If the positions in the first position sets of the bit array corresponding to the multiple first hash values are all first values, the mapping is successful, indicating that there may be second data in the full data that matches the first data. The first value can be 1, meaning that when the second data in the full data is mapped to a position in the bit array, the position in the bit array corresponding to the second hash value of the second data can be mapped to 1, while other positions can keep the second value, such as 0, unchanged.
[0045] In other words, before data matching, all data needs to be mapped to the corresponding positions in the bit array to reduce memory resources and lower hardware costs.
[0046] In some embodiments, a second data corresponds to a second ordered set.
[0047] In some embodiments, prior to step 101, the method further includes:
[0048] Each second data point in the full dataset is segmented into word groups to obtain multiple word groups corresponding to each second data point.
[0049] Using the M hash functions, a hash operation is performed on each of the word groups to obtain a second ordered set of each of the word groups. The second ordered set of the second data includes the second ordered set of each of the word groups.
[0050] Map each second hash value in the second ordered set of the second data in the full data to the position corresponding to the second hash value in the second position set;
[0051] The target correspondence is established based on the second ordered set of each of the aforementioned word groups and each second data in the full dataset.
[0052] Specifically, the data segmentation unit in the data matching device can be used to segment each second data in the full data into phrases. This data segmentation unit is used to segment each second data in the full data into phrases to obtain multiple phrases corresponding to each second data.
[0053] The specific segmentation process is as follows: the segmentation process is divided into corresponding steps according to the number of characters contained in the second data; for each step, the first character is selected, and starting from the first character, the number of characters is increased one by one according to the sentence content of the second data until the last character in the sentence content is reached, and each additional character forms a phrase.
[0054] The object processed by the data segmentation unit is the full amount of data. The reason for segmenting the second data is mainly that when performing data matching, it is generally to determine which data in the full amount of data contains a certain sentence or keyword. For example, to match which users who have疑似 taken "China Eastern Airlines" from the user industry SMS content data. This implementation method is to match whether the relevant keywords such as "China Eastern Airlines" are included in the booking SMS received by the user. Therefore, when matching, it is necessary to segment the data, splitting a sentence into individual independent phrases, which is convenient for subsequent matching links to perform matching according to keywords and so on. The specific segmentation process is exemplified as follows:
[0055] Taking the segmentation of user industry SMS content as an example, the original SMS content data: Thank you for taking China Eastern Airlines. When performing phrase segmentation, the following rules will be used to generate a phrase set:
[0056] In the first step, starting with "感", the phrase segmentation will obtain:
[0057] 感, 感谢, 感谢您, 感谢您乘, 感谢您乘坐, 感谢您乘坐东, 感谢您乘坐东方, 感谢您乘坐东方航, 感谢您乘坐东方航空。
[0058] In the second step, starting with "谢", the phrase segmentation will obtain:
[0059] 谢, 谢您, 谢您乘, 谢您乘坐, 谢您乘坐东, 谢您乘坐东方, 谢您乘坐东方航, 谢您乘坐东方航空。
[0060] Repeat the above steps to complete the phrase segmentation of the original SMS content data according to this rule. Finally, the original SMS content data will be segmented into the following phrase set:
[0061] 感, 感谢, 感谢您, 感谢您乘, 感谢您乘坐, 感谢您乘坐东, 感谢您乘坐东方, 感谢您乘坐东方航, 感谢您乘坐东方航空;
[0062] 谢, 谢您, 谢您乘, 谢您乘坐, 谢您乘坐东, 谢您乘坐东方, 谢您乘坐东方航, 谢您乘坐东方航空;
[0063] 您, 您乘, 您乘坐, 您乘坐东, 您乘坐东方, 您乘坐东方航, 您乘坐东方航空;
[0064] 乘, 乘坐, 乘坐东, 乘坐东方, 乘坐东方航, 乘坐东方航空;
[0065] 坐, 坐东, 坐东方, 坐东方航, 坐东方航空;
[0066] 东, 东方, 东方航, 东方航空;
[0067] 方, 方航, 方航空;
[0068] air, aviation;
[0069] sky.
[0070] The hash operation unit in the data matching device can perform hash operations on the data. The hash operation unit can perform multiple hash operations on each phrase corresponding to the second data respectively using multiple hash functions in a preset order, and correspondingly obtain multiple second hash values. Here, the multiple hash operations refer to performing hash operations on the phrases respectively using multiple hash functions, and the hash functions used for each phrase and the order of use of each hash function are the same.
[0071] The hash operation unit mainly performs hash operations on each phrase obtained by the data segmentation unit in sequence according to a series of hash functions to obtain multiple second hash values.
[0072] Specifically, taking the above example, the set of segmented phrases is {感, 感谢, 感谢您, 感谢您乘, 感谢您乘坐, 感谢您乘坐东, 感谢您乘坐东方, 感谢您乘坐东方航, 感谢您乘坐东方航空, 谢, 谢您, 谢您乘, 谢您乘坐, 谢您乘坐东, 谢您乘坐东方, 谢您乘坐东方航, 谢您乘坐东方航空, 您, 您乘, 您乘坐, 您乘坐东, 您乘坐东方, 您乘坐东方航, 您乘坐东方航空, 乘, 乘坐, 乘坐东, 乘坐东方, 乘坐东方航, 乘坐东方航空, 坐, 坐东, 坐东方, 坐东方航, 坐东方航空, 东, 东方, 东方航, 东方航空, 方, 方航, 方航空, 航, 航空, 空}.
[0073] Figure 2 It is a schematic diagram of the process of performing hash operations on the data, as Figure 2 shown. Input each phrase such as "感" into the hash operation unit. The hash operation unit can use M hash functions in a preset order, such as hash-1,..., hash-M, to perform hash operations on the phrase, and can obtain a second ordered set for each phrase, such as value 1... value 99.
[0074] In addition, the data matching device may further include a storage unit, and the storage unit can map the multiple second hash values obtained in sequence in the second ordered set to multiple positions in the bit array in sequence.
[0075] In some embodiments, the mapping of each second hash value in the second ordered set of the second data in the full amount of data to the position corresponding to the second hash value in the second position set includes:
[0076] Based on the second ordered set of the second data in the full amount of data, map the positions corresponding to each of the second hash values in the second position set of the bit array to a first value, where the first value is different from the values of other positions in the bit array except for the second position set.
[0077] That is, the values of multiple positions corresponding to the bit array can be set to 1, and the values of other positions remain unchanged at 0. In this way, based on the second ordered set of each phrase and each second data in the full amount of data, a target correspondence between the second data and the second position set in the bit array can be established.
[0078] In some embodiments, the target correspondence can be the correspondence between the second data and the second position set in the bit array. In some embodiments, the target correspondence can be the correspondence between the phrases in the second data and the second position set in the bit array.
[0079] In some embodiments, the target correspondence can include a first correspondence and a second correspondence. The first correspondence can be the correspondence between the service code of the second data and the second position set formed by the positions in the bit array. Among them, the second position set corresponds to multiple second hash values corresponding to the second data, and the values in the second position set of the bit array are the first value such as 1. The second correspondence can be the correspondence between the service code of the second data and the second data.
[0080] In some embodiments, the storage unit is responsible for mapping the second hash values of each phrase obtained by the hash operation unit to multiple positions in the bit array and setting these positions to 1, as Figure 3 shown. After the phrase "感" is operated by the hash operation unit, a second ordered set including values 1 and 99 is obtained, and the values of the 1st and 99th positions in the bit array can be mapped to 1.
[0081] In this embodiment, converting the original data in the full amount of data into 0 and 1 values for storage can greatly increase the data magnitude of the full amount of data to be matched, and one-time matching of the full amount of data can be achieved.
[0082] In addition to modifying the corresponding values of the bit array according to the hash values, the storage unit is also responsible for retaining the correspondence between the service code of the second data and the second ordered set corresponding to the phrase in the relationship set (usually stored in the memory library). The relationship between the second data and the second ordered set can be one-to-many. Among them, the service code of the second data is the unique code of the second data, generally from the original service data. For example, the service code of the industry short message content "Thank you for taking China Eastern Airlines" in the original service data is the short message sending serial number.
[0083] Taking the industry short message content as an example, the storage format in the relationship set is as shown in Table 1 below.
[0084] Table 1. Correspondence between the business codes of the second data and the second data.
[0085]
[0086] The first correspondence is shown in Table 2 below.
[0087] Table 2. Correspondence between the business codes of the second data and the first ordered set.
[0088]
[0089] Since different phrases may have the same value after being mapped by a hash function, different business codes for the second data may have the same second ordered set.
[0090] In step 103, the matching unit in the data matching device can map the first ordered set of the first data to the first position set in the bit array. If all positions in the bit array corresponding to the first hash value in the first ordered set are 1, the mapping is successful, and further data matching can be performed. The data matching process can be the matching of the first ordered set with the second ordered set in the target correspondence. In this way, the matching process does not require the participation of actual data, but only the matching of hash values of a series of hash functions, reducing the risk of data leakage during the matching process.
[0091] The matching unit mainly searches within the bit array of the first ordered set obtained by the hash operation unit on the first data. If all the position values of the first set in the bit array are 1, the mapping match is successful, indicating that the first data may be in the full data. The business code of the corresponding second data can be found based on the first correspondence and the first ordered set. The implementation process is as follows:
[0092] For example, such as Figure 4 As shown, the first ordered set obtained after the input "China Eastern Airlines" is processed by the hash unit is the value 3, 88. The value of the 3rd and 88th positions in the bit array is 1, so the phrase "China Eastern Airlines" may be in the full data.
[0093] It should be noted that the fact that the first ordered set corresponds to the first position set in the bit array does not necessarily mean that the first data exists in the entire data. The position order still needs to be matched. For example, if the first ordered set is 88 and 3, and the values of positions 3 and 88 in the bit array are both 1, the position order still needs to be matched.
[0094] In some embodiments, the matching of positional order can be a matching of the first ordered set with the second ordered set in the target correspondence. Since both are sorted according to the results of multiple hash functions, positional order matching can be performed when they are matched. Furthermore, after mapping and matching the positions in the first ordered set and the bit array, further matching of the first ordered set with the second ordered set in the target correspondence is performed, thereby achieving the matching of the first data in the full dataset. By converting the data to 0 and 1 values for storage, the data volume of the full dataset can be greatly increased, and by matching the first ordered set of the first data with the second ordered set of the second data, a single matching of the full dataset can be achieved.
[0095] In some embodiments, step 103 specifically includes:
[0096] If a second ordered set that is identical to the first ordered set exists in the target correspondence, then based on the target correspondence, the second data corresponding to the second ordered set that is identical to the first ordered set is obtained from the full data, and the matching result includes the second data corresponding to the second ordered set that is identical to the first ordered set.
[0097] If there is no second ordered set identical to the first ordered set in the target correspondence, the matching result indicates that there is no second data matching the first data in the full data.
[0098] In other words, if the first ordered set corresponds to the first position set in the bit array, further position matching is required. Only if a second ordered set identical to the first ordered set exists in the target correspondence can it be concluded that the first data definitely exists in the full dataset. Otherwise, the matching result indicates that there is no second data matching the first data in the full dataset.
[0099] Furthermore, by matching ordered sets after each position in the first set of positions in the pre-stored bit array is mapped to a first hash value, the matching accuracy can be improved.
[0100] In some embodiments, based on the matching of the first ordered set and the second ordered set, only one second data point may be matched. In this case, the second data point can be output as the data that matches the first data point.
[0101] In some embodiments, obtaining second data corresponding to a second ordered set that is identical to the first ordered set from the full dataset based on the target correspondence includes:
[0102] Based on the target correspondence, N second data that match the first ordered set are determined from the full data, where N is a positive integer greater than or equal to 1;
[0103] Perform regular expression matching on the N second data and the first data to determine the second data that matches the first data from the N second data.
[0104] Among them, data matching can be performed on the full dataset through the original data mapping unit in the data matching device. The original data mapping unit can perform data matching on the full dataset based on the pre-stored correspondence between the second data and the second ordered set.
[0105] In some embodiments, determining N second data points that match the first ordered set from the full dataset based on the target correspondence includes:
[0106] Based on a pre-stored first correspondence, N service codes that match the first ordered set are determined from the service codes of the full data. The first correspondence is the correspondence between the service code of each second data in the full data and the second ordered set of the second data. The target correspondence includes the first correspondence.
[0107] Based on the pre-stored second correspondence, N second data corresponding to the N service codes are determined from the full data. The second correspondence is the correspondence between the service code of each second data in the full data and the second data. The target correspondence also includes the second correspondence.
[0108] In other words, the original data mapping unit can first match the first ordered set with the second ordered set based on the first correspondence between the pre-stored business codes of the second data and the second ordered set, so as to match N business codes that match the first ordered set. Then, based on the second correspondence between the business codes of each second data in the full data and the second data, it can obtain N second data corresponding to the N business codes, thereby determining N second data that match the first data in the full data.
[0109] Furthermore, the original data mapping unit can perform regular expression matching between N second data and the first data to determine the second data that ultimately matches the first data in the full dataset.
[0110] As can be seen, the original data mapping unit mainly searches for N second data that match the first data in the relation set according to the matching results of the first ordered set and the second ordered set, and outputs the matched original data after performing a second precise matching, which is used for subsequent secondary processing in business.
[0111] During the processing of the original data mapping unit, according to the second ordered set matched with the first ordered set, there may be multiple original data that meet the conditions in the relation set. Therefore, the original data mapping unit includes the function of secondary precise matching. Its purpose is to filter out the data that does not meet the conditions again. Because only a small number of datasets meet the conditions after the ordered set matching, the second precise matching will not consume much time and the efficiency will not be affected. Compared with the traditional precise matching based on the full data, it can improve efficiency by a great deal.
[0112] Taking SMS content matching as an example, the processing flow of the raw data mapping unit is as follows: Figure 5 As shown, after the matching unit has obtained the first ordered set of "China Eastern Airlines" as the value 3.88, it can use the value 3.88 to find the business codes of the two original data in the relation set (which can store the first and second correspondences), namely dx000001 and dx000002. These two business codes are then used to find the original data in the second correspondence, with the SMS messages being "Thank you for flying with China Eastern Airlines" and "Your ticket order ****068229 on Caizhiyun has been issued." Afterwards, a simple regular expression match can be performed between the first data "China Eastern Airlines" and the two original data to obtain the most accurate matching result, i.e., the original data obtained by matching the phrase "China Eastern Airlines" is "Thank you for flying with China Eastern Airlines".
[0113] The following example illustrates in detail the data matching method provided in the embodiments of this application.
[0114] Figure 6 This is a logic diagram of the implementation of a data matching method provided in a specific example of an embodiment of this application, as shown below. Figure 6 As shown, this embodiment provides a data matching device based on a Bloom filter, suitable for matching scenarios that require matching and filtering massive amounts of data while demanding low resource consumption and maintaining a certain level of confidentiality for business data. Here, a Bloom filter refers to filtering the entire dataset to obtain multiple original data sets that match the first set of data.
[0115] The data matching device, starting from the business scenario of big data matching, is constructed from input, matching, and output. It first preprocesses the full dataset, segmenting each raw data point into word groups. Each segmented word group is then hashed using multiple hash functions in a preset order, yielding multiple second hash values. The device then locates the corresponding bit array position for each second hash value and sets the value of each corresponding bit array position to 1, thus establishing a first correspondence between the original data business code and a second ordered set. This second ordered set corresponds to the second position set in the bit array. This first correspondence is stored in a relation set. The correspondence between the second ordered set and the second position set means that the corresponding positions in the bit array within the set are the same; that is, the multiple second hash values in the second ordered set have been mapped to the positions in the second position set of the bit array.
[0116] In addition, a second correspondence between the original data service code and the original data can be established. In the matching process, the first hash value of the first data is calculated based on each hash function in a preset order. The obtained first ordered set is mapped and matched with the first position set in the bit array. If the value of each position in the first position set is 1, and there is a second ordered set in the first correspondence that is the same as the first ordered set, then at least one original data service code corresponding to the first ordered set is obtained, and the matching relationship between the first data and at least one original data service code is obtained. Based on the second correspondence, at least one original data that matches the first data is obtained, thereby realizing the filtering function of the Bloom filter and obtaining the filtering result.
[0117] Then, at least one corresponding original data can be matched with the first data using regular expressions to determine the original data that finally matches the first data, thereby achieving secondary matching and obtaining the mapping result.
[0118] Figure 7This is a schematic diagram of the implementation architecture of a data matching method provided in a specific example of this application. The hash operation unit, storage unit, bit array, and matching unit together constitute a Bloom filter. To compensate for the false positive rate of the Bloom filter, a raw data mapping unit is added. This unit performs a secondary matching on the Bloom filter results and outputs successfully matched data according to the mapping relationship, ensuring the accuracy of the matching results. A Bloom filter false positive occurs when the Bloom filter determines that an element is not in the full dataset, then that element is definitely not in the full dataset; however, if the Bloom filter determines that an element is in the full dataset, then that element may not be present. This is caused by hash collisions and the inherent principle of the Bloom filter. The false positive rate can generally be reduced by adjusting the hash function, but it cannot be completely eliminated. By adding a raw data mapping unit to perform regular expression matching again based on the Bloom filter's matching results, the problem of Bloom filter false positives can be eliminated. The raw data mapping unit and the Bloom filter together constitute the data matching device of this application embodiment.
[0119] In this embodiment, each word group of the original data is hashed using multiple hash functions in a preset order to obtain multiple second hash values. These second hash values are then mapped to positions in a bit array to determine the correspondence between the original data and the bit array positions. During first data matching, the first data is hashed using multiple hash functions in a preset order, and the resulting multiple first hash values are mapped to positions in a bit array. If a correspondence exists (i.e., the bit array positions corresponding to multiple first hash values are all 1, and the multiple first hash values are identical to the multiple second hash values in the first correspondence), then the position order also corresponds, and the matching relationship between the first data and the original data of the same second ordered set is determined. By incorporating the order constraint of the bit array positions during Bloom filtering, the false positive rate of the Bloom filter can be reduced.
[0120] Furthermore, by introducing a data segmentation unit, accurate word segmentation is provided for the original data, forming a set of words corresponding to the original data. In this way, when performing the first data matching, the word matching based on the Bloom filter provides efficient matching capability, can obtain accurate matching results, and can greatly reduce costs.
[0121] Moreover, the secondary precise matching logic of the original data mapping unit makes up for the shortcomings of the Bloom filter matching results having a certain false judgment rate, ensuring the accuracy of the final matching results. By performing secondary matching on the matching results obtained from the Bloom filter, the results that accurately match the first data in the full data can be obtained.
[0122] Compared with related technologies, the embodiments of this application have the following technical advantages.
[0123] Compared to traditional big data matching methods based on linked lists, trees, and other structures, using a Bloom filter to transform the raw data in the full dataset into 0 and 1 values can reduce memory resources and lower hardware costs. The matching process does not require actual data, reducing the risk of data leakage. This reduced reliance on storage resources significantly increases the data volume of the full dataset, enabling one-time matching based on the entire dataset and improving matching efficiency.
[0124] Furthermore, by mapping the hash values of the original data to their positions in the bit array, and establishing a correspondence between the hash values of the sequentially arranged original data and the original data themselves (where the order is a preset order of multiple hash functions), this sequential mapping allows filtering out original data where all positions in the bit array are 1 but the order does not match when performing a matching query for the first data, thus reducing the false positive rate of the original Bloom filter. When there are multiple original data obtained based on the Bloom filter, a secondary matching using regular expressions can yield accurate matching results.
[0125] The embodiments of this application have the following commercial value.
[0126] This application's embodiments have broad market application prospects in the field of big data applications, especially in the mining and matching of massive amounts of data. It provides a highly effective solution for solving big data matching problems currently prevalent in the industry. It proposes a primary screening of massive data based on a Bloom filter, and to compensate for the inherent false positive rate of Bloom filter matching results, a secondary matching is designed. This ensures both efficient data screening and matching while maintaining the accuracy of the matching results. It has extremely high practical value and can be applied to numerous business scenarios, possessing broad market prospects and extensive market application potential.
[0127] The embodiments of this application can be applied to a deep packet inspection (DPI) parsing platform for in-depth analysis of user internet browsing log behavior, and can also be applied to an information operations platform (IOP) platform for matching marketing opportunities through user behavior tags.
[0128] The embodiments of this application can provide efficient and accurate big data matching solutions for industries such as transportation, healthcare, and finance.
[0129] The embodiments of this application have significant effects on cost reduction and efficiency improvement, as specifically achieved as follows:
[0130] 1. Reduce the hardware costs of big data matching services.
[0131] By leveraging the bit array storage mechanism of Bloom filters, the dependence of traditional large-scale business models on large-scale memory resources is avoided, thus reducing hardware costs.
[0132] 2. Improve data security.
[0133] Since no actual business logic needs to be loaded into memory and no actual business data is required during the matching process, the risk of business data leakage can be reduced.
[0134] 3. Improve the efficiency of big data matching.
[0135] The Bloom filter performs efficient initial screening of the entire dataset. Compared with the traditional full-text matching model, the matching efficiency is greatly improved. Other matching and screening processes involving large amounts of data can also refer to this model, which has great reference value.
[0136] This application embodiment belongs to the Smart Middle Platform project. By using a big data matching device based on Bloom filters to perform industry information mining and matching on industry SMS content, it provides more accurate and high-value potential customers for industry clients of the Data Management Platform (DMP) platform.
[0137] The embodiments of this application can be applied to industry SMS content parsing platforms, which can efficiently match the information retained in industry SMS content, provide accurate and efficient matching results for data segmentation in industry applications, and can be extended to similar businesses that require secure and efficient matching of big data.
[0138] See Figure 8 , Figure 8 This is a schematic diagram of the structure of a data matching device provided in an embodiment of this application, as shown below. Figure 8 As shown, the data matching device 800 includes:
[0139] Module 801 is used to obtain the first data to be matched;
[0140] The first hash processing module 802 is used to perform hash operations on the first data using M hash functions to obtain a first ordered set of the first data. The first ordered set is an ordered arrangement of M first hash values corresponding to the M hash functions, where M is a positive integer greater than 1.
[0141] The matching module 803 is used to match the first data based on the pre-stored target correspondence and the first ordered set, when each position in the first position set of the pre-stored bit array is mapped to the first hash value, to obtain a matching result. The matching result is used to indicate whether there is second data in the full data that matches the first data. The target correspondence is the correspondence between each second data in the full data and the second ordered set of the second data. The second ordered set is the M second hash values that are arranged in an ordered manner and correspond one-to-one with the M hash functions. The second ordered set corresponds to the second position set of the bit array.
[0142] Optionally, the matching module 803 is specifically used for:
[0143] If a second ordered set that is identical to the first ordered set exists in the target correspondence, then based on the target correspondence, the second data corresponding to the second ordered set that is identical to the first ordered set is obtained from the full data, and the matching result includes the second data corresponding to the second ordered set that is identical to the first ordered set.
[0144] If there is no second ordered set identical to the first ordered set in the target correspondence, the matching result indicates that there is no second data matching the first data in the full data.
[0145] Optionally, the matching module 803 is further configured to:
[0146] Based on the target correspondence, N second data that match the first ordered set are determined from the full data, where N is a positive integer greater than or equal to 1;
[0147] Perform regular expression matching on the N second data and the first data to determine the second data that matches the first data from the N second data.
[0148] Optionally, the matching module 803 is further configured to:
[0149] Based on a pre-stored first correspondence, N service codes that match the first ordered set are determined from the service codes of the full data. The first correspondence is the correspondence between the service code of each second data in the full data and the second ordered set of the second data. The target correspondence includes the first correspondence.
[0150] Based on the pre-stored second correspondence, N second data corresponding to the N service codes are determined from the full data. The second correspondence is the correspondence between the service code of each second data in the full data and the second data. The target correspondence also includes the second correspondence.
[0151] Optionally, the device further includes:
[0152] The word segmentation module is used to segment each second data in the full data into word segments to obtain multiple word segments corresponding to each second data;
[0153] The second hash operation module is used to perform hash operation on each of the M hash functions to obtain a second ordered set of each of the M hash functions. The second ordered set of the second data includes the second ordered set of each of the M hash functions.
[0154] The mapping module is used to map each second hash value in the second ordered set of the second data in the full data to the position corresponding to the second hash value in the second position set;
[0155] A module is established to establish the target correspondence based on the second ordered set of each of the phrases and each second data in the full data.
[0156] Optionally, the mapping module is specifically used for:
[0157] Based on the second ordered set of the second data in the full data, the position corresponding to each second hash value in the second position set of the bit array is mapped to a first value, and the first value is different from the value of other positions in the bit array other than the second position set.
[0158] The data matching device 800 can implement all the processes implemented in the above data matching method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.
[0159] See Figure 9 The figure shows a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 9 As shown, the electronic device 900 includes: a processor 901, a memory 902, a user interface 903, and a bus interface 904.
[0160] Processor 901 is used to read the program from memory 902 and execute the following procedures:
[0161] Get the first data to be matched;
[0162] Using M hash functions, hash operations are performed on the first data to obtain a first ordered set of the first data. The first ordered set is an ordered arrangement of M first hash values corresponding to the M hash functions, where M is a positive integer greater than 1.
[0163] When each position in the first position set of the pre-stored bit array is mapped to the first hash value, the first data is matched based on the pre-stored target correspondence and the first ordered set to obtain a matching result. The matching result is used to indicate whether there is second data in the full data that matches the first data. The target correspondence is the correspondence between each second data in the full data and the second ordered set of the second data. The second ordered set is the M second hash values that are ordered and correspond one-to-one with the M hash functions. The second ordered set corresponds to the second position set of the bit array.
[0164] exist Figure 9 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 901 and memory represented by memory 902 together. The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 904 provides an interface. For different user devices, user interface 903 can also be an interface capable of connecting external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.
[0165] The processor 901 is responsible for managing the bus architecture and general processing, while the memory 902 can store the data used by the processor 901 when performing operations.
[0166] In some embodiments, the processor 901 is further configured to:
[0167] If a second ordered set that is identical to the first ordered set exists in the target correspondence, then based on the target correspondence, the second data corresponding to the second ordered set that is identical to the first ordered set is obtained from the full data, and the matching result includes the second data corresponding to the second ordered set that is identical to the first ordered set.
[0168] If there is no second ordered set identical to the first ordered set in the target correspondence, the matching result indicates that there is no second data matching the first data in the full data.
[0169] In some embodiments, the processor 901 is further configured to:
[0170] Based on the target correspondence, N second data that match the first ordered set are determined from the full data, where N is a positive integer greater than or equal to 1;
[0171] Perform regular expression matching on the N second data and the first data to determine the second data that matches the first data from the N second data.
[0172] In some embodiments, the processor 901 is further configured to:
[0173] Based on a pre-stored first correspondence, N service codes that match the first ordered set are determined from the service codes of the full data. The first correspondence is the correspondence between the service code of each second data in the full data and the second ordered set of the second data. The target correspondence includes the first correspondence.
[0174] Based on the pre-stored second correspondence, N second data corresponding to the N service codes are determined from the full data. The second correspondence is the correspondence between the service code of each second data in the full data and the second data. The target correspondence also includes the second correspondence.
[0175] In some embodiments, the processor 901 is further configured to:
[0176] Each second data point in the full dataset is segmented into word groups to obtain multiple word groups corresponding to each second data point.
[0177] Using the M hash functions, a hash operation is performed on each of the word groups to obtain a second ordered set of each of the word groups. The second ordered set of the second data includes the second ordered set of each of the word groups.
[0178] Map each second hash value in the second ordered set of the second data in the full data to the position corresponding to the second hash value in the second position set;
[0179] The target correspondence is established based on the second ordered set of each of the aforementioned word groups and each second data in the full dataset.
[0180] In some embodiments, the processor 901 is further configured to:
[0181] Based on the second ordered set of the second data in the full data, the position corresponding to each second hash value in the second position set of the bit array is mapped to a first value, and the first value is different from the value of other positions in the bit array other than the second position set.
[0182] Preferably, the present invention also provides an electronic device 900, including a processor 901, a memory 902, and a computer program stored in the memory 902 and executable on the processor 901. When the computer program is executed by the processor 901, it implements the various processes of the above-described data matching method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.
[0183] This invention also provides a readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described data matching method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.
[0184] This application also provides a computer program product, including computer instructions. When executed by a processor, the computer instructions implement the various processes of the above-described data matching method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.
[0185] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0186] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0187] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0188] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.
[0189] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0190] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.
[0191] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A data matching method, characterized in that, The method includes: Get the first data to be matched; Using M hash functions, hash operations are performed on the first data to obtain a first ordered set of the first data. The first ordered set is an ordered arrangement of M first hash values corresponding to the M hash functions, where M is a positive integer greater than 1. When each position in the first position set of the pre-stored bit array is mapped to the first hash value, the first data is matched based on the pre-stored target correspondence and the first ordered set to obtain a matching result. The matching result is used to indicate whether there is second data in the full data that matches the first data. The target correspondence is the correspondence between each second data in the full data and the second ordered set of the second data. The second ordered set is the M second hash values that are ordered and correspond one-to-one with the M hash functions. The second ordered set corresponds to the second position set of the bit array.
2. The method according to claim 1, characterized in that, The matching of the first data based on the pre-stored target correspondence and the first ordered set to obtain the matching result includes: If a second ordered set that is identical to the first ordered set exists in the target correspondence, then based on the target correspondence, the second data corresponding to the second ordered set that is identical to the first ordered set is obtained from the full data, and the matching result includes the second data corresponding to the second ordered set that is identical to the first ordered set. If there is no second ordered set identical to the first ordered set in the target correspondence, the matching result indicates that there is no second data matching the first data in the full data.
3. The method according to claim 2, characterized in that, The step of obtaining second data corresponding to a second ordered set that is identical to the first ordered set from the full dataset based on the target correspondence includes: Based on the target correspondence, N second data that match the first ordered set are determined from the full data, where N is a positive integer greater than or equal to 1; Perform regular expression matching on the N second data and the first data to determine the second data that matches the first data from the N second data.
4. The method according to claim 3, characterized in that, The step of determining N second data points that match the first ordered set from the full dataset based on the target correspondence includes: Based on a pre-stored first correspondence, N service codes that match the first ordered set are determined from the service codes of the full data. The first correspondence is the correspondence between the service code of each second data in the full data and the second ordered set of the second data. The target correspondence includes the first correspondence. Based on the pre-stored second correspondence, N second data corresponding to the N service codes are determined from the full data. The second correspondence is the correspondence between the service code of each second data in the full data and the second data. The target correspondence also includes the second correspondence.
5. The method according to claim 1 or 4, characterized in that, Before obtaining the first data to be matched, the method further includes: Each second data point in the full dataset is segmented into word groups to obtain multiple word groups corresponding to each second data point. Using the M hash functions, a hash operation is performed on each of the word groups to obtain a second ordered set of each of the word groups. The second ordered set of the second data includes the second ordered set of each of the word groups. Map each second hash value in the second ordered set of the second data in the full data to the position corresponding to the second hash value in the second position set; The target correspondence is established based on the second ordered set of each of the aforementioned word groups and each second data in the full dataset.
6. The method according to claim 5, characterized in that, The step of mapping each second hash value in the second ordered set of the second data in the full data to the position corresponding to the second hash value in the second position set includes: Based on the second ordered set of the second data in the full data, the position in the second position set of the bit array corresponding to each second hash value is mapped to a first value, and the first value is different from the value of other positions in the bit array other than the second position set.
7. A data matching device, characterized in that, The device includes: The acquisition module is used to acquire the first data to be matched; The first hash processing module is used to perform hash operations on the first data using M hash functions respectively to obtain a first ordered set of the first data. The first ordered set is an ordered arrangement of M first hash values corresponding one-to-one with the M hash functions, where M is a positive integer greater than 1. A matching module is used to match the first data based on a pre-stored target correspondence and the first ordered set, where each position in the first position set of the pre-stored bit array is mapped to the first hash value, to obtain a matching result. The matching result is used to indicate whether there is second data in the full data that matches the first data. The target correspondence is the correspondence between each second data in the full data and the second ordered set of the second data. The second ordered set is the M second hash values that are ordered and correspond one-to-one with the M hash functions. The second ordered set corresponds to the second position set of the bit array.
8. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the data matching method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data matching method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, It includes computer instructions that, when executed by a processor, implement the steps of the data matching method as described in any one of claims 1 to 6.