Data processing methods, apparatus, electronic devices and readable storage media

Abstract

This invention provides a data processing method, apparatus, electronic device, and readable storage medium, relating to the field of computer technology. The method includes: calculating the hash value of each row of data; matching the hash value of each row of data in a Bloom filter; for any hash value, if it matches a test hash value in the hash value cluster corresponding to the current matching state, determining the hash value as a clustered hash value; inputting the clustered hash value into a hash table to record the correspondence between each clustered hash value and a count value in the hash table; traversing all hash values ​​to determine all clustered hash values; determining any test hash value not present in the clustered hash values ​​as an isolated hash value, until all isolated hash values ​​are determined. This invention can reduce the workload of hash tables, improve the performance of time-consuming operators, and improve database query efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] In traditional algorithms, when faced with real-time computation of large datasets, hash table-based data structures are typically used. However, when dealing with large datasets where detailed data cannot be linked and aggregation is limited, hash table-based data structures lead to reduced computational efficiency and decreased performance of aggregation operators, failing to meet actual computational needs. Summary of the Invention

[0003] This invention provides a data processing method, apparatus, electronic device, and readable storage medium to solve the technical problem of low computational efficiency in the face of large amounts of data with low aggregation strength when dealing with large amounts of data details that cannot be connected. It provides a technical solution to improve the performance of time-consuming operators by combining a Bloom filter with an auxiliary hash table.

[0004] In a first aspect, the present invention provides a data processing method, comprising:

[0005] Calculate the hash value of each row of data;

[0006] In the Bloom filter, the hash value of each row of data is matched. For any hash value, if it matches the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is determined as the cluster hash value. The cluster hash value is input into the hash table to record the correspondence between each cluster hash value and the count value in the hash table.

[0007] Iterate through all hash values, identify all clustered hash values, and identify any hash value to be tested that does not exist in the clustered hash values ​​as an isolated hash value, until all isolated hash values ​​are identified;

[0008] The hash value cluster is determined during the process of matching the hash values ​​of each row of data;

[0009] The cluster hash value is the hash value whose count exceeds a preset value;

[0010] The isolated hash value is a hash value whose count does not exceed a preset value.

[0011] According to the data processing method provided by the present invention, after matching the hash value of each row of data in the Bloom filter, the method further includes:

[0012] Build a hash cluster in a Bloom filter;

[0013] For any hash value, if it does not match the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is marked as the hash value to be tested;

[0014] Input the hash value to be tested into the hash value cluster to update the hash value cluster.

[0015] According to the data processing method provided by the present invention, after traversing all hash values, it further includes:

[0016] Input all the hash values ​​to be tested into the hash value cluster to determine the final hash value cluster based on all the hash values ​​to be tested.

[0017] According to the data processing method provided by the present invention, determining any hash value to be tested that does not exist in the clustered hash values ​​as an isolated hash value, until all isolated hash values ​​are determined, includes:

[0018] Determine all cluster hash values;

[0019] Each hash value to be tested in the final hash value cluster is matched with all clustered hash values;

[0020] If any hash value to be tested cannot be matched with any clustered hash value, the hash value to be tested is determined to be an isolated hash value, until all isolated hash values ​​are determined.

[0021] According to the data processing method provided by the present invention, the step of recording the correspondence between each cluster hash value and the count value in the hash table includes:

[0022] If any cluster hash value does not exist in the hash table, create the cluster hash value and establish the correspondence between the cluster hash value and the initial count value;

[0023] If any clustered hash value exists in the hash table, aggregate the clustered hash values.

[0024] According to the data processing method provided by the present invention, the aggregation of the cluster hash value includes:

[0025] Query the count value corresponding to the cluster hash value;

[0026] The updated count value is determined based on the count value and a preset constant.

[0027] According to the data processing method provided by the present invention, after determining all isolated hash values, the method further includes:

[0028] Construct a first correspondence between all isolated hash values ​​and the preset value corresponding to each isolated hash value;

[0029] Construct a second correspondence between all cluster hash values ​​and the corresponding count values ​​for each cluster hash value;

[0030] After determining the first correspondence and the second correspondence as the result set, the result set is output.

[0031] In a second aspect, the present invention provides a data processing apparatus, comprising:

[0032] Calculation unit: Used to calculate the hash value of each row of data;

[0033] First determining unit: used to match the hash value of each row of data in the Bloom filter. For any hash value, if it matches the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is determined as a cluster hash value. The cluster hash value is input into the hash table to record the correspondence between each cluster hash value and the count value in the hash table.

[0034] The second determining unit is used to traverse all hash values, determine all clustered hash values, and determine any hash value to be tested that does not exist in the clustered hash values ​​as an isolated hash value, until all isolated hash values ​​are determined.

[0035] The hash value cluster is determined during the process of matching the hash values ​​of each row of data;

[0036] The cluster hash value is the hash value whose count exceeds a preset value;

[0037] The isolated hash value is a hash value whose count does not exceed a preset value.

[0038] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the data processing methods described above.

[0039] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described above.

[0040] The present invention provides a data processing method, apparatus, electronic device, and readable storage medium. This invention calculates the hash value of each row of data in the original input data. Before inputting the data into a hash table, it searches for the hash value in a Bloom filter. If the hash value matches the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is determined as a clustered hash value. The correspondence between the clustered hash value and the count value is recorded in the hash table. Any hash value to be tested that does not exist in the clustered hash value is determined as an isolated hash value. This process continues until all isolated hash values ​​are determined. Therefore, the hash values ​​used for insertion and aggregation operations in the hash table are hash values ​​after deduplication of isolated hash values, thereby reducing the workload of the hash table, improving the performance of time-consuming operators, and increasing database query efficiency. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is one of the flowcharts illustrating the data processing method provided by the present invention;

[0043] Figure 2 This is the second flowchart of the data processing method provided by the present invention;

[0044] Figure 3 This is a schematic diagram of the process for determining all isolated hash values ​​provided by the present invention;

[0045] Figure 4 This is a flowchart illustrating the correspondence between the hash value and the count value of each cluster, provided by the present invention.

[0046] Figure 5 This is the third flowchart of the data processing method provided by the present invention;

[0047] Figure 6 This is the fourth flowchart of the data processing method provided by the present invention;

[0048] Figure 7 This is a schematic diagram of the data processing device provided by the present invention;

[0049] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] Databases are a product of data management, and their core requirements include data classification, encoding, computation, analysis, and maintenance. From the perspective of database technology development, it has gone through the stages of manual management, file systems, and database systems. The emergence of massive amounts of data is undoubtedly one of the key factors driving database technology development. With societal progress and the surge in historical data, various databases are constantly being optimized and improved to handle large volumes of data, continuously enhancing their computational performance in big data scenarios.

[0052] OLAP (On-Line Analytical Processing) databases connect detailed and summary data in real time according to user requests, perform calculations, and return the results to the user. For real-time calculations involving gigabyte (GB) or terabyte (TB) data volumes, the conventional approach is to use data structures (HashTable, HT) for row processing of time-consuming operators. This leverages the near-O(1) time complexity to improve real-time query efficiency by trading space for time. Examples include JOIN operators (for connecting detailed data) and AGG operators (for summarizing data).

[0053] When dealing with real-time computation of large datasets, the HT data structure is undoubtedly an efficient choice. However, when faced with a large amount of data details that cannot be connected and have very low aggregation capabilities, a large amount of isolated data cannot be connected or aggregated. In other words, this batch of isolated data can be directly output. Using a hash table-based data structure will lead to a decrease in overall computational efficiency.

[0054] To solve the above technical problems, Figure 1 This is one of the flowcharts illustrating the data processing method provided by the present invention. The present invention provides a data processing method, including:

[0055] Calculate the hash value of each row of data;

[0056] In the Bloom filter, the hash value of each row of data is matched. For any hash value, if it matches the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is determined as the cluster hash value. The cluster hash value is input into the hash table to record the correspondence between each cluster hash value and the count value in the hash table.

[0057] Iterate through all hash values, identify all clustered hash values, and identify any hash value to be tested that does not exist in the clustered hash values ​​as an isolated hash value, until all isolated hash values ​​are identified;

[0058] The hash value cluster is determined during the process of matching the hash values ​​of each row of data;

[0059] The cluster hash value is the hash value whose count exceeds a preset value;

[0060] The isolated hash value is a hash value whose count does not exceed a preset value.

[0061] In step 101, raw data input from the terminal is first received. The raw data contains multiple rows of data. Taking each row of data as input, all rows are traversed to calculate the hash value of each row of data. Hash is the process of transforming an input of arbitrary length into a fixed-length output through a hash algorithm. This output is the hash value. This transformation is a compression mapping. The hash algorithm can convert a piece of data into a flag, and this flag is closely related to each byte of the source data. This invention calculates the hash value of each row of data through an aggregation operator, thereby classifying different groups according to different hash values. That is, grouping according to certain rules, dividing a dataset into several small regions according to certain rules, and then processing the data for several small regions.

[0062] In step 102, the hash value of each row of data is matched in the Bloom filter. For any hash value, if it matches the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is determined as a cluster hash value. The cluster hash value is input into the hash table to record the correspondence between each cluster hash value and the count value in the hash table. The Bloom filter is a long binary vector and a series of random mapping functions. The Bloom filter can be used to retrieve whether an element is in a set.

[0063] In the initial stage of a Bloom filter, there are no hash values ​​in the Bloom filter. However, as the matching continues, for example, when the Bloom filter first matches the hash value of a row of data, since the Bloom filter does not record the hash value, the hash value is input into the Bloom filter so that a successful match can be achieved in the Bloom filter the next time the hash value is encountered.

[0064] The hash value cluster is determined during the process of matching the hash values ​​of each row of data. The hash value cluster contains one or more hash values ​​to be tested. The hash value cluster is dynamically updated during the process of matching the hash values ​​of each row of data. That is, in the current state, if a hash value is different from all the hash values ​​to be tested in the hash value cluster in the current state, then the hash value is determined as the hash value to be tested, thereby expanding the hash value cluster so that when the hash value is input again, it can be successfully matched in the hash value cluster. Then, if it matches the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is determined as the cluster hash value and input into the hash table.

[0065] The cluster hash value is a hash value whose count exceeds a preset value, which can optionally be 1. In this case, since the hash value has been matched twice in the Bloom filter, the number of groups corresponding to this hash value must be greater than or equal to 2. Thus, such a hash value is determined as a cluster hash value. The present invention aims to filter out hash values ​​whose count of the group corresponding to the hash value is less than 2. For hash values ​​whose count is greater than or equal to 2, they need to be input into a hash table, and subsequent mapping relationship processing is completed in the hash table. In the process of continuously inputting hash values ​​into the Bloom filter and inputting them into the hash table after filtering, the correspondence between each cluster hash value and the count value is recorded in the hash table.

[0066] In step 103, all hash values ​​are traversed to determine all clustered hash values. Any hash value not present in the clustered hash values ​​is identified as an isolated hash value. This process continues until all isolated hash values ​​are identified. An isolated hash value is a hash value whose count does not exceed a preset value, which can optionally be 1. First, all hash values ​​need to be traversed, i.e., all hash values ​​are input to the Bloom filter to determine all clustered hash values. At this point, the expansion of the hash values ​​to be tested is also completed, and all hash values ​​to be tested are identified. Obviously, the group category of the hash values ​​to be tested in the Bloom filter is greater than the group category corresponding to the clustered hash values. Therefore, any hash value not present in the clustered hash values ​​is identified as an isolated hash value. Since all hash values ​​that can become clustered hash values ​​have been filtered by the Bloom filter and successfully identified as having a count of at least two, the hash values ​​that fail to become clustered hash values ​​are isolated hash values, i.e., hash values ​​with a count of 1. The preset value can also be adjusted according to actual application needs, such as setting it to 2, 3 or other values, which will not be elaborated here.

[0067] The present invention provides a data processing method, apparatus, electronic device, and readable storage medium. This invention calculates the hash value of each row of data in the original input data. Before inputting the data into a hash table, it searches for the hash value in a Bloom filter. If the hash value matches the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is determined as a clustered hash value. The correspondence between the clustered hash value and the count value is recorded in the hash table. Any hash value to be tested that does not exist in the clustered hash value is determined as an isolated hash value. This process continues until all isolated hash values ​​are determined. Therefore, the hash values ​​used for insertion and aggregation operations in the hash table are hash values ​​after deduplication of isolated hash values, thereby reducing the workload of the hash table, improving the performance of time-consuming operators, and increasing database query efficiency.

[0068] Figure 2 This is a second flowchart illustrating the data processing method provided by the present invention. After matching the hash value of each row of data in the Bloom filter, it further includes:

[0069] Build a hash cluster in a Bloom filter;

[0070] For any hash value, if it does not match the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is marked as the hash value to be tested;

[0071] Input the hash value to be tested into the hash value cluster to update the hash value cluster.

[0072] In step 201, for a Bloom filter that is ready to process data but has not yet started processing, in the initial stage of data processing, the Bloom filter does not build a hash value cluster, or even has a hash value to be tested. However, during the data processing, as hash values ​​are input into the Bloom filter, the Bloom filter performs filtering operations while updating the hash value cluster.

[0073] In step 202, for any hash value, the Bloom filter that matches it must be the Bloom filter corresponding to the hash value cluster in the current matching state. In the real-time matching process, any hash value is matched or not matched. When the match is successful, the hash value is further input into the hash table to complete the construction of the correspondence between hash values ​​and count values ​​in the hash table. When the match is unsuccessful, the hash value corresponding to the unsuccessful match is determined as the hash value to be tested. Optionally, the marking process can be implemented based on marking the data rows in the original data, or it can be implemented in the aggregation operator.

[0074] In step 203, if the hash value to be tested does not match the hash value in the hash value cluster corresponding to the current matching state, the hash value is marked as the hash value to be tested. At the same time, the hash value to be tested is input into the hash value cluster of the Bloom filter, thereby updating the hash value cluster.

[0075] Optionally, after traversing all hash values, the following is also included:

[0076] Input all the hash values ​​to be tested into the hash value cluster to determine the final hash value cluster based on all the hash values ​​to be tested.

[0077] The process of inputting hash values ​​is actually a process of finding the corresponding hash table and updating the hash value cluster. This continues until all hash values ​​are traversed, all hash values ​​to be tested are determined, and all hash values ​​to be tested are input into the hash value cluster. The final hash value cluster is determined based on all hash values ​​to be tested. The final hash value cluster is the group category of all hash values ​​corresponding to the original data after traversing all hash values.

[0078] Those skilled in the art will understand that, unlike existing technologies, the process of adding a hash value to be tested in a Bloom filter is a direct and simple input process. Compared with the process of forming a correspondence between hash values ​​and count values ​​in a hash table, its processing efficiency is high, it can effectively filter out isolated hash values, reduce the processing burden of the hash table, and thus improve the overall computational efficiency.

[0079] Figure 3 This is a flowchart illustrating the process of determining all isolated hash values ​​provided by the present invention. The step of determining any hash value to be tested that does not exist in the clustered hash values ​​as an isolated hash value, until all isolated hash values ​​are determined, includes:

[0080] Determine all cluster hash values;

[0081] Each hash value to be tested in the final hash value cluster is matched with all clustered hash values;

[0082] If any hash value to be tested cannot be matched with any clustered hash value, the hash value to be tested is determined to be an isolated hash value, until all isolated hash values ​​are determined.

[0083] In step 1031, after traversing all hash values, all clustered hash values ​​are determined. The clustered hash values ​​are the set of hash values ​​in the hash table that have a correspondence between hash values ​​and count values.

[0084] In step 1032, after all clustered hash values ​​are determined, each hash value to be tested in the final hash value cluster is matched with all clustered hash values, that is, each hash value to be tested is matched with all clustered hash values ​​to determine whether there is a corresponding hash value in the clustered hash values ​​that corresponds to the hash value to be tested. All hash values ​​to be tested are traversed until the matching of all hash values ​​to be tested is completed.

[0085] In step 1033, for any hash value to be tested, if it matches any clustered hash value, then the clustered hash value corresponding to the hash value to be tested is considered to be a hash value that has been filtered by the Bloom filter and has formed a corresponding relationship in the hash table. If it cannot match any clustered hash value, then the hash value to be tested is determined to be an isolated hash value. This process continues until all isolated hash values ​​are determined. The isolated hash values ​​are those that have been intercepted by the Bloom filter. In the subsequent hash value input process, no identical hash value is found that is the same as the isolated hash value. Then, the isolated hash values ​​are determined to be those data with a large amount of data details that cannot be connected and have low aggregation granularity. These data are output directly as separate data without the need to form a corresponding relationship in the hash table. Through step 1033, all isolated hash values ​​are determined.

[0086] Figure 4 This is a flowchart illustrating the process of recording the correspondence between each cluster hash value and the count value provided by the present invention. Recording the correspondence between each cluster hash value and the count value in the hash table includes:

[0087] If any cluster hash value does not exist in the hash table, create the cluster hash value and establish the correspondence between the cluster hash value and the initial count value;

[0088] If any clustered hash value exists in the hash table, aggregate the clustered hash values.

[0089] In step 1021, if any cluster hash value does not exist in the hash table, it is considered that the cluster hash value is appearing in the hash table for the first time. However, since the cluster hash value that can appear in the hash table has already been recorded in the Bloom filter before, that is, it is the second time the cluster hash value has appeared, the initial count value can be set to 2 at this time, while in other embodiments it can also be set to 1.

[0090] In step 1022, if any cluster hash value exists in the hash table, the cluster hash values ​​are aggregated. If any cluster hash value exists in the hash table, it is considered that the cluster hash value already exists in the current hash table, and the count value corresponding to the hash value in the cluster can be updated through the aggregation operation.

[0091] Optionally, the aggregation of the cluster hash values ​​includes:

[0092] Query the count value corresponding to the cluster hash value;

[0093] The updated count value is determined based on the count value and a preset constant.

[0094] In the process of aggregating the cluster hash values, the count value corresponding to the cluster hash value is first queried. For example, if the count value corresponding to a certain cluster hash value is 6, and the preset constant is set to 1, then the updated count value is determined to be 7 based on the sum of the count value and the preset constant. When the cluster hash value is matched again the next time, the count value is updated to 8, and so on.

[0095] Figure 5 This is the third flowchart of the data processing method provided by the present invention. After determining all isolated hash values, it further includes:

[0096] Construct a first correspondence between all isolated hash values ​​and the preset value corresponding to each isolated hash value;

[0097] Construct a second correspondence between all cluster hash values ​​and the corresponding count values ​​for each cluster hash value;

[0098] After determining the first correspondence and the second correspondence as the result set, the result set is output.

[0099] In step 301, if the preset value is 1, then the first correspondence can be constructed in the form of tuples, such as (isolated hash value a, 1), (isolated hash value m, 1), etc. The "isolated hash value a" is the isolated hash value, and "1" is the preset value corresponding to the isolated hash value. (isolated hash value a, 1) is the first correspondence.

[0100] In step 302, the count value corresponds to each cluster hash value, and a second correspondence can be constructed as follows: (cluster hash value b, 6) (cluster hash value n, 21), where "cluster hash value b" is the cluster hash value, and "6" is the count value corresponding to the isolated hash value. (cluster hash value b, 6) is the second correspondence.

[0101] After determining the first correspondence and the second correspondence as a result set, the result set is output. Optionally, all first correspondences are determined, all second correspondences are determined, all first correspondences and all second correspondences are determined as a result set, and the result set is output as the data processing result.

[0102] Figure 6 This is the fourth flowchart of the data processing method provided by the present invention, as shown below. Figure 6 As shown, the user executor inputs data into the aggregation operator HashAggXO. In the loop process, each row of data is traversed, a hash value is calculated, and the hash value is searched in the Bloom filter. If the hash value is not found, it is inserted into the Bloom filter, and the row corresponding to the hash value is marked. If the hash value is successfully found, it is entered into the hash table HashTab. If the hash value does not exist in the hash table, it is inserted into the hash table. If the hash value exists in the hash table, the aggregation operation AGG(data) is performed.

[0103] After processing all hash values ​​in the Bloom filter and hash table, it iterates through all marked rows, matches them with the data in the hash table, and directly adds marked rows that do not exist in the hash table to the output result set, and finally outputs the result.

[0104] Figure 7 This is a schematic diagram of the data processing device provided by the present invention. The present invention discloses a data processing device, including a calculation unit 1: used to calculate the hash value of each row of data. The working principle of the calculation unit 1 can be referred to the aforementioned step 101, and will not be repeated here.

[0105] The data processing device further includes a first determining unit 2: used to match the hash value of each row of data in the Bloom filter. For any hash value, if it matches the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is determined as a cluster hash value. The cluster hash value is input into the hash table to record the correspondence between each cluster hash value and the count value in the hash table. The working principle of the first determining unit 2 can be referred to the aforementioned step 102, and will not be repeated here.

[0106] The data processing device further includes a second determining unit 3: used to traverse all hash values, determine all clustered hash values, and determine any hash value to be tested that does not exist in the clustered hash values ​​as an isolated hash value, until all isolated hash values ​​are determined. The working principle of the second determining unit 3 can be referred to the aforementioned step 103, and will not be repeated here.

[0107] The hash value cluster is determined during the process of matching the hash values ​​of each row of data;

[0108] The cluster hash value is the hash value whose count exceeds a preset value;

[0109] The isolated hash value is a hash value whose count does not exceed a preset value.

[0110] The present invention provides a data processing method, apparatus, electronic device, and readable storage medium. This invention calculates the hash value of each row of data in the original input data. Before inputting the data into a hash table, it searches for the hash value in a Bloom filter. If the hash value matches the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is determined as a clustered hash value. The correspondence between the clustered hash value and the count value is recorded in the hash table. Any hash value to be tested that does not exist in the clustered hash value is determined as an isolated hash value. This process continues until all isolated hash values ​​are determined. Therefore, the hash values ​​used for insertion and aggregation operations in the hash table are hash values ​​after deduplication of isolated hash values, thereby reducing the workload of the hash table, improving the performance of time-consuming operators, and increasing database query efficiency.

[0111] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. For example... Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a data processing method, which includes: calculating the hash value of each row of data; matching the hash value of each row of data in a Bloom filter; for any hash value, if it matches the hash value to be tested in the hash value cluster corresponding to the current matching state, determining the hash value as a cluster hash value; inputting the cluster hash value into a hash table to record the correspondence between each cluster hash value and a count value in the hash table; traversing all hash values, determining all cluster hash values, and determining any hash value to be tested that does not exist in the cluster hash values ​​as an isolated hash value, until all isolated hash values ​​are determined; the hash value cluster is determined during the process of matching the hash value of each row of data; the cluster hash value is the hash value whose count value exceeds a preset value; the isolated hash value is the hash value whose count value does not exceed a preset value.

[0112] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute a data processing method provided by the above methods. The method includes: calculating the hash value of each row of data; matching the hash value of each row of data in a Bloom filter; for any hash value, if it matches the hash value to be tested in the hash value cluster corresponding to the current matching state, determining the hash value as a cluster hash value; inputting the cluster hash value into a hash table to record the correspondence between each cluster hash value and a count value in the hash table; traversing all hash values, determining all cluster hash values, and determining any hash value to be tested that does not exist in the cluster hash values ​​as an isolated hash value, until all isolated hash values ​​are determined; the hash value cluster is determined in the process of matching the hash value of each row of data; the cluster hash value is the hash value whose count value exceeds a preset value; the isolated hash value is the hash value whose count value does not exceed a preset value.

[0114] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the data processing methods provided by the above methods. The method includes: calculating the hash value of each row of data; matching the hash value of each row of data in a Bloom filter; for any hash value, if it matches a test hash value in the hash value cluster corresponding to the current matching state, determining the hash value as a cluster hash value; inputting the cluster hash value into a hash table to record the correspondence between each cluster hash value and a count value in the hash table; traversing all hash values, determining all cluster hash values, and determining any test hash value not present in the cluster hash values ​​as an isolated hash value, until all isolated hash values ​​are determined; the hash value cluster is determined during the process of matching the hash value of each row of data; the cluster hash value is a hash value whose count value exceeds a preset value; the isolated hash value is a hash value whose count value does not exceed a preset value.

[0115] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0116] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that, include: Calculate the hash value of each row of data; In the Bloom filter, the hash value of each row of data is matched. For any hash value, if it matches the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is determined as the cluster hash value. The cluster hash value is input into the hash table to record the correspondence between each cluster hash value and the count value in the hash table. Iterate through all hash values, identify all clustered hash values, and identify any hash value to be tested that does not exist in the clustered hash values ​​as an isolated hash value, until all isolated hash values ​​are identified; The hash value cluster is determined during the process of matching the hash values ​​of each row of data; The cluster hash value is the hash value whose count exceeds a preset value; The isolated hash value is a hash value whose count does not exceed a preset value; After matching the hash value of each row of data in the Bloom filter, the following is also included: A hash value cluster is constructed in the Bloom filter. When the Bloom filter first matches the hash value of the row data, since the Bloom filter does not record the hash value, the hash value is input into the Bloom filter and used as the first hash value to be tested in the hash value cluster. For any hash value, if it does not match the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is marked as the hash value to be tested; Input the hash value to be tested into the hash value cluster to update the hash value cluster.

2. The data processing method according to claim 1, characterized in that, After iterating through all hash values, it also includes: Input all hash values ​​to be tested into the hash value cluster to determine the final hash value cluster based on all hash values ​​to be tested; All hash values ​​to be tested are determined as follows: during the input of hash values, the corresponding hash table is searched, the hash value cluster is updated, and so on until all hash values ​​are traversed, thus determining all hash values ​​to be tested.

3. The data processing method according to claim 2, characterized in that, The step of identifying any hash value to be tested that does not exist in the clustered hash values ​​as an isolated hash value, until all isolated hash values ​​are identified, includes: Determine all cluster hash values; Each hash value to be tested in the final hash value cluster is matched with all clustered hash values; If any hash value to be tested cannot be matched with any clustered hash value, the hash value to be tested is determined to be an isolated hash value, until all isolated hash values ​​are determined.

4. The data processing method according to claim 1, characterized in that, The process of recording the correspondence between each cluster hash value and its count value in the hash table includes: If any cluster hash value does not exist in the hash table, create the cluster hash value and establish the correspondence between the cluster hash value and the initial count value; If any cluster hash value exists in the hash table, aggregate the cluster hash values; The aggregation of the cluster hash values ​​includes: Query the count value corresponding to the cluster hash value; The updated count value is determined based on the count value and a preset constant, where the preset constant is 1.

5. The data processing method according to claim 1, characterized in that, After identifying all isolated hash values, the following is also included: Construct a first correspondence between all isolated hash values ​​and the preset value corresponding to each isolated hash value; Construct a second correspondence between all cluster hash values ​​and the corresponding count values ​​for each cluster hash value; After determining the first correspondence and the second correspondence as the result set, the result set is output.

6. A data processing apparatus, characterized in that, include: Calculation unit: Used to calculate the hash value of each row of data; First determining unit: used to match the hash value of each row of data in the Bloom filter. For any hash value, if it matches the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is determined as a cluster hash value. The cluster hash value is input into the hash table to record the correspondence between each cluster hash value and the count value in the hash table. The second determining unit is used to traverse all hash values, determine all clustered hash values, and determine any hash value to be tested that does not exist in the clustered hash values ​​as an isolated hash value, until all isolated hash values ​​are determined. The hash value cluster is determined during the process of matching the hash values ​​of each row of data; The cluster hash value is the hash value whose count exceeds a preset value; The isolated hash value is a hash value whose count does not exceed a preset value; After matching the hash value of each row of data in the Bloom filter, the first determining unit: is used for A hash value cluster is constructed in the Bloom filter. When the Bloom filter first matches the hash value of the row data, since the Bloom filter does not record the hash value, the hash value is input into the Bloom filter and used as the first hash value to be tested in the hash value cluster. For any hash value, if it does not match the hash value to be tested in the hash value cluster corresponding to the current matching state, the hash value is marked as the hash value to be tested; Input the hash value to be tested into the hash value cluster to update the hash value cluster.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data processing method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 5.

Images