Data filtering method and device, equipment, storage medium and program product

Abstract

The invention discloses a data filtering method and device, equipment, a storage medium and a program product, and the specific technical scheme comprises the steps that a connection query instruction is received, the connection query instruction comprises table names of a first data table and a second data table and non-equivalent filtering conditions, and the non-equivalent filtering conditions are used for representing a data filtering range; under the condition that the data size difference between the first data table and the second data table is larger than a preset threshold value, a data range corresponding to the non-equivalent filtering condition is constructed according to the non-equivalent filtering condition and the first data table, and the data size of the first data table is smaller than that of the second data table; and filtering the data in the second data table according to the data range. In this way, the data size in the second data table can be reduced, and therefore occupied computing resources and running time are reduced.

Description

Technical Field

[0001] The present application belongs to the field of big data technology, and in particular relates to a method, apparatus, device, storage medium and program product for data filtering. Background Art

[0002] When performing data queries in a distributed database, involving multiple data sets, the data in each data set can be associated using an equivalence condition. An equivalence condition is a pre-set association condition. For example, for a personal location data set and a spatiotemporal data set, the personal location data set stores the location information of an individual at various time points, while the spatiotemporal data set stores both location information and time information. The equivalence condition can be time information, and the device can associate data with the same time in the personal location data set and the spatiotemporal data set based on the pre-set time information.

[0003] In this way, when the device uses the equivalence condition to associate the data in the data set, it needs to scan the entire data in the data set and determine whether the above equivalence condition is met, resulting in more computing resources and longer running time. Summary of the Invention

[0004] The embodiments of the present application provide a data filtering method, apparatus, device, storage medium, and program product that can reduce the amount of data in a second data table, thereby reducing occupied computing resources and running time.

[0005] In a first aspect, an embodiment of the present application provides a method for data filtering, comprising:

[0006] Receive a connection query instruction, the connection query instruction including the table names of the first data table and the second data table and a non-equivalent filtering condition, the non-equivalent filtering condition being used to indicate a data filtering range;

[0007] When the difference in data volume between the first data table and the second data table is greater than a preset threshold, constructing a data range corresponding to the non-equivalent filtering condition according to the non-equivalent filtering condition and the first data table, and the data volume of the first data table is smaller than the data volume of the second data table;

[0008] The data in the second data table is filtered according to the data range.

[0009] In a possible implementation, constructing a data range corresponding to the non-equivalent filtering condition according to the non-equivalent filtering condition and the first data table includes:

[0010] Determine the endpoint value of the decision formula corresponding to the non-equivalent filtering condition according to a preset strategy and the non-equivalent filtering condition;

[0011] Searching the first data table for target data corresponding to the endpoint value;

[0012] According to the target data and the decision formula, a data range corresponding to the non-equivalent filtering condition is constructed.

[0013] In a possible implementation, the non-equivalence filtering condition includes multiple non-equivalence conditions; and constructing a data range corresponding to the non-equivalence filtering condition based on the target data and the decision formula includes:

[0014] For each non-equivalent condition, determining a data range corresponding to the non-equivalent condition according to target data corresponding to the non-equivalent condition;

[0015] For each non-equivalent condition, the non-equivalent conditions are merged according to the data range corresponding to the non-equivalent condition to obtain the data range corresponding to the non-equivalent filtering condition.

[0016] In a possible implementation, after receiving the connection query instruction, the method further includes:

[0017] Obtaining first statistical data of the first data table and second statistical data of the second data table, wherein the first statistical data includes a first estimated data volume corresponding to the first data table, and the second statistical data includes a second estimated data volume corresponding to the second data table;

[0018] comparing the first estimated data volume with a first preset threshold and a second preset threshold;

[0019] comparing the second estimated data volume with the first preset threshold and the second preset threshold;

[0020] When the first estimated data amount is greater than a first preset threshold and the second estimated data amount is less than a second preset threshold, it is determined that the data amount difference between the first data table and the second data table is greater than the preset threshold.

[0021] In a possible implementation, the second statistical data further includes data distribution corresponding to the second data table; and filtering the data in the second data table according to the data range includes:

[0022] Comparing the data distribution corresponding to the second data table with the data range corresponding to the non-equivalent filtering condition;

[0023] Filter the data in the second data table that is within the data range corresponding to the non-equal value filtering condition.

[0024] In a possible implementation, after filtering the data in the second data table according to the data range, the method further includes:

[0025] Acquire connection data in the second data table, where the connection data is data obtained by filtering the second data table;

[0026] A correspondence between the connection data and the data in the first data table is established.

[0027] In a second aspect, an embodiment of the present application provides a data filtering device, comprising:

[0028] A receiving module, configured to receive a connection query instruction, wherein the connection query instruction includes the table names of the first data table and the second data table and a non-equivalent filtering condition, wherein the non-equivalent filtering condition is used to indicate a data filtering range;

[0029] a construction module, configured to construct, in a case where the difference in data volume between the first data table and the second data table is greater than a preset threshold, a data range corresponding to the non-equivalent filtering condition according to the non-equivalent filtering condition and the first data table, and the data volume of the first data table is smaller than the data volume of the second data table;

[0030] A filtering module is used to filter the data in the second data table according to the data range.

[0031] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor and a memory storing computer program instructions;

[0032] When the processor executes the computer program instructions, the data filtering method as described in any one of the first aspects is implemented.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the data filtering method as described in any one of the first aspects is implemented.

[0034] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data filtering method as described in any one of the first aspects.

[0035] A method, apparatus, device, storage medium and program product for data filtering in an embodiment of the present application receives a connection query instruction, and determines the data volume difference between the first data table and the second data table based on the table name carried in the connection query instruction. When the data volume difference is greater than a preset threshold, it indicates that there is a large amount of irrelevant data in the second data table. In order to improve subsequent operation efficiency, the irrelevant data in the second data table can be filtered out in advance. Therefore, according to the non-equivalent filtering condition and the first data table, the data range corresponding to the non-equivalent filtering condition is determined, and the data within the data range is the filtered data. Then, the data in the second data table can be filtered according to the data corresponding to the non-equivalent filtering condition. In this way, the amount of data in the second data table can be reduced, and when the second data table is subsequently searched for data associated with the first data table, the occupied computing resources can be reduced, thereby improving query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0037] Figure 1 This is a flow chart of a data filtering method provided in an embodiment of the present application;

[0038] Figure 2 This is a flow chart of a method for determining a data range provided in an embodiment of the present application;

[0039] Figure 3 is an exemplary schematic diagram of a data filtering method provided in an embodiment of the present application;

[0040] Figure 4 This is a schematic diagram of the structure of a data filtering device provided in an embodiment of the present application;

[0041] Figure 5 This is a structural diagram of an electronic device provided in yet another embodiment of the present application. DETAILED DESCRIPTION

[0042] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0043] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0044] Currently, common dynamic filtering solutions include the following five:

[0045] The first method uses Bloom filters to filter data. Systems that support dynamic Bloom filters, such as Apache Hive, Spark, and Impala, first scan a smaller table with less data when performing a join operation. Then, they generate a Bloom filter based on the data in the JOIN column of the smaller table. When the larger table is scanned, the Bloom filter is used to filter out data that is unlikely to match the smaller table.

[0046] The second method uses a hash table to dynamically filter data. The electronic device first scans the contents of a small data table, then calculates a hash value for the scanned data and stores the calculated hash value in a hash table. Later, when scanning a large data table, the electronic device calculates the hash value for each piece of data in the table and compares the hash value of each piece of data in the table with the hash value in the hash table, enabling rapid filtering and search of data in the table.

[0047] The third method uses partition pruning to dynamically filter data. The Apache Spark system first performs a subquery calculation on the data table on one side of the JOIN operation, and then uses the partition information corresponding to the subquery result as a filter condition on the data table on the other side of the JOIN operation to filter data.

[0048] The fourth method is to filter data using a scan node. This uses PostgreSQL's JOIN FilterPushdown to push the filter conditions in the join condition down to the scan node, which reduces the amount of data to be processed.

[0049] Fifth, when executing a query operation, the electronic device generates a scan filter condition according to the filter condition of the small data table, and then uses the scan filter condition to scan data in the large data table to filter the data.

[0050] It should be noted that the above dynamic filtering solutions are applicable to equijoin scenarios, where the join keys of the two datasets are equal. In other words, the above filtering conditions are used to query the corresponding data in the two datasets. However, in non-equijoin scenarios, such as when the filtering conditions include the greater than (>) or less than (<) symbol, the above common dynamic filtering solutions are not applicable.

[0051] In order to solve the problems existing in the prior art, the embodiments of the present application provide a method, apparatus, device, storage medium and program product for data filtering. Figure 1 Introducing a data filtering method provided by an embodiment of the present application, such as Figure 1 As shown, the method is applied to an electronic device, and the method includes:

[0052] S101: Receive a connection query instruction.

[0053] The connection query instruction includes the table names of the first data table and the second data table and a non-equivalent filtering condition, where the non-equivalent filtering condition is used to indicate a data filtering range.

[0054] Specifically, the connection query instruction is used to instruct the electronic device to simultaneously query corresponding data in the first data table and the second data table. The non-equivalence filtering condition can be a query statement input by the user, and the query statement includes a non-equivalence condition. For example, the non-equivalence condition can be greater than, less than, not equal to, greater than or less than, etc.

[0055] In an example, the query statement may be select * from T_A, T_B where T_B.b1>T_A.a1and T_B.b1>T_A.a2 and T_B.b2<(T_A.a3+T_A.a4).

[0056] Where T_A is the name of the first data table, T_B is the name of the second data table, bn is the nth column in the first data table, and an is the nth column in the second data table. Specifically, the non-equal value filtering condition is that the data in column b1 of table T_B is greater than the data in column a1 of table T_A, the data in column b1 of table T_B is greater than the data in column a2 of table T_A, and the data in column b2 of table T_B is less than the sum of the data in columns a3 and a4 of table T_A.

[0057] S102: When the data volume difference between the first data table and the second data table is greater than a preset threshold, construct a data range corresponding to the non-equivalent filtering condition according to the non-equivalent filtering condition and the first data table.

[0058] The amount of data in the first data table is smaller than that in the second data table. Therefore, if the difference in the amount of data in the first data table and the second data table is greater than a preset threshold, it indicates that the amount of data in the first data table and the second data table is sufficiently large. To improve query efficiency, the data in the second data table can be pre-trimmed, that is, the data in the second data table can be filtered.

[0059] S103: Filter the data in the second data table according to the data range.

[0060] Using the method provided in the embodiment of the present application, a connection query instruction is received, and the data volume difference between the first data table and the second data table is determined based on the table name carried in the connection query instruction. When the data volume difference is greater than a preset threshold, it indicates that a large amount of irrelevant data exists in the data in the second data table. In order to improve subsequent operation efficiency, the irrelevant data in the second data table can be filtered out in advance. Therefore, according to the non-equivalent filtering condition and the first data table, the data range corresponding to the non-equivalent filtering condition is determined, and the data within the data range is the filtered data. Then, the data in the second data table can be filtered according to the data corresponding to the non-equivalent filtering condition. In this way, the data in the second data table can be reduced, and when the second data table is subsequently searched for data associated with the first data table, the occupied computing resources can be reduced, thereby improving query efficiency.

[0061] It should be noted that, in order to improve query efficiency, after receiving the connection query instruction, the electronic device can determine the data volume difference between the first data table and the second data table. If the data volume difference is large, it means that there is a lot of irrelevant data in the data table with a larger data volume, and thus the data in the data table with a larger data volume is filtered to improve subsequent query efficiency. Specifically, after receiving the connection query instruction in the above S101, the method includes:

[0062] Step A: Obtain first statistical data of the first data table and second statistical data of the second data table.

[0063] The first statistical data includes a first estimated data volume corresponding to the first data table, and the second statistical data includes a second estimated data volume corresponding to the second data table.

[0064] It should be noted that the database stores the first data table and the second data table, and the first statistical data of the first data table and the second statistical data of the second data table can be updated periodically.

[0065] Step B: comparing the first estimated data volume with the first preset threshold and the second preset threshold.

[0066] The first preset threshold and the second preset threshold are preset based on experience.

[0067] For the first preset threshold, if the amount of data in the data table is greater than the first preset threshold, it is considered that pre-filtering the data can improve the query efficiency of the electronic device. If the amount of data in the data table is less than the second preset threshold, it is considered that directly scanning the data table can improve the query efficiency of the electronic device.

[0068] Step C: comparing the second estimated data volume with the first preset threshold and the second preset threshold.

[0069] Step D: When the first estimated data volume is greater than a first preset threshold and the second estimated data volume is less than a second preset threshold, determine that the data volume difference between the first data table and the second data table is greater than the preset threshold.

[0070] It should be noted that, for the above two data tables, when the estimated data volume corresponding to one data table is greater than the first preset threshold and the estimated data volume corresponding to the other data table is less than the second preset threshold, the electronic device determines to filter the data table with larger data volume in advance.

[0071] Using the method provided in an embodiment of the present application, first statistical data from a first data table and second statistical data from a second data table are obtained. In this way, the relationship between the first estimated data volume and the first and second preset thresholds can be compared, as can the relationship between the second estimated data volume and the first and second preset thresholds. This allows for a determination of whether the difference between the data volume in the first and second data tables is greater than a preset threshold. If the data volume difference is greater than the preset threshold, this indicates that the data in the data tables can be filtered to improve subsequent query efficiency.

[0072] In some embodiments of the present application, in the above S102, according to the non-equivalent filtering condition and the first data table, a data range corresponding to the non-equivalent filtering condition is constructed, such as Figure 2 As shown, it can be implemented as follows:

[0073] S201: Determine the endpoint value of the decision formula corresponding to the non-equivalent filtering condition according to a preset strategy and the non-equivalent filtering condition.

[0074] As can be understood, the connection query instruction includes a query statement that includes non-equivalence filtering conditions. Based on the non-equivalence filtering conditions, the decision formula corresponding to each non-equivalence filtering condition can be obtained. However, the electronic device cannot recognize the decision formula. Therefore, according to a preset strategy, the decision formula is converted into a form that the electronic device can recognize, thereby obtaining the endpoint values of the decision formula. Based on the endpoint values and the decision formula, the data range can be determined.

[0075] S202: Search the first data table for target data corresponding to the endpoint value.

[0076] S203: Construct a data range corresponding to the non-equivalent filtering condition according to the target data and the decision formula.

[0077] Continuing with the above example, if the query statement is select * from T_A, T_B where T_B.b1>T_A.a1 andT_B.b1>T_A.a2 and T_B.b2<(T_A.a3+T_A.a4), the decision formula corresponding to the above non-equal value filtering condition is T_B.b1>T_A.a1 and T_B.b1>T_A.a2 and T_B.b2<(T_A.a3+T_A.a4). According to the preset strategy, the above query statement is converted to: the value of column b1 of table T_B is greater than the minimum value of column a1 of table T_A, and the value of column b1 of table T_B is greater than the minimum value of column a2 of table T_A, and the value of column b2 of table T_B is less than the maximum sum of columns a3 and a4 of table T_A. That is, the endpoint values of the decision formula are the minimum value of column a1 of table T_A, the minimum value of column a2 of table T_A, and the maximum value of the sum of columns a3 and a4 of table T_A.

[0078] Using the method provided in the embodiments of the present application, non-equivalence filter conditions can be converted into a decision formula that can be recognized by the device according to a preset strategy, thereby obtaining the endpoint value of the decision formula corresponding to the non-equivalence filter condition. Based on this endpoint value, the corresponding target data is then searched in the first data table. Finally, using the decision formula and the target data corresponding to the decision formula, the data range of the non-equivalence filter condition can be determined. This data range represents the data to be filtered. In this way, the data to be filtered in the second data table can be determined based on the non-equivalence filter condition and the data in the first data table, ensuring the accuracy of the calculation results.

[0079] Specifically, if the non-equivalence filtering condition may include multiple non-equivalence conditions, the non-equivalence conditions with overlapping data can be determined based on the data range corresponding to each non-equivalence condition, thereby merging multiple non-equivalence conditions to improve the query efficiency of the electronic device. Based on this, the above S203, constructing the data range corresponding to the non-equivalence filtering condition based on the target data and the decision formula, can be implemented as follows:

[0080] Step 1: For each non-equivalence condition, determine the data range corresponding to the non-equivalence condition according to the target data corresponding to the non-equivalence condition.

[0081] Among them, each judgment formula in the above non-equivalence filtering condition corresponds to a non-equivalence condition. Correspondingly, by using the method provided in the above embodiment, the data range corresponding to each non-equivalence condition can be determined.

[0082] Step 2: For each non-equivalence condition, merge the non-equivalence conditions according to the data range corresponding to the non-equivalence condition to obtain the data range corresponding to the non-equivalence filtering condition.

[0083] It can be understood that there is an overlap between the non-equivalence conditions and the data ranges corresponding to the non-equivalence conditions. Therefore, the non-equivalence conditions can be merged.

[0084] Continuing the above example, when the non-equivalence filtering condition is T_B.b1>min(a1) and T_B.b1>min(a2) and T_B.b2<max(a3 + a4), for the b1 column in T_B, there are two non-equivalence conditions, T_B.b1>min(a1) and T_B.b1>min(a2). Therefore, the electronic device can respectively obtain the minimum value of the a1 column and the minimum value of the a2 column in the T_A table, and then compare the magnitudes of the minimum value of the a1 column and the minimum value of the a2 column, so as to merge the non-equivalence conditions. Specifically, when min(a1) >= min(a2), filtering the non-equivalence conditions, the non-equivalence filtering condition can be obtained as T_B.b1>min(a1) and T_B.b2<max(a3 + a4).

[0085] By using the method provided in the embodiment of the present application, when there are multiple non-equivalence conditions in the non-equivalence filtering condition, the data range corresponding to each non-equivalence condition is respectively confirmed, and then the non-equivalence conditions are merged according to the data range corresponding to each non-equivalence condition, thereby reducing the number of non-equivalence conditions in the non-equivalence filtering condition. When the electronic device subsequently uses the non-equivalence conditions to filter the data in the second data table, the computing resources occupied by filtering the data are reduced.

[0086] After the electronic device merges the non-equivalence conditions to obtain the data range corresponding to the non-equivalence filtering condition, the electronic device can use the data range corresponding to the non-equivalence filtering condition to filter the data in the second data table. Specifically, the second statistical data further includes the data distribution corresponding to the second data table. The above S103, filtering the data in the second data table according to the data range, can be implemented as:

[0087] Compare the data distribution corresponding to the second data table with the data range corresponding to the non-equivalent filtering condition; filter the data in the second data table that falls within the data range corresponding to the non-equivalent filtering condition.

[0088] Among them, the data distribution corresponding to the second data table is used to represent the number of data in the second data table within different data ranges.

[0089] Specifically, the electronic device can obtain the data distribution of the second data table by reading the statistical data of the second data table or through dynamic sampling technology, so as to construct a data distribution histogram corresponding to the second data table. The electronic device compares according to the data distribution histogram and the non-equivalent filtering condition. And, the execution order of the non-equivalent filtering condition can be optimized, and the non-equivalent filtering condition with a higher clipping rate is preferentially used to filter the data in the second data table. Among them, the clipping rate is used to represent the magnitude of the number of filtered data.

[0090] Continuing the above example, the merged non-equivalent filtering condition is T_B.b1>min(a1) and T_B.b2<max(a3+a4). The electronic device obtains the data distribution of the T_B table through the statistical data of the T_B table and constructs a data distribution histogram according to the data distribution of the T_B table. The electronic device can obtain the data distribution of each column in the T_B table according to the data distribution histogram, so as to compare the data in the b1 column of the T_B table with min(a1), and compare the data in the b2 column of the T_B table with max(a3+a4), and filter the data in the T_B table.

[0091] In this way, through the data distribution in the second statistical data, the electronic device can obtain the number of data in the second data table within different data intervals, and thus determine the data that can be filtered according to the data range corresponding to the non-equivalent filtering condition, so as to achieve data filtering of the second data table.

[0092] After filtering the data in the data table according to the data range as described above, establish an association relationship between the first data table and the second data table. Specifically, after filtering the data in the second data table according to the data range in S103 as described above, the method further includes:

[0093] Obtain the connection data in the second data table, where the connection data is the data after filtering the second data table; establish the corresponding relationship between the connection data and the data in the first data table.

[0094] By using the method provided in the embodiment of the present application, after filtering the data in the second data table by using the non-equivalent filtering condition as described above, the number of connection data can be reduced. When establishing the corresponding relationship between the connection data and the data in the first data table subsequently, the workload can be reduced and the efficiency can be improved.

[0095] The following combination Figure 3 This paper introduces a method for electronic equipment to pre-filter data in a data table under non-equivalent filtering conditions. Figure 3 As shown, the method includes:

[0096] S301. JOIN operator input.

[0097] S302: JOIN operator input.

[0098] The JOIN operators input into S301 and S302 are the data sets to be queried, that is, the JOIN operators are the first data set and the second data set mentioned above.

[0099] S303: Construct a JOIN operator.

[0100] Here, constructing a JOIN operator means using the above-input JOIN operator as the data table to be queried.

[0101] S304: Set a data volume threshold for generating dynamic filtering conditions.

[0102] The dynamic filtering condition data volume threshold is the first preset threshold and the second preset threshold in the above embodiment.

[0103] S305: Obtain JOIN data set statistics or estimated data volume.

[0104] The statistical information is the statistical data in the above embodiment, and the statistical data includes the estimated data volume corresponding to the data table.

[0105] S306: Check whether there is a data set with a data volume lower than a first data volume threshold in the JOIN data set.

[0106] If yes, execute S308; if no, execute S307.

[0107] The estimated data volume corresponding to the JOIN data set is compared with the data volume threshold to determine whether there is a data set below the data volume threshold.

[0108] S307. Scan the data directly.

[0109] S308: Check whether there is a data set with a data volume higher than a second data volume threshold in the JOIN data set.

[0110] If yes, execute S310; if no, execute S309.

[0111] The estimated data volume corresponding to the JOIN data set is compared with the data volume threshold to determine whether there is a data set below the data volume threshold.

[0112] It should be noted that, in the above two JOIN data sets, when the estimated data volume of one JOIN data set is less than the first data volume threshold and the estimated data volume of the other is greater than the second data volume threshold, it is determined that the data volume difference between the two JOIN data sets is large. In order to improve the query efficiency of subsequent connection operations, when the data volume difference between the two JOIN data sets is large, data filtering can be performed on the JOIN data set with a larger data volume to reduce the data volume of the connection operation.

[0113] S309: directly scan the data set.

[0114] S310 , pre-scan the JOIN data set to obtain information such as max / min of the data set.

[0115] The JOIN dataset is the first dataset in the above embodiment. The electronic device converts the query statement according to a preset strategy to obtain the endpoint values of the decision formula for the non-equal value filtering condition, thereby obtaining information such as the max / min value corresponding to the endpoint values in the first dataset. The specific acquisition method is described in the above embodiment and is not repeated here.

[0116] S311. Generate several filtering conditions based on the JOIN conditions and the statistical data of the small data set.

[0117] The filtering condition is the non-equivalence condition and the corresponding data range in the above embodiment. The specific method of generating the non-equivalence condition and the corresponding data range is described in the above embodiment and will not be repeated here.

[0118] S312: merge repeated conditions in the filtering conditions and adjust the execution order of the filtering conditions.

[0119] The method for merging non-equivalent conditions is described in the above embodiments and will not be repeated here.

[0120] S313. Apply filtering conditions during the data scanning phase to reduce the amount of connection data.

[0121] S314: Perform connection operation.

[0122] By adopting the method provided in the embodiment of the present application, after inputting the JOIN operator, the electronic device can determine whether it is necessary to filter the data in the data set in advance according to the set data volume threshold and the statistical information of the data set to reduce the data volume of the connection operation. In the case of judging that the data volume difference between the two data sets is large, the maximum value and minimum value and other information in the data set can be obtained according to the non-equivalent filtering conditions, so as to construct a data filtering range corresponding to the non-equivalent filtering conditions, and use the data filtering range corresponding to the non-equivalent filtering conditions to filter the data in the data table, thereby reducing the data volume of the connection operation, thereby improving the query efficiency and reducing the consumption of system resources. In addition, the electronic device scans a small data set with a small amount of data to generate a filtering condition, which can significantly reduce the amount of data involved in the connection operation and reduce the consumption of system resources.

[0123] Based on the same concept, the embodiment of the present application provides a data filtering device, such as Figure 4 As shown, the device includes:

[0124] A receiving module 401 is configured to receive a connection query instruction, wherein the connection query instruction includes the table names of the first data table and the second data table and a non-equivalence filtering condition, wherein the non-equivalence filtering condition is used to indicate a data filtering range;

[0125] A construction module 402 is configured to construct a data range corresponding to the non-equivalence filtering condition according to the non-equivalence filtering condition and the first data table when the difference in data volume between the first data table and the second data table is greater than a preset threshold, and the data volume of the first data table is smaller than the data volume of the second data table;

[0126] The filtering module 403 is configured to filter the data in the second data table according to the data range.

[0127] In a possible implementation, the construction module 402 is specifically configured to:

[0128] Determine the endpoint value of the decision formula corresponding to the non-equivalent filtering condition according to a preset strategy and the non-equivalent filtering condition;

[0129] Searching the first data table for target data corresponding to the endpoint value;

[0130] According to the target data and the decision formula, a data range corresponding to the non-equivalent filtering condition is constructed.

[0131] In a possible implementation, the non-equivalence filtering condition includes multiple non-equivalence conditions; the construction module 402 is specifically configured to:

[0132] For each non-equivalent condition, determining a data range corresponding to the non-equivalent condition according to target data corresponding to the non-equivalent condition;

[0133] For each non-equivalent condition, the non-equivalent conditions are merged according to the data range corresponding to the non-equivalent condition to obtain the data range corresponding to the non-equivalent filtering condition.

[0134] In a possible implementation, the device further includes:

[0135] an acquisition module, configured to acquire first statistical data of the first data table and second statistical data of the second data table, wherein the first statistical data includes a first estimated data volume corresponding to the first data table, and the second statistical data includes a second estimated data volume corresponding to the second data table;

[0136] a comparing module, configured to compare the first estimated data volume with a first preset threshold and a second preset threshold;

[0137] a comparing module, further configured to compare the second estimated data volume with the first preset threshold and the second preset threshold;

[0138] The determination module is used to determine that the data volume difference between the first data table and the second data table is greater than the preset threshold when the first estimated data volume is greater than a first preset threshold and the second estimated data volume is less than a second preset threshold.

[0139] In a possible implementation, the second statistical data further includes data distribution corresponding to the second data table; the filtering module 403 is specifically configured to:

[0140] Comparing the data distribution corresponding to the second data table with the data range corresponding to the non-equivalent filtering condition;

[0141] Filter the data in the second data table that is within the data range corresponding to the non-equal value filtering condition.

[0142] In a possible implementation, the device further includes:

[0143] An acquisition module, configured to acquire connection data in the second data table, wherein the connection data is data obtained by filtering the second data table;

[0144] An establishing module is used to establish a corresponding relationship between the connection data and the data in the first data table.

[0145] It should be noted that the data filtering device is a device corresponding to the above-mentioned data filtering method. All implementation methods in the above-mentioned method embodiments are applicable to the embodiments of the device and can achieve the same technical effects.

[0146] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0147] The electronic device may include a processor 501 and a memory 502 storing computer program instructions.

[0148] Specifically, the processor 501 may include a central processing unit (CPU) or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0149] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 502 may include removable or non-removable (or fixed) media. Where appropriate, the memory 502 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 502 is a non-volatile solid-state memory.

[0150] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present disclosure.

[0151] The processor 501 implements any one of the data filtering methods in the above embodiments by reading and executing computer program instructions stored in the memory 502 .

[0152] In one example, the electronic device may further include a communication interface 503 and a bus 504. Figure 5 As shown, the processor 501 , the memory 502 , and the communication interface 503 are connected via a bus 504 and communicate with each other.

[0153] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0154] The bus 504 includes hardware, software, or both that couples components of the electronic device to each other. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Super Transmission (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, the bus 504 may include one or more buses. Although embodiments herein describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

[0155] In addition, in conjunction with the data filtering method in the above embodiments, embodiments of the present application may provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any of the data filtering methods in the above embodiments is implemented.

[0156] In combination with the data filtering method in the above embodiments, an embodiment of the present application may provide a computer program product. When the instructions in the computer program product are executed by the processor of an electronic device, the electronic device executes any data filtering method in the above embodiments.

[0157] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0158] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (erasable read-only memory, EROM), floppy disks, compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical discs, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0159] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0160] Aspects of the present disclosure have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or flowchart and the combination of the boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0161] The above description is only a specific embodiment of the present application. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. A data filtering method, characterized in that: include: Receive a connection query instruction, the connection query instruction including the table names of the first data table and the second data table and a non-equivalent filtering condition, the non-equivalent filtering condition being used to filter the data; When the difference in data volume between the first data table and the second data table is greater than a preset threshold, constructing a data range corresponding to the non-equivalent filtering condition according to the non-equivalent filtering condition and the first data table, and the data volume of the first data table is smaller than the data volume of the second data table; The data in the second data table is filtered according to the data range.

2. The method according to claim 1, characterized in that The step of constructing a data range corresponding to the non-equivalent filtering condition according to the non-equivalent filtering condition and the first data table includes: Determine the endpoint value of the decision formula corresponding to the non-equivalent filtering condition according to a preset strategy and the non-equivalent filtering condition; Searching the first data table for target data corresponding to the endpoint value; According to the target data and the decision formula, a data range corresponding to the non-equivalent filtering condition is constructed.

3. The method according to claim 2, characterized in that The non-equivalence filtering condition includes multiple non-equivalence conditions; constructing a data range corresponding to the non-equivalence filtering condition based on the target data and the decision formula includes: For each non-equivalent condition, determining a data range corresponding to the non-equivalent condition according to target data corresponding to the non-equivalent condition; For each non-equivalent condition, the non-equivalent conditions are merged according to the data range corresponding to the non-equivalent condition to obtain the data range corresponding to the non-equivalent filtering condition.

4. The method according to claim 1, wherein After receiving the connection query instruction, the method further includes: Obtaining first statistical data of the first data table and second statistical data of the second data table, wherein the first statistical data includes a first estimated data volume corresponding to the first data table, and the second statistical data includes a second estimated data volume corresponding to the second data table; comparing the first estimated data volume with a first preset threshold and a second preset threshold; comparing the second estimated data volume with the first preset threshold and the second preset threshold; When the first estimated data amount is greater than a first preset threshold and the second estimated data amount is less than a second preset threshold, it is determined that the data amount difference between the first data table and the second data table is greater than the preset threshold.

5. The method according to claim 4, characterized in that The second statistical data also includes data distribution corresponding to the second data table; and filtering the data in the second data table according to the data range includes: Comparing the data distribution corresponding to the second data table with the data range corresponding to the non-equivalent filtering condition; Filter the data in the second data table that is within the data range corresponding to the non-equal value filtering condition.

6. The method according to claim 1, characterized in that After filtering the data in the second data table according to the data range, the method further includes: Acquire connection data in the second data table, where the connection data is data obtained by filtering the second data table; A correspondence between the connection data and the data in the first data table is established.

7. A data filtering device, characterized in that: include: A receiving module, configured to receive a connection query instruction, wherein the connection query instruction includes the table names of the first data table and the second data table and a non-equivalent filtering condition, wherein the non-equivalent filtering condition is used to indicate a data filtering range; a construction module, configured to construct, in a case where the difference in data volume between the first data table and the second data table is greater than a preset threshold, a data range corresponding to the non-equivalent filtering condition according to the non-equivalent filtering condition and the first data table, and the data volume of the first data table is smaller than the data volume of the second data table; A filtering module is used to filter the data in the second data table according to the data range.

8. An electronic device, characterized in that: The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the data filtering method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the data filtering method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is caused to perform the data filtering method according to any one of claims 1 to 6.

Images