Data query processing method, device, equipment, storage medium and processor

Abstract

Embodiments of the present invention disclose a data query processing method, apparatus, device, storage medium, and processor. The data query processing method includes: determining multiple data sets based on the data to be queried, the data sets including multiple data, the multiple data in the data sets being arranged in order of size; determining the amount of data in each data set and the maximum and / or minimum values ​​of the data in each data set; deleting at least one data set from the multiple data sets based on the sorting interval of the target data, the amount of data in each data set, and the maximum and / or minimum values ​​of the data in each data set; merging the data in the remaining multiple data sets to obtain a sorted file; and obtaining and outputting the target data from the sorted file based on the sorting interval of the target data. The present invention solves the technical problem of slow query speed when querying data within a specific sorting interval in the related art.

Description

Technical Field

[0001] The present invention relates to the field of data query, and in particular to a data query processing method, apparatus, device, storage medium and processor. Background Art

[0002] Data querying is a common operation in database usage. In many query scenarios, it is necessary to retrieve data within a specific sorting range after sorting the data according to certain conditions. For example, in one TopK query scenario, it is necessary to retrieve the top K pieces of data after sorting the data according to specific conditions. In another query scenario, it is necessary to retrieve the top m pieces of data starting from the nth piece after sorting the data according to specific conditions.

[0003] In the related art, when performing the above-mentioned query process for data in a specific sorting interval, it is necessary to sort all the data in the query data, and then obtain the data in the corresponding sorting position for output. In this process, all the data needs to be sorted, which requires a large number of data sorting operations, resulting in low data query efficiency and slow query speed.

[0004] Therefore, in the related art, when querying data within a specific sorting interval, there is a problem of slow query speed. In view of the above problem, no effective solution has been proposed yet.

[0005] The above information disclosed in the Background section is only intended to enhance the understanding of the background technology of the technology described herein. Therefore, the Background section may contain some information that does not form the known prior art for those skilled in the art. Summary of the Invention

[0006] Embodiments of the present invention provide a data query processing method, apparatus, device, storage medium, and processor to at least solve the technical problem in related technologies of slow query speed when querying data within a specific sorting interval.

[0007] According to a first aspect of an embodiment of the present invention, a data query processing method is provided, comprising: determining multiple data sets based on data to be queried, the data sets comprising multiple data, the multiple data in the data sets being arranged in order of size; determining the amount of data in the data sets and the maximum and / or minimum value of the data in the data sets; deleting at least one data set from the multiple data sets based on the sorting interval of the target data, the amount of data in each data set, and the maximum and / or minimum value of the data in each data set; merging the data in the remaining multiple data sets to obtain a sorted file, the sorted file comprising multiple data arranged in order of size; and obtaining the target data from the sorted file based on the sorting interval of the target data and outputting it.

[0008] Furthermore, based on the sorting interval of the target data, the amount of data in each data set, and the maximum and / or minimum value of the data in each data set, deleting at least one data set from multiple data sets includes: for any first threshold, if the sum of the amount of data of the data sets belonging to the first type and the data sets belonging to the second type is less than the lower limit of the sorting interval of the target data, deleting the data set of the first type from the multiple data sets; and / or, for any second threshold, if the sum of the amount of data of the data sets belonging to the third type is greater than the upper limit of the sorting interval of the target data, deleting the data set of the fourth type from the multiple data sets; wherein the first type of data set is a data set whose maximum value of data is less than the first threshold, the second type of data set is a data set whose minimum value of data is less than the first threshold and the maximum value of data is greater than the first threshold, the third type of data set is a data set whose maximum value of data is less than the second threshold, and the fourth type of data set is a data set whose minimum value of data is greater than the second threshold.

[0009] Furthermore, the first threshold is the maximum threshold that satisfies the first goal, and the first goal is: the sum of the data volume of the data set belonging to the first type and the data set belonging to the second type is less than the lower limit of the sorting interval of the target data; and / or, the second threshold is the minimum threshold that satisfies the second goal, and the second goal is: the sum of the data volume of the data set belonging to the third type is greater than the upper limit of the sorting interval of the target data.

[0010] Furthermore, based on the sorting interval of the target data, the amount of data in each data set, and the maximum and / or minimum value of the data in each data set, deleting at least one data set from multiple data sets includes: starting from a first initial value, searching for a first threshold in a direction from small to large; and / or, starting from a second initial value, searching for a second threshold in a direction from large to small.

[0011] Furthermore, in the case of deleting a first type of data set from multiple data sets, obtaining the target data from the sorting file and outputting it according to the sorting interval of the target data includes: determining the sum p of the amount of data in the first type of data set; subtracting the sum p from the upper and lower limits of the sorting interval of the target data to obtain an updated sorting interval; obtaining the data corresponding to the updated sorting interval from the sorting file and outputting it.

[0012] Furthermore, determining multiple data sets according to the data to be queried includes a data set determination step that is executed cyclically, and the data set determination step includes: obtaining a preset number of data from the data to be queried; and sorting the preset number of data in order of size to obtain a data set.

[0013] Furthermore, merging the data in the remaining multiple data sets to obtain a sorted file includes a data merging step that is executed in a loop, and the data merging step includes: obtaining the largest or smallest data from the multiple data sets and adding it to the sorted file in order of size; determining whether the amount of data in the sorted file reaches the upper limit of the sorting interval of the target data; when the amount of data in the sorted file reaches the upper limit of the sorting interval of the target data, ending the data merging operation on the multiple data sets.

[0014] According to a second aspect of an embodiment of the present invention, a data query processing device is also provided, including: a first determination unit, used to determine multiple data sets based on the data to be queried, the data sets including multiple data, and the multiple data in the data sets are arranged in order of size; a second determination unit, used to determine the amount of data in the data sets and the maximum and / or minimum values ​​of the data in the data sets; a deletion unit, used to delete at least one data set from the multiple data sets based on the sorting interval of the target data, the amount of data in each data set, and the maximum and / or minimum values ​​of the data in each data set; a merging unit, used to merge the data in the remaining multiple data sets to obtain a sorted file, the sorted file including multiple data arranged in order of size; an acquisition unit, used to acquire and output the target data from the sorted file based on the sorting interval of the target data.

[0015] Furthermore, the deletion unit includes: a first deletion module, configured to delete a data set of the first type from the multiple data sets if, for any first threshold value, the sum of the data amounts of the data sets belonging to the first type and the data sets belonging to the second type is less than the lower limit of the sorting interval of the target data; and / or, a second deletion module, configured to delete a data set of the fourth type from the multiple data sets if, for any second threshold value, the sum of the data amounts of the data sets belonging to the third type is greater than the upper limit of the sorting interval of the target data; wherein, the data set of the first type is a data set whose maximum value is less than the first threshold value, the data set of the second type is a data set whose minimum value is less than the first threshold value and whose maximum value is greater than the first threshold value, the data set of the third type is a data set whose maximum value is less than the second threshold value, and the data set of the fourth type is a data set whose minimum value is greater than the second threshold value;

[0016] The first threshold is a maximum threshold that satisfies a first objective, where the sum of the data volumes of the datasets belonging to the first type and the datasets belonging to the second type is less than a lower limit of the sorting interval for the target data; and / or the second threshold is a minimum threshold that satisfies a second objective, where the sum of the data volumes of the datasets belonging to the third type is greater than an upper limit of the sorting interval for the target data;

[0017] The deletion unit includes: a search module for searching for a first threshold value in a direction from small to large starting from a first initial value; and / or searching for a second threshold value in a direction from large to small starting from a second initial value;

[0018] The acquisition unit includes: a determination module for determining a total amount p of data in the first type of data set when a first type of data set is deleted from the plurality of data sets; a calculation module for subtracting the total amount p from both the upper and lower limits of the sorting interval of the target data to obtain an updated sorting interval; and an acquisition module for acquiring data corresponding to the updated sorting interval from the sorting file and outputting the data;

[0019] The first determining unit is used to cyclically execute the data set determining step, the data set determining step comprising: obtaining a preset number of data from the data to be queried; sorting the preset number of data in order of size to obtain a data set;

[0020] The merging unit is used to cyclically execute the data merging steps, which include: obtaining the largest or smallest data from multiple data sets and adding it to the sorting file in order of university; determining whether the amount of data in the sorting file reaches the upper limit of the sorting interval of the target data; and ending the data merging operation on the multiple data sets when the amount of data in the sorting file reaches the upper limit of the sorting interval of the target data.

[0021] According to a third aspect of an embodiment of the present invention, a non-volatile storage medium is further provided, wherein the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above-mentioned data query processing method.

[0022] According to a fourth aspect of an embodiment of the present invention, a processor is further provided, wherein the processor is configured to run a program, wherein the program executes the above-mentioned data query processing method when running.

[0023] According to a fifth aspect of an embodiment of the present invention, a data query processing device is also provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data query processing method when executing the computer program.

[0024] In the data query processing method of the embodiment of the present invention, in the process of querying target data, the query data will first be processed to obtain multiple data sets, and the multiple data in the data sets will be arranged in order of size. Then, the data volume and the maximum and / or minimum values ​​of the data in the data sets will be determined. Based on this information, from a macro perspective, at least one data set will be deleted based on the data volume of each data set, the maximum value of the data in each data set, and / or the minimum value of the data in each data set, combined with the sorting interval of the target data to be queried. Since the sorting interval of the target data to be queried is known, the focus is on sorting and outputting the data within this sorting interval. For data sets that are clearly not within the sorting interval, they can be deleted. Then, the data in the remaining data sets are merged, and then the target data is obtained and output from them according to the sorting interval of the target data. Since redundant data sets are eliminated based on the sorting interval of the target data, the data volume of each data set, the maximum and / or minimum values ​​of the data in each data set, and other information before merging the data in the multiple data sets, the data processing volume is effectively reduced during the merging of the remaining data sets, thereby facilitating the improvement of the query efficiency of the target data and solving the problem of slow data query speed for a specific sorting interval in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 A flowchart of a data query processing method provided by an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of a data query processing device provided by an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of a process of classifying and deleting a data set according to a first threshold or a second threshold in a data query processing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification, claims and drawings of the present invention are used to distinguish different objects rather than to limit a specific order.

[0031] Figure 1 The data query processing method according to an embodiment of the present invention is used to obtain data (target data) sorted in order of size and located in a specific sorting interval from the data to be queried. Figure 1 As shown, the method includes the following steps:

[0032] Step S102, determining multiple data sets based on the data to be queried, where the data sets include multiple data, and the multiple data in the data sets are arranged in order of size;

[0033] Step S104, determining the amount of data in the data set and the maximum value and / or minimum value of the data in the data set;

[0034] Step S106, deleting at least one data set from the multiple data sets based on the sorting interval of the target data, the amount of data in each data set, and the maximum value and / or minimum value of the data in each data set;

[0035] Step S108, merging the data in the remaining multiple data sets to obtain a sorted file, the sorted file including multiple data arranged in order of size;

[0036] Step S110 , according to the sorting interval of the target data, obtain the target data from the sorting file and output it.

[0037] The data query processing method using the above scheme includes: determining multiple data sets based on the data to be queried, the data sets including multiple data, and the multiple data in the data sets are arranged in order of size; determining the amount of data in the data sets and the maximum and / or minimum value of the data in the data sets; deleting at least one data set from the multiple data sets based on the sorting interval of the target data, the amount of data in each data set, and the maximum and / or minimum value of the data in each data set; merging the data in the remaining multiple data sets to obtain a sorted file, the sorted file including multiple data arranged in order of size; and obtaining the target data from the sorted file based on the sorting interval of the target data and outputting it.

[0038] In a data query processing method using this scheme, during the process of querying target data, the query data is first processed to obtain multiple data sets. The multiple data in the data sets are arranged in order of size. Then, the data volume and the maximum and / or minimum values ​​of the data in the data sets are determined. Based on this information, at least one data set is deleted from a macro perspective based on the data volume of each data set, the maximum value of the data in each data set, and / or the minimum value of the data in each data set, combined with the sorting interval of the target data to be queried. Since the sorting interval of the target data to be queried is known, the focus is on sorting and outputting the data within this sorting interval. Data sets that are clearly not within the sorting interval can be deleted, thereby merging the data in the remaining data sets, and then obtaining and outputting the target data from them according to the sorting interval of the target data. Since redundant data sets are eliminated based on the sorting interval of the target data, the data volume of each data set, the maximum and / or minimum values ​​of the data in each data set, and other information before merging the data in the multiple data sets, the data processing volume is effectively reduced during the merging of the remaining data sets, thereby improving the query efficiency of the target data and solving the problem of slow data query speed for a specific sorting interval in the related art.

[0039] The data to be queried is the entire data set, and the target data is the data you want to retrieve from the data to be queried. For example, in a query scenario, you want to retrieve data from 999,900 to 1,000,000 rows sorted according to certain conditions from 10 million rows. The 10 million rows are the query data, the data sorted between 999,900 and 1,000,000 is the target data, and the range of 999,900 to 1,000,000 is the sorting interval of the target data.

[0040] In a preferred embodiment, deleting at least one data set from multiple data sets based on the sorting interval of the target data, the amount of data in each data set, and the maximum and / or minimum value of the data in each data set includes: for any first threshold, if the sum of the amount of data of the data sets belonging to the first type and the data sets belonging to the second type is less than the lower limit of the sorting interval of the target data, deleting the data set of the first type from the multiple data sets; and / or, for any second threshold, if the sum of the amount of data of the data sets belonging to the third type is greater than the upper limit of the sorting interval of the target data, deleting the data set of the fourth type from the multiple data sets; wherein the first type of data set is a data set whose maximum value is less than the first threshold, the second type of data set is a data set whose minimum value is less than the first threshold and the maximum value of the data is greater than the first threshold, the third type of data set is a data set whose maximum value is less than the second threshold, and the fourth type of data set is a data set whose minimum value is greater than the second threshold.

[0041] In this embodiment, for any first threshold, if the sum of the data volumes of the first and second type datasets is less than the lower limit of the sorting interval for the target data, the first type dataset is removed from the multiple datasets. A first type dataset is a dataset whose maximum value is less than the first threshold, and a second type dataset is a dataset whose minimum value is less than the first threshold and whose maximum value is greater than the first threshold. Because the sum of the data volumes of the first and second type datasets is less than the lower limit of the sorting interval for the target data, the data ranking of the first type dataset must fall below the lower limit of the sorting interval for the target data. In other words, the data in the first type dataset must not fall within the sorting interval. Therefore, after removing these first type datasets and merging the remaining datasets, the data processing load during the dataset merging process can be effectively reduced while ensuring the accuracy of the query results, thereby improving data query efficiency. Similarly, for any second threshold, if the sum of the data volumes of the third type datasets is greater than the upper limit of the sorting interval for the target data, the fourth type dataset is removed from the multiple datasets. A third type dataset is a dataset whose maximum value is less than the second threshold, and a fourth type dataset is a dataset whose minimum value is greater than the second threshold. Since the total amount of data in the third type of data set is greater than the upper limit of the sorting interval of the target data, that is, the data in the third type of data set will exceed the sorting interval of the target data after sorting, the fourth type of data set must be outside the sorting interval. At this time, excluding this part of the fourth type of data set can effectively reduce the data processing amount in the data set merging process while ensuring the accuracy of the query results, which is conducive to improving data query efficiency.

[0042] Based on the above analysis, for any first and second thresholds, as long as the above requirements are met, the corresponding first type or fourth type data set can be eliminated. Therefore, to minimize the amount of data processing, it is necessary to eliminate as much first type and fourth type data as possible. In this embodiment, the first threshold is the maximum threshold that meets the first objective, where the sum of the data volumes of the first type and second type data sets is less than the lower limit of the sorting interval for the target data; and / or the second threshold is the minimum threshold that meets the second objective, where the sum of the data volumes of the third type data sets is greater than the upper limit of the sorting interval for the target data.

[0043] That is, in this embodiment, the first threshold is selected as the maximum threshold to meet the first objective, which is that the sum of the data volumes of the first and second type datasets is less than the lower limit of the sorting interval for the target data. The second threshold is selected as the minimum threshold to meet the second objective, which is that the sum of the data volumes of the third type datasets is greater than the upper limit of the sorting interval for the target data. This ensures that as many first type datasets and / or fourth type datasets as possible are ultimately eliminated, thereby minimizing the amount of data processing required for subsequent dataset merging and improving data query efficiency.

[0044] In a preferred embodiment, based on the sorting interval of the target data, the amount of data in each data set, and the maximum and / or minimum value of the data in each data set, deleting at least one data set from multiple data sets includes: starting from the first initial value, searching for the first threshold value in a direction from small to large; and / or, starting from the second initial value, searching for the second threshold value in a direction from large to small. In actual implementation, the first initial value and the second initial value can be flexibly determined according to actual conditions, and they can be user-defined values ​​or default values. For example, in an optional embodiment, the first initial value is the smallest of the maximum values ​​of the data in multiple data sets, that is, each data set has a maximum value of the data, and the smallest value from these maximum values ​​is selected as the first initial value, and the second initial value is the largest of the maximum values ​​of the data in multiple data sets. This can ensure the integrity of the screening of the first threshold value and the second threshold value, and avoid missing possible first threshold values ​​and second threshold values.

[0045] In the case where a first type of data set is removed from multiple data sets, the sorting of the final merged data will also change. In this case, in order to accurately find the target data, when the first type of data set is removed from the multiple data sets, the target data is obtained from the sorting file and output according to the sorting interval of the target data, including: determining the sum p of the amount of data in the first type of data set; subtracting the sum p from the upper and lower limits of the sorting interval of the target data to obtain an updated sorting interval; and obtaining data corresponding to the updated sorting interval from the sorting file and outputting it. That is, in the process of obtaining the target data from the sorting file and outputting it according to the sorting interval of the target data, the sum p of the amount of data in the removed first type of data set is first determined, and then the sum p is subtracted from the upper and lower limits of the sorting interval of the target data to obtain an updated sorting interval, and the target data is searched from the sorting file according to the updated sorting interval.

[0046] In this embodiment, determining multiple data sets based on the data to be queried includes a data set determination step that is executed cyclically. The data set determination step includes: obtaining a preset number of data from the data to be queried; and sorting the preset number of data in order of size to obtain a data set. In this embodiment, in order to determine multiple data sets based on the data to be queried, the data set determination step is executed cyclically. Specifically, the data set determination step includes: obtaining a preset number of data from the data to be queried; and then sorting the data in order of size to obtain a data set. In this way, by executing the data set determination step multiple times, multiple data sets can be obtained, and the data in each data set is arranged in order of size.

[0047] In one specific embodiment, merging the data from the remaining multiple data sets to obtain a sorted file includes a recursive data merging step, which includes: obtaining the largest or smallest data from the multiple data sets and adding it to the sorted file in order of size; determining whether the amount of data in the sorted file reaches the upper limit of the sorting interval for the target data; and terminating the data merging operation on the multiple data sets if the amount of data in the sorted file reaches the upper limit of the sorting interval for the target data. In this embodiment, during the merging of the data sets remaining after removing some data sets, the largest or smallest data from the multiple data sets are sequentially obtained and added to the sorted file in order of size. In this process, for example, if the data in the sorted files are arranged in ascending order, then in each data set merging step, the smallest data from the multiple data sets is obtained and added to the sorted file, followed by the smallest data from the remaining data and added to the sorted file. This recursive execution results in a sorted file in which the data is arranged in ascending order. During the data merging process, a determination is made as to whether the amount of data in the sorted file reaches the upper limit of the sorting interval for the target data. If the upper limit is reached, the data merging operation is terminated. Since the data in the sorting file is arranged in order of size, if the amount of data in it reaches the upper limit of the sorting interval where the target data is located, it means that the data in the sorting file has already covered the target data. At this time, the useless sorting operation of the remaining data can be omitted, which is conducive to improving data query efficiency.

[0048] The data query processing method of the present invention is described below with reference to a specific embodiment. The query scenario in this embodiment is a case in a TopK query, which requires sorting the data to be queried according to specific conditions and obtaining the first m pieces of data starting from the nth piece. TopK query is a type of query that frequently appears in analytical database workloads. For example, a common scenario in database systems is as follows: a batch of records are filtered according to given conditions, and these records are sorted according to user-specified conditions and displayed in a paginated manner. In this query scenario, a TopK query method such as ORDER BY column LIMIT n,m can be used to implement it, where n is the offset, m is the limit, and K is offset+limit.

[0049] In the absence of an index, when K is small, TopK queries can be implemented using the classic TopK algorithm based on priority queuing. However, when K is large, the priority queue data of size K cannot be cached in memory. In this case, the query data must be sorted externally, and then the records in the corresponding sorted position are retrieved. This process involves sorting all the data, resulting in a large amount of data sorting computation, low query efficiency, and poor performance.

[0050] The data query processing method of an embodiment of the present invention is as follows: multiple data sets are determined based on the data to be queried (each data set is referred to herein as a sorted run or run), and then the maximum and minimum value information of each sorted run is collected. A Zone Map is then constructed based on the collected maximum and minimum value information, and the maximum and minimum values ​​of each data set are recorded. The sorted run is pruned (deleted) with the help of the Zone Map. The pruned data sets will no longer participate in the subsequent data merging process, thereby avoiding unnecessary data sorting operations. Among them, the Zone Map is a data structure in the database for storing regional information of a table.

[0051] like Figure 3 As shown, Figure 3 The arrows in represent the number axis. Figure 3 Each rectangle (run) in represents a data set. The corresponding positions of the left and right ends of the rectangle on the number axis represent the minimum value (min) and maximum value (max) of the data in the data set. Barrier represents the first threshold or the second threshold mentioned above.

[0052] Any barrier can divide all data sets into three categories:

[0053] Type A corresponds to the first type of data set as described above when Barrier is the first threshold, and corresponds to the third type of data set as described above when Barrier is the second threshold. This type of data set satisfies minvalue of sorted run < Barrier && max value of sorted run < Barrier, that is, the minimum value of the data in this type of data set is less than Barrier, and the maximum value of the data is also less than Barrier, such as Figure 3 Run1 and Run2 in

[0054] Type B corresponds to the second type of data set as described above. This type of data set satisfies min value of sortedrun < Barrier && max value of sorted run > Barrier, that is, the minimum value of the data in this type of data set is less than Barrier, but the maximum value of the data is greater than Barrier, such as Figure 3 Run3 in

[0055] Type C corresponds to the fourth type of data set as described above. This type of data set satisfies min value of sortedrun > Barrier && max value of sorted run > Barrier, that is, the minimum value of the data in this type of data set is greater than Barrier, and the maximum value of the data is also greater than Barrier, such as Figure 3 Run4 and Run5 in

[0056] For any Barrier, if the data volume in Type A and Type B < the offset in the TopK query (i.e., the lower limit of the sorting interval corresponding to the target data), then the data in Type A must be ranked from the 0th to the offsetth position. At this time, the data set in Type A can be excluded so that it does not participate in the subsequent data merging operation.

[0057] For any Barrier, if the data volume in Type A > the offset + limit in the TopK query (i.e., the upper limit of the sorting interval corresponding to the target data), then the data in Type C must be ranked after the offset + limit position. At this time, the data set in Type C can be excluded so that it does not participate in the subsequent data merging operation.

[0058] The specific process of an embodiment of the data query processing method of the present invention is as follows:

[0059] Read partial data from the data to be queried into memory in sequence, sort the data in memory to generate a sorted run (i.e., a data set), collect the min and max values of the sorted run, and persist the sorted run to disk.

[0060] After generating all sorted runs, construct a Zone Map based on the collected min and max information.

[0061] Based on the Zone Map, find a Barrier1 as large as possible (corresponding to the first threshold above) such that the data volume in type A and type B < the offset in the TopK query.

[0062] Based on the Zone map, find a Barrier2 as small as possible (corresponding to the second threshold above) such that the data volume in type A > the offset + limit in the TopK query.

[0063] Use Barrier1 and Barrier2 to eliminate relevant sorted runs, and at the same time count the total sum p of the data volume of type A that is eliminated.

[0064] Then update the offset to new offset = old offset - p.

[0065] Merge the remaining sorted runs to obtain a sorted file, and then obtain the records at the corresponding positions in the sorted file as the final result of the TopK query.

[0066] The data query processing method in the embodiment of the present invention eliminates sorted runs, thereby avoiding operations on non-result set data as much as possible and efficiently implementing the TopK query. This algorithm can significantly improve the efficiency of the TopK query in scenarios where the available execution memory is not sufficient to cache the first K records, and solves the technical problem of slow query speed when querying data in a specific sorting range in the related art.

[0067] Secondly, as Figure 2As shown, an embodiment of the present invention further provides a data query processing device, including: a first determination unit, used to determine multiple data sets based on the data to be queried, the data sets including multiple data, and the data in the data sets are arranged in order of size; a second determination unit, used to determine the amount of data in the data sets and the maximum and / or minimum values ​​of the data in the data sets; a deletion unit, used to delete at least one data set from the multiple data sets based on the sorting interval of the target data, the amount of data in each data set, and the maximum and / or minimum values ​​of the data in each data set; a merging unit, used to merge the data in the remaining multiple data sets to obtain a sorted file, the sorted file including multiple data arranged in order of size; an acquisition unit, used to acquire and output the target data from the sorted file based on the sorting interval of the target data. In a data query processing device using this setting, during the process of querying target data, the first determination unit will first process the query data to obtain multiple data sets, and the multiple data in each data set are arranged in order of size. Then, the second determination unit will determine the amount of data in each data set and the maximum and / or minimum values ​​of the data. Based on this information, the deletion unit will delete at least one data set from a macro perspective based on the amount of data in each data set, the maximum value of the data in each data set, the minimum value of the data in each data set, and other information, combined with the sorting interval of the target data to be queried. Since the sorting interval of the target data to be queried is known, the focus is on sorting and outputting the data within this sorting interval. For data sets that are clearly not within the sorting interval, they can be deleted, so that the data in the remaining data sets are merged by the merging unit, and then the acquisition unit acquires and outputs the target data from them according to the sorting interval of the target data. Because before merging the data in multiple data sets, redundant data sets are eliminated based on the sorting interval of the target data, the data volume of each data set, the maximum value of the data in each data set, the minimum value of the data in each data set and other information, the data processing volume is effectively reduced in the process of merging the remaining data sets, which is conducive to improving the query efficiency of the target data and solving the problem of slow query speed for data in a specific sorting interval in related technologies.

[0068] In this embodiment, the deletion unit includes: a first deletion module, configured to delete a data set of the first type from the multiple data sets if, for any first threshold value, the sum of the data amounts of the data sets belonging to the first type and the data sets belonging to the second type is less than the lower limit of the sorting interval of the target data; and / or, a second deletion module, configured to delete a data set of the fourth type from the multiple data sets if, for any second threshold value, the sum of the data amounts of the data sets belonging to the third type is greater than the upper limit of the sorting interval of the target data; wherein the data set of the first type is a data set whose maximum value is less than the first threshold value, the data set of the second type is a data set whose minimum value is less than the first threshold value and whose maximum value is greater than the first threshold value, the data set of the third type is a data set whose maximum value is less than the second threshold value, and the data set of the fourth type is a data set whose minimum value is greater than the second threshold value;

[0069] The first threshold is a maximum threshold that satisfies a first objective, where the sum of the data volumes of the datasets belonging to the first type and the datasets belonging to the second type is less than a lower limit of the sorting interval for the target data; and / or the second threshold is a minimum threshold that satisfies a second objective, where the sum of the data volumes of the datasets belonging to the third type is greater than an upper limit of the sorting interval for the target data;

[0070] The deletion unit includes: a search module for searching for a first threshold value in a direction from small to large starting from a first initial value; and / or searching for a second threshold value in a direction from large to small starting from a second initial value;

[0071] The acquisition unit includes: a determination module for determining a total amount p of data in the first type of data set when a first type of data set is deleted from the plurality of data sets; a calculation module for subtracting the total amount p from both the upper and lower limits of the sorting interval of the target data to obtain an updated sorting interval; and an acquisition module for acquiring data corresponding to the updated sorting interval from the sorting file and outputting the data;

[0072] The first determining unit is used to cyclically execute the data set determining step, the data set determining step comprising: obtaining a preset number of data from the data to be queried; sorting the preset number of data in order of size to obtain a data set;

[0073] The merging unit is used to cyclically execute the data merging steps, which include: obtaining the largest or smallest data from multiple data sets and adding it to the sorting file in order of university; determining whether the amount of data in the sorting file reaches the upper limit of the sorting interval of the target data; and ending the data merging operation on the multiple data sets when the amount of data in the sorting file reaches the upper limit of the sorting interval of the target data.

[0074] In addition, an embodiment of the present invention further provides a non-volatile storage medium, wherein the non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the above-mentioned data query processing method.

[0075] Again, an embodiment of the present invention further provides a processor, wherein the processor is used to run a program, wherein the above-mentioned data query processing method is executed when the program is running.

[0076] Finally, an embodiment of the present invention further provides a data query processing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned data query processing method when executing the computer program.

[0077] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0078] The serial numbers of the embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. Moreover, the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system, such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be performed in a different order than shown here.

[0079] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0080] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0081] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0082] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0083] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0084] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A data query processing method, comprising: Determine multiple data sets based on the data to be queried, wherein the data sets include multiple data, and the multiple data in the data sets are arranged in order of size; determining the amount of data in the data set and the maximum and / or minimum value of the data in the data set; Deleting at least one data set from the plurality of data sets according to a sorting interval of the target data, an amount of data in each of the data sets, and a maximum value and / or a minimum value of data in each of the data sets; Merging the data in the remaining plurality of data sets to obtain a sorted file, wherein the sorted file includes a plurality of data arranged in order of size; According to the sorting interval of the target data, the target data is obtained from the sorting file and outputted; The deleting of at least one data set from the plurality of data sets according to the sorting interval of the target data, the amount of data in each of the data sets, and the maximum value and / or minimum value of the data in each of the data sets includes: For any first threshold, if the sum of the data amounts of the data sets belonging to the first type and the data sets belonging to the second type is less than the lower limit of the sorting interval of the target data, then deleting the data sets of the first type from the plurality of data sets; and / or, For any second threshold, if the total amount of data of the data sets belonging to the third type is greater than the upper limit of the sorting interval of the target data, deleting the data set of the fourth type from the multiple data sets; Among them, the first type of data set is the data set whose maximum value is less than the first threshold, the second type of data set is the data set whose minimum value is less than the first threshold and the maximum value is greater than the first threshold, the third type of data set is the data set whose maximum value is less than the second threshold, and the fourth type of data set is the data set whose minimum value is greater than the second threshold.

2. The data query processing method according to claim 1, wherein: The first threshold is a maximum threshold that satisfies a first goal, wherein the first goal is that the sum of the data volumes of the data sets belonging to the first type and the data sets belonging to the second type is less than the lower limit of the sorting interval of the target data; and / or The second threshold is a minimum threshold that satisfies a second goal, and the second goal is that the total amount of data of the third type of data sets is greater than the upper limit of the sorting interval of the target data.

3. The data query processing method according to claim 2, wherein: Deleting at least one data set from the plurality of data sets according to the sorting interval of the target data, the amount of data in each of the data sets, and the maximum value and / or minimum value of the data in each of the data sets includes: Starting from the first initial value, the first threshold is searched in a direction from small to large; and / or starting from the second initial value, the second threshold is searched in a direction from large to small.

4. The data query processing method according to claim 1, wherein: In the case where the first type of data set is deleted from the plurality of data sets, obtaining the target data from the sorted file and outputting the target data according to the sorting interval of the target data includes: Determine the total amount p of data in the first type of data set: The sum p is subtracted from both the upper and lower limits of the sorting interval of the target data to obtain an updated sorting interval; and data corresponding to the updated sorting interval is obtained from the sorting file and output.

5. The data query processing method according to claim 1, wherein: Determining multiple data sets according to the data to be queried includes a data set determination step that is executed cyclically, and the data set determination step includes: Acquire a preset amount of data from the data to be queried; The preset amount of data is sorted in order of size to obtain the data set.

6. The data query processing method according to claim 1, wherein: Merging the data in the remaining plurality of data sets to obtain a sorted file includes a data merging step that is performed cyclically, and the data merging step includes: Obtaining the largest or smallest data from the plurality of data sets and adding the data to the sorting file in order of size; Determine whether the amount of data in the sorted file reaches the upper limit of the sorting interval of the target data; When the amount of data in the sorted file reaches the upper limit of the sorting interval of the target data, the data merging operation on the plurality of data sets is terminated.

7. A data query processing device comprising: A first determining unit is configured to determine a plurality of data sets according to the data to be queried, wherein the data sets include a plurality of data, and the plurality of data in the data sets are arranged in order of size; a second determining unit, configured to determine the amount of data in the data set and the maximum value and / or minimum value of the data in the data set; a deleting unit, configured to delete at least one data set from the plurality of data sets based on a sorting interval of target data, an amount of data in each of the data sets, and a maximum value and / or a minimum value of data in each of the data sets; a merging unit, configured to merge the data in the remaining plurality of data sets to obtain a sorted file, wherein the sorted file includes a plurality of data arranged in order of size; an acquiring unit, configured to acquire the target data from the sorting file according to the sorting interval of the target data and output the target data; The deletion unit includes: a first deletion module, which is used to delete the first type of data set from the multiple data sets if the total amount of data of the first type of data set and the second type of data set is less than the lower limit of the sorting interval of the target data for any first threshold; and / or, a second deletion module, which is used to delete the fourth type of data set from the multiple data sets if the total amount of data of the third type of data set is greater than the upper limit of the sorting interval of the target data for any second threshold; wherein, the first type of data set is the data set whose maximum value is less than the first threshold, the second type of data set is the data set whose minimum value is less than the first threshold and the maximum value is greater than the first threshold, the third type of data set is the data set whose maximum value is less than the second threshold, and the fourth type of data set is the data set whose minimum value is greater than the second threshold.

8. The data query processing device according to claim 7, wherein: The first threshold is a maximum threshold that satisfies a first objective, where the sum of the data volumes of the datasets belonging to the first type and the datasets belonging to the second type is less than a lower limit of the sorting interval of the target data; and / or the second threshold is a minimum threshold that satisfies a second objective, where the sum of the data volumes of the datasets belonging to the third type is greater than an upper limit of the sorting interval of the target data; The deletion unit includes: a search module for searching for the first threshold value in a direction from small to large starting from the first initial value; and / or searching for the second threshold value in a direction from large to small starting from the second initial value; The acquisition unit includes: a determination module for determining a total amount p of data in the first type of data sets when the first type of data sets are deleted from the plurality of data sets; a calculation module for subtracting the total amount p from both the upper and lower limits of the sorting interval of the target data to obtain an updated sorting interval; and an acquisition module for acquiring and outputting data corresponding to the updated sorting interval from the sorting file; The first determining unit is used to cyclically execute the data set determining step, wherein the data set determining step includes: obtaining a preset number of data from the data to be queried; sorting the preset number of data in order of size to obtain the data set; The merging unit is used to cyclically execute the data merging steps, which include: obtaining the largest or smallest data from the multiple data sets and adding it to the sorting file in order of size; determining whether the amount of data in the sorting file reaches the upper limit of the sorting interval of the target data; and ending the data merging operation on the multiple data sets when the amount of data in the sorting file reaches the upper limit of the sorting interval of the target data.

9. A non-volatile storage medium, wherein: The non-volatile storage medium includes a stored program, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the data query processing method according to any one of claims 1 to 6.

10. A processor, wherein: The processor is used to run a program, wherein the program executes the data query processing method according to any one of claims 1 to 6 when running.

11. A data query processing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data query processing method according to any one of claims 1 to 6 is implemented.

Images