TopK query method, system and electronic device for massive data

Abstract

The present application provides a TopK query method, system, and electronic device for massive data. The method comprises: receiving an input data sequence; performing one or more descent operations on the input data sequence, wherein each descent operation includes performing maximum downsampling on a pending data sequence to obtain a local data sequence, the pending data sequence for the first descent operation being the input data sequence, and the pending data sequence for each descent operation other than the first being the local data sequence obtained by the previous descent operation; calculating, based on the local data sequence obtained by each descent operation, the TopK data in the local data sequence obtained by the first descent operation; and calculating and outputting, based on the TopK data in the local data sequence obtained by the first descent operation, the TopK data in the input data sequence. The method reduces query computation and improves query efficiency while ensuring result accuracy, and is suitable for performing TopK queries on massive data.

Description

Technical Field

[0001] The present application relates to the field of data analysis technology, and in particular to a TopK query method, system, and electronic device for massive data. Background Art

[0002] TopK queries, as a fundamental database operation, have been widely used in various scenarios, such as search engines, business analytics, and intelligent computing (such as face comparison). Traditional methods for executing TopK queries in databases typically include receiving a user query, determining a base dataset in the database, numerically sorting the data in the base dataset to produce a data sequence, and selecting the top K data in the sequence and returning them to the user. Parallel sorting algorithms are often used to improve data sorting efficiency. However, commonly used parallel sorting algorithms are nonlinear (such as bitonic sort, where the number of data exchanges is proportional to the number of data to be sorted), resulting in excessive computational overhead. Alternatively, linear algorithms such as counting sort require multiple random accesses to all data to be sorted and frequent CSUM operations (CSUM is a cumulative function used to calculate continuous cumulative values), resulting in memory bottlenecks when performing data sorting on parallel systems.

[0003] To improve the efficiency of TopK queries, researchers have proposed a method that groups the underlying dataset, performs a TopK query on each group, and finally selects the TopK data from multiple groups of TopK data. This method groups the data before sorting and extracting the TopK data. Although different groups can be processed in parallel, thus improving computational efficiency, it still suffers from the problem of high computational complexity.

[0004] Currently, with the widespread application of technologies such as distributed computing, cloud computing, the Internet of Things, and mobile Internet, the ability to process and analyze massive data has also been greatly improved. However, TopK queries on massive data still face challenges. Existing TopK query methods are not suitable for massive data due to their large amount of computation and low efficiency. How to quickly and accurately find TopK data in massive data is an urgent problem to be solved. Summary of the Invention

[0005] To overcome the problems existing in the above-mentioned prior art, one aspect of the present application provides a TopK query method for massive data, comprising: receiving an input data sequence; performing one or more descent operations on the input data sequence, wherein each descent operation includes performing maximum downsampling on the data sequence to be processed to obtain a local data sequence, the data sequence to be processed for the first descent operation is the input data sequence, and the data sequence to be processed for each descent operation except the first is the local data sequence obtained by the previous descent operation; based on the local data sequence obtained by each descent operation, calculating the TopK data in the local data sequence obtained by the first descent operation; and based on the TopK data in the local data sequence obtained by the first descent operation, calculating and outputting the TopK data in the input data sequence. Starting from the last descent operation, the TopK data in the local data sequence obtained by each descent operation is calculated, wherein the TopK data in the local data sequence obtained by the descent operation is used to calculate the TopK data in the local data sequence obtained by the previous descent operation, until the TopK data in the local data sequence obtained by the first descent operation is obtained.

[0006] In the above method, each descent operation includes performing a first maximum value downsampling on the data sequence to be processed to obtain a first local data sequence, and performing a second maximum value downsampling on the first local data sequence to obtain a second local data sequence. The data sequence to be processed for the first descent operation is the input data sequence, and the data sequence to be processed for each descent operation except the first is the second local data sequence obtained by the previous descent operation. Wherein, calculating the TopK data in the local data sequence obtained by the first descent operation based on the local data sequence obtained by each descent operation includes: starting from the last descent operation, calculating the TopK data in the second local data sequence obtained by each descent operation, and calculating the TopK data in the first local data sequence obtained by the descent operation based on the TopK data in the second local data sequence obtained by the descent operation, wherein the TopK data in the first local data sequence obtained by the descent operation is used to calculate the TopK data in the second local data sequence obtained by the previous descent operation, until the TopK data in the first local data sequence obtained by the first descent operation is obtained. Specifically, calculating the TopK data in the local data sequence obtained by the first descent operation based on the local data sequence obtained by each descent operation may include:

[0007] 1) Calculate the TopK data in the second local data sequence obtained by the last descent operation, find K data subsets in the first local data sequence obtained by the last descent operation that contain the TopK data, and calculate the TopK data in all the data in the K data subsets as the TopK data in the first local data sequence obtained by the last descent operation;

[0008] 2) Determine whether there is a previous descent operation. If so, use the previous descent operation as the current descent operation and execute step 3). If not, use the most recently calculated TopK data as the TopK data in the local data sequence obtained by the first descent operation and skip steps 3)-4).

[0009] 3) Find K data subsets in the second local data sequence obtained by the current descent operation that respectively contain the most recently calculated TopK data, and calculate the TopK data among all the data in the K data subsets as the TopK data in the second local data sequence obtained by the current descent operation;

[0010] 4) Find K data subsets in the first local data sequence obtained by the current descent operation that respectively contain the TopK data in the second local data sequence obtained by the current descent operation, calculate the TopK data among all the data in the K data subsets as the TopK data in the first local data sequence obtained by the current descent operation, and return to step 2).

[0011] In the above method, the number of data in the data subset searched for in the first partial data sequence is associated with the second maximum downsampling, and the number of data in the data subset searched for in the second partial data sequence is associated with the first maximum downsampling. The first maximum downsampling includes calculating the maximum data among every n data in the data sequence to be processed, and the second maximum downsampling includes calculating the maximum data among every m data in the first partial data sequence.

[0012] In the above method, in each data drop operation, a first maximum value downsampling is performed on the processed data sequence to obtain a first local data sequence, and a second maximum value downsampling is performed on the first local data sequence to obtain a second local data sequence, which is executed in a single thread warp. After each drop operation, the number of data obtained by performing the drop operation on the local data sequence obtained by the drop operation is estimated. If the number of data is greater than K, the drop operation is continued on the local data sequence obtained by the drop operation. If the number of data is less than or equal to K, the drop operation is stopped on the local data sequence obtained by the drop operation.

[0013] In the above method, calculating and outputting the TopK data in the input data sequence based on the TopK data in the local data sequence obtained by the first descent operation includes: searching for K data subsets in the input data sequence that respectively contain the TopK data in the local data sequence obtained by the first descent operation, calculating the TopK data in all data of the K data subsets as the TopK data in the input data sequence, and outputting the TopK data in the input data sequence.

[0014] One aspect of the present application further provides a TopK query system for massive data, the system comprising:

[0015] An input device for receiving an input data sequence;

[0016] a data downscaling device for performing one or more downscaling operations on an input data sequence; wherein each downscaling operation includes performing maximum downsampling on a data sequence to be processed to obtain a local data sequence, the data sequence to be processed for the first downscaling operation being the input data sequence, and the data sequence to be processed for each downscaling operation except the first one being the local data sequence obtained by the previous downscaling operation;

[0017] A small amount of TopK means is used to calculate the TopK data in the local data sequence obtained by the first descent operation based on the local data sequence obtained by each descent operation, and to calculate the TopK data in the input data sequence based on the TopK data in the local data sequence obtained by the first descent operation; and

[0018] The output device is used to output the TopK data in the input data sequence.

[0019] One aspect of the present application also provides an electronic device, including: a processor; and a memory, on which a computer program is stored and coupled to the processor, when the computer program is executed by the processor, the electronic device implements the above-mentioned TopK query method for massive data.

[0020] The TopK query method and system for massive data provided by this application not only ensures the accuracy of the obtained TopK data, but also reduces the amount of computation required during the query process and improves query efficiency. The time complexity of obtaining TopK data from N input data is close to O(N).

[0021] It should be understood that the foregoing general description and the following detailed description are merely illustrative and explanatory and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application, in which:

[0023] Figure 1 The flowchart of the TopK query method for massive data according to one embodiment of the present application is schematically shown;

[0024] Figure 2 Schematically shows the Figure 1The method shown is a flowchart of an example method for performing a Top2 query on an input data sequence including 64 data;

[0025] Figure 3 A flowchart of a TopK query method for massive data including multiple descending operations and a small number of TopK operations is schematically shown;

[0026] Figure 4 A block diagram of a TopK query system for massive data according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0027] To make the purpose, technical solutions and advantages of this application more clearly understood, the following further describes this application in detail through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0028] Existing sorting algorithms usually perform global sorting at the cost of a large number of random accesses or an even larger number of continuous accesses. However, extracting the TopK from data does not require global sorting. In view of this, this application uses the idea of ​​elimination to quickly narrow the scope of query results, thereby reducing query computation and improving query efficiency.

[0029] One aspect of the present application provides a TopK query method for massive data, which is used to quickly and accurately query TopK data from massive data.

[0030] Figure 1 The flowchart of the TopK query method for massive data according to one embodiment of the present application is schematically shown. The method is suitable for execution under the GPU (Graphics Processing Unit) architecture. In the GPU architecture, multiple (for example, 32) threads can form a warp. The threads in a warp can execute simultaneously and share data. The threads in the same warp can access each other's private data. Figure 1 The steps shown, combined with Figure 2 Describe the specific process of the TopK query method for massive data.

[0031] Step S11: Receive an input data sequence as a data sequence to be processed.

[0032] The input data sequence can be a data sequence retrieved by performing a keyword search in a database. Figure 2 A flowchart schematically illustrates an example method for performing a Top2 query on an input data sequence comprising 64 data. Figure 2As shown, the input data sequence is represented as {A1, A2, A3, A4, B1, B2, B3, B4, …, P1, P2, P3, P4}, with a total of N=64 data. The positions of these 64 data in the input data sequence are recorded as [0, 1, …, 63], and each data can be floating-point data.

[0033] Step S12: Perform a data descent operation (hereinafter referred to as the descent operation) on the data sequence to be processed to obtain a local data sequence. Specifically, step S12 includes the following sub-steps:

[0034] S121. Perform first maximum downsampling on the data sequence to be processed to obtain a first local maximum data sequence (hereinafter referred to as the first local data sequence). Performing the first maximum downsampling on the data sequence to be processed includes: calculating the local maximum data for every n (n greater than 1) data points in the data sequence to be processed, i.e., calculating the local maximum data for data points 0 to n-1, for data points n to 2n-1, and so on; and adding all calculated local maximum data points to the first local data sequence, with the order of these local maximum data points in the first local data sequence being consistent with their order in the data sequence to be processed. In a GPU architecture, a thread can be allocated for every n data points in the data sequence to perform operations such as loading the n data points and calculating the local maximum data points for the n data points. Furthermore, multiple threads can be allocated to perform the loading of the n data points and the calculation of the local maximum data points for the n data points in parallel, thereby improving processing efficiency.

[0035] S122. Perform a second maximum downsampling on the first local data sequence to obtain a second local maximum data sequence (hereinafter referred to as the second local data sequence). Performing the second maximum downsampling on the first local data sequence includes: calculating the local maximum data for every m (m greater than 1) data items in the first local data sequence, i.e., calculating the local maximum data for data items 0 to m-1, and for data items m to 2m-1, and so on; adding all calculated local maximum data items to the second local data sequence, with the order of these local maximum data items in the second local data sequence being consistent with their order in the first local data sequence and consistent with their order in the data sequence to be processed. As described above, in a GPU architecture, multiple threads can be allocated to load n data items and calculate the local maximum data items in the n data items in parallel. More specifically, multiple threads within a wrap can be assigned to execute the loading of n data and the calculation of local maximum data in parallel to obtain a first local data sequence, and the first local data sequence can be shared within the wrap using the shuffle instruction, so that the local maximum data of every m data in the first local data sequence can be efficiently calculated to obtain a second local data sequence.

[0036] S123. Record the first local data sequence obtained in sub-step S121 and the second local data sequence obtained in sub-step S122. As mentioned above, the first local data sequence is obtained from the local maximum data of every n data in the data sequence to be processed, so the first local data sequence includes data, where N i Indicates the number of data in the data sequence to be processed. The positions of the data in the first local data sequence can be recorded as [0, 1, ..., ]; The second local data sequence is obtained from the local maximum data of each m data in the first local data sequence, so the second local data sequence includes data, this The positions of the data in the second local data sequence can be recorded as [0, 1, ..., It should be noted that here N i Represents the number of data in the data sequence to be processed, where i indicates that the i-th descent operation is currently being performed (1≤i≤I, the I-th descent operation indicates the last descent operation), and the first descent operation is performed on the input data sequence (that is, the data sequence to be processed for the first descent operation is the input data sequence), so N1=N.

[0037] Figure 2FIG shows a first local data sequence and a second local data sequence obtained by performing a drop operation on an input data sequence including 64 data when n=4 and m=4. Figure 2 As shown, the first maximum value downsampling is performed on the input data sequence {A1, A2, A3, A4, B1, B2, B3, B4, ..., P1, P2, P3, P4} to obtain The first local data sequence of data is {A, B, C, D, ..., M, N, O, P}, where A represents the local maximum data among the four data A1, A2, A3, and A4, and B represents the local maximum data among the four data B1, B2, B3, and B4. Similarly, the positions of the 16 data in the first local data sequence can be recorded as [0, 1, ..., 15]. Further, the second maximum value downsampling is performed on the first local data sequence to obtain the data including The second local data sequence of the data is {C, F, K, M}, where C represents the local maximum data among the four data of A, B, C, and D. Similarly, the positions of the four data in the second local data sequence can be recorded as [0, 1, ..., 3]. It should be understood that although Figure 2 The exemplary method is described taking n=m=4 as an example, but in other embodiments, n and m may not be equal and may be other integers greater than 1.

[0038] Step S13: Determine whether the iteration end condition is met. If so, execute step S14. If not, use the most recently obtained second partial data sequence as the data sequence to be processed and return to execute step S12.

[0039] Specifically, determining whether the iteration termination condition has been met includes: estimating, based on the most recently obtained second partial data sequence, the number of data in the second partial data sequence that would be obtained by performing a descending operation on the most recently obtained second partial data sequence; if the estimated number of data is greater than K, determining that the iteration termination condition has not been met; and if the estimated number of data is less than or equal to K, determining that the iteration termination condition has been met. By setting the iteration termination condition so that the number of data in the second partial data sequence obtained by the last descending operation is just greater than K (i.e., if a further descending operation is performed on the second partial data sequence obtained by the last descending operation, the number of data in the second partial data sequence obtained will be less than or equal to K), the computational complexity of subsequent steps can be reduced while increasing efficiency.

[0040] See also Figure 2 After performing a drop operation on the input data sequence as the data sequence to be processed, a second local data sequence including 4 data is obtained, namely {C, F, K, M}. Performing a drop operation on these 4 data will be expected to obtain The second local data sequence of data (N2 represents the number of data in the data sequence to be processed for the second descent operation, and the data sequence to be processed for the second descent operation is the second local data sequence obtained by the first descent operation, so N2 = 4), since 1 ≤ K (in Figure 2 In the example, K=2), it is determined that the iteration end condition is reached, and further descent operations on {C, F, K, M} are stopped, and step S14 is executed next.

[0041] From steps S12-S13, it can be seen that in a reduction operation, the size of the data sequence to be processed is first reduced to (e.g. 1 / 4), and further reduced to (For example, 1 / 16), the non-TopK data is eliminated, thereby narrowing the query scope, reducing the amount of calculation during the query process, and improving query efficiency.

[0042] Step S14: Based on the local data sequence obtained by each descent operation, calculate the TopK data in the local data sequence obtained by the first descent operation.

[0043] Specifically, starting from the last descent operation (i.e., the first descent operation), the TopK data in the second local data sequence obtained in each descent operation are calculated. The TopK data in the first local data sequence obtained in that descent operation are then used to calculate the TopK data in the first local data sequence obtained in that descent operation. The TopK data in the first local data sequence obtained in that descent operation are used to calculate the TopK data in the second local data sequence obtained in the previous descent operation (if any). This process is repeated until the TopK data in the first local data sequence obtained in the first descent operation are obtained, which are used as the TopK data in the local data sequence obtained in the first descent operation. Step S14 is described below through sub-steps S141-S148:

[0044] S141. Calculate the TopK data in the second local data sequence obtained by the last descent operation (i.e., the Ith descent operation), and obtain the K positions of the TopK data in the second local data sequence obtained by the last descent operation in the second local data sequence obtained by the last descent operation. Since the number of data in the second local data sequence obtained by the last descent operation is just greater than K, calculating the TopK data therein is a small TopK operation, and its computational complexity can be ignored.

[0045] S142. The K positions of the TopK data in the second local data sequence obtained by the last descent operation in the second local data sequence obtained by the last descent operation. , find the K data subsets in the first local data sequence obtained by the last descent operation that respectively include the TopK data (i.e., the TopK data in the second local data sequence obtained by the last descent operation). The K starting positions of the K data subsets in the first local data sequence obtained by the last descent operation are respectively ,in And 1≤k≤K; the K end positions of the K data subsets in the first local data sequence obtained by the last descent operation are respectively When searching for K data subsets that include the TopK data, for the kth data in the TopK data (1≤k≤K), you can use A position in the first local data sequence obtained by the last descending operation is obtained, and m data are selected starting from the position as a data subset including the k-th data in the first local data sequence obtained by the last descending operation.

[0046] S143. Calculate the top K data of all the data in the K data subsets found (a total of m*K data), and use them as the top K data in the first local data sequence obtained by the last descent operation. Obtain the K positions of the top K data in the first local data sequence obtained by the last descent operation in the first local data sequence obtained by the last descent operation. Calculating the TopK data among m*K data is a small TopK operation, and its computational complexity can be ignored.

[0047] S144. Determine whether there is a previous descent operation. If so, use the previous descent operation as the current descent operation and execute step S145. If not, use the most recently calculated TopK data as the TopK data in the local data sequence obtained by the first descent operation and skip the following steps S145-S148.

[0048] S145. The current descent operation is the i-th (1≤i<I) descent operation. The K positions of the most recently calculated TopK data in the first local data sequence obtained in the next descent operation are , find the K data subsets in the second local data sequence obtained by the current descent operation that respectively include the TopK data (i.e., the TopK data in the first local data sequence obtained by the next descent operation). The K starting positions of the K data subsets in the second local data sequence obtained by the current descent operation are respectively ,in And 1≤k≤K; the K end positions of the K data subsets in the second local data sequence obtained by the current descending operation are respectively When searching for K data subsets that include the TopK data, for the kth data in the TopK data (1≤k≤K), you can use A position in the second partial data sequence obtained by the current descending operation is obtained, and n data are selected starting from the position as a data subset including the k-th data in the second partial data sequence obtained by the current descending operation.

[0049] S146. Calculate the top K data in all the data in the K data subsets found (a total of n*K data), and use them as the top K data in the second local data sequence obtained by the current descent operation. Obtain the K positions of the top K data in the second local data sequence obtained by the current descent operation in the second local data sequence obtained by the current operation. . Among them, calculating the TopK data among n*K data is a small TopK operation, and its computational complexity can be ignored.

[0050] S147. The top K data in the second local data sequence obtained by the current descending operation are located at K positions in the second local data sequence obtained by the current operation. , find the K data subsets in the first local data sequence obtained by the current descent operation that respectively include the TopK data (i.e., the TopK data in the second local data sequence obtained by the current descent operation). The K starting positions of the K data subsets in the first local data sequence obtained by the current descent operation are respectively ,in And 1≤k≤K; the K end positions of the K data subsets in the first local data sequence obtained by the current descending operation are respectively When searching for K data subsets that include the TopK data, for the kth data in the TopK data (1≤k≤K), you can use A position in the first local data sequence obtained by the current descending operation is obtained, and m data are selected starting from the position as a data subset including the k-th data in the first local data sequence obtained by the current descending operation.

[0051] S148. Calculate the top K data of all the data in the K data subsets found (a total of m*K data), and use them as the top K data in the first local data sequence obtained by the current descent operation. Obtain the K positions of the top K data in the first local data sequence obtained by the current descent operation in the first local data sequence obtained by the current descent operation. , returning to step S144. Calculating the TopK data among the m*K data is a small TopK operation, and its computational complexity can be ignored.

[0052] As can be seen from the above sub-steps S141-S148, after performing one or more descent operations on the input data sequence, starting from the last descent operation, the TopK data in the second local data sequence and the TopK data in the first local data sequence obtained by each descent operation are calculated, and finally the TopK data in the first local data sequence obtained by the first descent operation are obtained (as the TopK data in the local data sequence obtained by the first descent operation). The flowchart of performing multiple descent operations and a small number of TopK operations can be found in Figure 3 , however in Figure 2 In the example method shown, only one descending operation is performed (ie, the last descending operation is the first descending operation). Figure 2 As shown, after the descent operation, based on the above sub-steps S141-S148, first calculate the Top2 data in the second local data sequence {C, F, K, M} obtained by the last descent operation, i.e., C and K; then find the two starting positions of C and K in the first local data sequence obtained by the last descent operation, and starting from these two positions, select 4 data in the first local data sequence obtained by the last descent operation, thereby obtaining two data subsets including C and K, i.e., {A, B, C, D} and {I, J, K, L}; then calculate the Top2 data among the 8 data in the two data subsets, i.e., B and C, as the Top2 data in the first local data sequence obtained by the last descent operation, and then select the Top2 data in the first local data sequence obtained by the last descent operation. Figure 2 The top 2 data in the local data sequence obtained by the first descent operation.

[0053] Step S15. Calculate the TopK data in the input data sequence based on the TopK data in the local data sequence obtained by the first descent operation. Specifically, step S15 includes the following sub-steps:

[0054] S151. The K positions of the TopK data in the local data sequence obtained by the first descent operation in the first local data sequence obtained by the first descent operation. , find the K data subsets in the input data sequence that include the TopK data. The starting positions of the K data subsets in the input data sequence are ,in And 1≤k≤K; the K end positions of the K data subsets in the input data sequence are When searching for K data subsets that include the TopK data, for the kth data in the TopK data (1≤k≤K), you can use A position in the input data sequence is obtained, and n data are selected starting from the position as a data subset in the input data sequence including the k-th data.

[0055] S152. Calculate the top K data from all the data in the obtained K data subsets and use them as the top K data in the input data sequence. Calculating the top K data from the n*K data is a small Top K operation, and its computational complexity can be ignored.

[0056] exist Figure 2 In the example method shown, the top two data in the local data sequence obtained by the first descent operation are B and C. Based on the positions of B and C in the first local data sequence obtained by the first descent operation, two data subsets containing B and C respectively in the input data sequence can be found, namely {B1, B2, B3, B4} and {C1, C2, C3, C4}. The top two data among all eight data in these two data subsets are calculated, namely C3 and B1. Then C3 and B1 are the top two data in the input data sequence.

[0057] Through steps S14-S15, the TopK data in the input data sequence can be accurately obtained by only one or more small TopK operations. Each small TopK operation only processes the TopK calculation from n*K to K or from m*K to K, and the amount of computation is very small and can be ignored.

[0058] Step S16: Output the TopK data in the input data sequence obtained in step S15.

[0059] According to the above embodiment, the input data sequence is only globally accessed once, and each drop operation reduces the data size to the size before the drop operation. After the descent operation is completed, a small number of TopK operations are performed forward, starting from the local data sequence obtained from the last descent operation, to finally obtain the TopK data in the input data sequence. This process ensures the accuracy of the query results while reducing the amount of query computation and improving query efficiency. When K is much smaller than the number of data points N in the input data sequence, the time complexity is linear and close to O(N), making it suitable for TopK queries on massive data.

[0060] In the above embodiment, each descent operation includes performing a first maximum downsampling on the processed data sequence to obtain a first local data sequence, and performing a second maximum downsampling on the first local data sequence to obtain a second local data sequence. In other embodiments, each descent operation may also include performing only one maximum downsampling on the processed data sequence, or may include performing three or more maximum downsampling on the processed data sequence. Where maximum downsampling is performed once on the processed data sequence in each descent operation, calculating the TopK data in the local data sequence obtained by the first descent operation includes: starting from the last descent operation, calculating the TopK data in the local data sequence obtained by each descent operation, wherein the TopK data in the local data sequence obtained by the previous descent operation is used to calculate the TopK data in the local data sequence obtained by the previous descent operation, and repeating this process until the TopK data in the local data sequence obtained by the first descent operation is obtained. After obtaining the TopK data in the local data sequence obtained by the first descent operation, searching for K data subsets in the input data sequence that each include the TopK data in the local data sequence obtained by the first descent operation, and calculating the TopK data in all data in the K data subsets as the TopK data in the input data sequence.

[0061] It should be noted that the exemplary methods described above are depicted as flow charts. Although the flow charts depict operations as being performed sequentially, it is understood that many of the operations can be performed in parallel, simultaneously, or synchronously, and that the order of the operations can be rearranged. The process can terminate upon completion of an operation and can include additional steps not included in the figures or embodiments.

[0062] The aforementioned TopK query method for massive data can be applied in a variety of scenarios, such as search engines, business analysis (such as information recommendation systems), and intelligent computing. For example, an information recommendation method based on the aforementioned TopK query method for massive data may include: collecting data from users; calculating the similarity between each piece of information in the information recommendation system and the user data; calculating, based on the aforementioned TopK query method for massive data, K pieces of information corresponding to the TopK similarities; and providing the obtained K pieces of information to the user.

[0063] Another aspect of the present application provides a TopK query system for massive data.

[0064] Figure 4 The block diagram of a TopK query system for massive data according to an embodiment of the present application is schematically shown. Figure 4 As shown in the figure, the TopK query system for massive data includes:

[0065] An input device for receiving an input data sequence as a data sequence to be processed;

[0066] a data downscaling device for performing one or more downscaling operations on an input data sequence; wherein each downscaling operation includes performing maximum downsampling on a data sequence to be processed to obtain a local data sequence, the data sequence to be processed for the first downscaling operation being the input data sequence, and the data sequence to be processed for each downscaling operation except the first one being the local data sequence obtained by the previous downscaling operation;

[0067] A control device, configured to determine whether an iteration end condition is met, and if so, to control the small amount TopK device to perform a small amount TopK operation; if not, to control the data descent device to continue performing the descent operation;

[0068] A small number of TopK devices are used to calculate the TopK data in the local data sequence obtained by the first descent operation based on the local data sequence obtained by each descent operation, and to calculate the TopK data in the input data sequence based on the TopK data in the local data sequence obtained by the first descent operation;

[0069] The output device is used to output the TopK data in the input data sequence.

[0070] Each device in the TopK query system for massive data corresponds to each step of the above-mentioned TopK query method for massive data. Therefore, for details not disclosed in the embodiment of the device of this application, please refer to the embodiment of the above-mentioned TopK query method for massive data.

[0071] It should be understood that although several modules or units of the system are described above, the division of modules and units is not limited to this. In fact, the features and functions of two or more modules or units described above can also be implemented in one module or unit. Conversely, the features and functions of one module or unit described above can also be further divided into multiple modules or units to implement.

[0072] Another aspect of the present application provides an electronic device that can be used to execute the aforementioned TopK query method for massive data.

[0073] Exemplarily, the electronic device may include a bus, and a processor, memory, input devices (such as a keyboard, mouse, sensor, etc.), output devices (such as a display, printer, speaker, etc.), communication interfaces (such as a parallel port, serial port, modem, network card, etc.) and other devices (such as detachable devices, drive devices, etc.) coupled to the bus.

[0074] The memory (such as ROM, PROM, EEPROM, RAM, SRAM, etc.) is used to store data and can store computer instructions or programs, including computer instructions or programs for implementing the TopK query method for massive data described above. The processor is used to execute a series of actions specified by the computer instructions or programs, such as executing the computer instructions or programs stored in the memory. When the processor executes the computer instructions or programs stored in the memory, the electronic device can implement the embodiment of the TopK query method for massive data described above, including Figure 1 Follow the steps shown in .

[0075] Another aspect of the present application provides a computer-readable medium, which includes but is not limited to: a floppy disk, a hard disk, a magnetic tape, other magnetic media, a CDROM, a CDRW, a DVD, other optical media, a punched card, other physical media, a ROM, a PROM, an EEPROM, a RAM, a SRAM, or other computer-readable media, and a transmission medium (such as a coaxial cable, an optical fiber cable, a carrier wave, etc.). The computer-readable medium may be included in the above-mentioned electronic device, or it may be an uninstalled separate medium. The computer-readable medium is used to carry computer instructions or programs, including computer instructions or programs for implementing the TopK query method for massive data described above. When the computer instructions or programs in the computer-readable medium are read and executed by a processor (for example, a processor in the above-mentioned electronic device), it can implement (for example, enable the above-mentioned electronic device to implement) the TopK query method embodiment for massive data described above, including Figure 1 Follow the steps shown in .

[0076] Although the present application has been described through preferred embodiments, it should be understood that the present application is not limited to the embodiments described above and shown in the accompanying drawings, and that those skilled in the art may make various changes and modifications without departing from the scope of the present application.

Claims

1. A TopK query method for massive data suitable for execution in a graphics processing unit (GPU) architecture. In this GPU architecture, each thread warp consists of multiple threads, and multiple threads in the same thread warp can execute simultaneously and access each other's private data. The method is characterized by: The method comprises: receiving an input data sequence; Perform one or more descent operations on the input data sequence; wherein each descent operation includes performing maximum downsampling on the data sequence to be processed to obtain a local data sequence, the data sequence to be processed for the first descent operation is the input data sequence, and the data sequence to be processed for each descent operation except the first is the local data sequence obtained by the previous descent operation; Based on the local data sequence obtained by each descent operation, calculate the TopK data in the local data sequence obtained by the first descent operation; and Calculate and output the TopK data in the input data sequence based on the TopK data in the local data sequence obtained by the first descent operation; The performing one or more descent operations on the input data sequence includes: allocating one thread for every n data in the data sequence to be processed, multiple threads in the same thread bundle performing the loading and descent operations on the n data in parallel, and in each descent operation, performing a first maximum value downsampling on the data sequence to be processed to obtain a first local data sequence and performing a second maximum value downsampling on the first local data sequence to obtain a second local data sequence in one thread bundle.

2. The method according to claim 1, characterized in that Based on the local data sequence obtained by each descent operation, the TopK data in the local data sequence obtained by the first descent operation are calculated, including: Starting from the last descent operation, the TopK data in the local data sequence obtained by each descent operation are calculated, where the TopK data in the local data sequence obtained by this descent operation are used to calculate the TopK data in the local data sequence obtained by the previous descent operation, until the TopK data in the local data sequence obtained by the first descent operation are obtained.

3. The method according to claim 1, characterized in that Each downsampling operation includes performing a first maximum value downsampling on the data sequence to be processed to obtain a first local data sequence, and performing a second maximum value downsampling on the first local data sequence to obtain a second local data sequence; The data sequence to be processed in the first descent operation is the input data sequence, and the data sequence to be processed in each descent operation except the first one is the second local data sequence obtained in the previous descent operation.

4. The method according to claim 3, characterized in that Based on the local data sequence obtained by each descent operation, the TopK data in the local data sequence obtained by the first descent operation are calculated, including: Starting from the last descent operation, the TopK data in the second local data sequence obtained by each descent operation are calculated, and the TopK data in the first local data sequence obtained by the descent operation are calculated based on the TopK data in the second local data sequence obtained by the descent operation, wherein the TopK data in the first local data sequence obtained by the descent operation is used to calculate the TopK data in the second local data sequence obtained by the previous descent operation, until the TopK data in the first local data sequence obtained by the first descent operation is obtained.

5. The method according to claim 4, characterized in that Based on the local data sequence obtained by each descent operation, the TopK data in the local data sequence obtained by the first descent operation are calculated, including: 1) Calculate the TopK data in the second local data sequence obtained by the last descent operation, find K data subsets in the first local data sequence obtained by the last descent operation that contain the TopK data, and calculate the TopK data in all the data in the K data subsets as the TopK data in the first local data sequence obtained by the last descent operation; 2) Determine whether there is a previous descent operation. If so, use the previous descent operation as the current descent operation and execute step 3). If not, use the most recently calculated TopK data as the TopK data in the local data sequence obtained by the first descent operation and skip steps 3)-4). 3) Find K data subsets in the second local data sequence obtained by the current descent operation that respectively contain the most recently calculated TopK data, and calculate the TopK data among all the data in the K data subsets as the TopK data in the second local data sequence obtained by the current descent operation; 4) Find K data subsets in the first local data sequence obtained by the current descent operation that respectively contain the TopK data in the second local data sequence obtained by the current descent operation, calculate the TopK data among all the data in the K data subsets as the TopK data in the first local data sequence obtained by the current descent operation, and return to step 2).

6. The method according to claim 5, characterized in that The number of data in the data subset found in the first partial data sequence is associated with the second maximum downsampling, and the number of data in the data subset found in the second partial data sequence is associated with the first maximum downsampling; Wherein, performing the first maximum value downsampling includes calculating the maximum data among every n data in the data sequence to be processed, and performing the second maximum value downsampling includes calculating the maximum data among every m data in the first local data sequence.

7. The method according to any one of claims 1 to 6, characterized in that Calculating and outputting the TopK data in the input data sequence according to the TopK data in the local data sequence obtained by the first descent operation includes: Search for K data subsets in the input data sequence that respectively contain the TopK data in the local data sequence obtained by the first descent operation, calculate the TopK data in all data of the K data subsets as the TopK data in the input data sequence, and output the TopK data in the input data sequence.

8. The method according to any one of claims 1 to 6, characterized in that After executing each descending operation, the number of data obtained by performing the descending operation on the local data sequence obtained by this descending operation is estimated. If the number of data is greater than K, the descending operation is continued to be performed on the local data sequence obtained by this descending operation. If the number of data is less than or equal to K, the descending operation is stopped on the local data sequence obtained by this descending operation.

9. A TopK query system for massive data suitable for execution on a graphics processing unit (GPU) architecture. In this GPU architecture, each thread warp consists of multiple threads, and multiple threads in the same thread warp can execute simultaneously and access each other's private data. The system is characterized by: The system comprises: An input device for receiving an input data sequence; A data drop device, configured to perform one or more drop operations on the input data sequence; wherein each drop operation includes performing maximum downsampling on the data sequence to be processed to obtain a local data sequence, the data sequence to be processed for the first drop operation is the input data sequence, and the data sequence to be processed for each drop operation except the first is the local data sequence obtained by the previous drop operation; wherein the performing one or more drop operations on the input data sequence includes: allocating one thread for every n data in the data sequence to be processed, multiple threads in the same thread bundle performing loading and drop operations on the n data in parallel, and in each drop operation, performing a first maximum downsampling on the data sequence to be processed to obtain a first local data sequence and performing a second maximum downsampling on the first local data sequence to obtain a second local data sequence in one thread bundle; A small amount of TopK means is used to calculate the TopK data in the local data sequence obtained by the first descent operation based on the local data sequence obtained by each descent operation, and calculate the TopK data in the input data sequence based on the TopK data in the local data sequence obtained by the first descent operation; and The output device is used to output the TopK data in the input data sequence.

10. An electronic device, characterized in that: include: processor; as well as A memory having a computer program stored thereon and coupled to the processor, which, when the computer program is executed by the processor, causes the electronic device to implement the method according to any one of claims 1 to 8.

Images