Aggregate query method, system, device and storage medium

Abstract

One or more embodiments of this specification provide an aggregate query method, system, device, and storage medium. The method includes: responding to a database aggregate query request, determining the data table to be queried, the grouping field, and the aggregation method; creating a grouping array; dividing the data table into at least two data sub-tables, and sequentially processing each data sub-table as follows: for multiple tuples in the data sub-table, performing the following operations in parallel: checking whether there is a target grouping field in the grouping array that is the same as the grouping field of the tuple; if so, recording the array index; after traversing all tuples in the data sub-table, for tuples to be processed that do not have an array index recorded, adding the grouping field of the tuple to be processed to the grouping array and recording the array index; performing aggregate calculations based on the array index, grouping array, and aggregation method recorded for each tuple to obtain an aggregate query result. This is conducive to improving the efficiency of aggregate queries.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of database query technology, and in particular, to an aggregate query method, system, device, and storage medium. Background Art

[0002] Database query functions include aggregation queries, including group by and distinct queries. These queries typically group fields that require aggregation analysis based on specified dimension fields in the database and then perform aggregation operations, such as sum, average, or count.

[0003] When implementing grouped aggregation queries, common database engines currently consider the possibility that different tuples belong to the same group. Therefore, they can only group the fields that require aggregation analysis sequentially and serially before performing aggregation operations, resulting in low aggregation query efficiency. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide an aggregate query method, system, device, and storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a method is proposed for determining a data table to be queried, grouping fields, and aggregation method in response to a database aggregation query request;

[0007] Creating a grouping array, wherein the grouping array is used to store different grouping fields in the data table;

[0008] The data table is divided into at least two data sub-tables, and each data sub-table is processed as follows:

[0009] For multiple tuples in the data table, the following operations are performed in parallel: checking whether there is a target grouping field in the grouping array that is the same as the grouping field of the tuple; if so, recording the array index of the target grouping field;

[0010] After traversing all tuples in the data sub-table, for tuples to be processed whose array index is not recorded, a grouping field of the tuple to be processed is added to the grouping array, and the array index of the grouping field of the tuple to be processed is recorded;

[0011] Aggregation calculation is performed according to the array index of each tuple record in the data sub-table, the grouping array and the aggregation method to obtain an aggregate query result.

[0012] Optionally, also include:

[0013] Creating a hash table; the hash table includes hash buckets, and different hash buckets are used to store array indexes pointing to different group fields in the group array;

[0014] The step of searching whether there is a target grouping field in the grouping array that is the same as the grouping field of the tuple includes:

[0015] Performing a hash operation on the grouping field in the tuple to obtain a hash value;

[0016] Determine, according to the hash value, a target hash bucket corresponding to the tuple in the hash table;

[0017] Determine whether the grouping field of the tuple is the same as the target grouping field indicated by the array index stored in the target hash bucket;

[0018] For a tuple to be processed whose array index is not recorded, the method further includes:

[0019] The array index of the grouping field of the tuple to be processed is stored in the target hash bucket corresponding to the tuple to be processed.

[0020] Optionally, the hash bucket in the hash table is used to store at least one array index using a linked list structure;

[0021] Determining whether the grouping field of the tuple is the same as the target grouping field indicated by the array index stored in the target hash bucket includes:

[0022] If the target hash bucket is not empty, read the first target group field from the group array according to the first array index stored in the linked list structure of the target hash bucket;

[0023] Compare the grouping field of the tuple with the target grouping field value read;

[0024] If the two are the same, record the array index of the target group field to be read;

[0025] If the two are different, read the next target grouping field value from the grouping array according to the next array index stored in the linked list structure of the target hash bucket, and repeat the step of comparing the grouping field of the tuple with the read target grouping field value.

[0026] Optionally, for a tuple to be processed whose array index is not recorded, a grouping field of the tuple to be processed is added to the grouping array, the array index of the grouping field of the tuple to be processed is stored in the target hash bucket corresponding to the tuple to be processed, and the array index of the grouping field of the tuple to be processed is recorded, including:

[0027] When there are multiple tuples to be processed whose array indexes are not recorded, the multiple tuples to be processed are processed in sequence as follows:

[0028] For each tuple to be processed, determine the target hash bucket corresponding to the tuple to be processed in the hash table;

[0029] If the target hash bucket is empty, or the grouping field of the tuple is different from the target grouping field indicated by the array index stored in the target hash bucket, the grouping field of the tuple to be processed is added to the grouping array, and the corresponding array index is stored in the target hash bucket corresponding to the tuple to be processed, and the array index is recorded;

[0030] If the grouping field of the tuple to be processed is the same as the target grouping field indicated by the array index stored in the target hash bucket, the array index of the target grouping field is recorded.

[0031] Optionally, the method further includes:

[0032] Creating an aggregation array, wherein the aggregation array is used to store the aggregation results of different grouping fields in the grouping array;

[0033] The performing aggregation calculation according to the array index of each tuple record in the data sub-table, the grouping array and the aggregation method includes:

[0034] For each tuple in the data sub-table, read the grouping field in the grouping array according to the array index, and determine whether there is an aggregation result of the grouping field in the aggregation array;

[0035] If it exists, update the aggregation result using the aggregation method and the tuple;

[0036] If it does not exist, the aggregation result corresponding to the grouping field of the tuple is calculated using the aggregation method and the tuple, and the aggregation result corresponding to the grouping field of the tuple is saved in the aggregation array.

[0037] Optionally, also include:

[0038] After processing all the data sub-tables, the grouping array and the aggregation array are returned as the aggregation query result of the database aggregation query request, so as to display the aggregation query result in a display interface.

[0039] Optionally, the number of hash buckets in the hash table is determined according to the number of tuples in the data table, and the two are positively correlated.

[0040] Optionally, the number of hash buckets in the hash table is at least twice the number of tuples in the data table.

[0041] Optionally, dividing the data table into at least two data sub-tables includes:

[0042] According to the preset number of tuples contained in the data sub-table, the data table is divided into at least two data sub-tables; wherein the preset number of tuples is determined according to the parallel processing capability of the processor.

[0043] According to a second aspect of one or more embodiments of this specification, a resource scheduling system is provided, the resource scheduling system including:

[0044] A target resource node-side scheduler corresponding to any target resource node in the resource node cluster, configured to execute the steps of any one of the methods described in the first aspect;

[0045] The central scheduler corresponding to the resource node cluster is used to execute the steps of any one of the methods described in the first aspect.

[0046] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0047] processor;

[0048] a memory for storing processor-executable instructions;

[0049] Wherein, when the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0050] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any of the above methods are implemented.

[0051] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0052] In an embodiment of the present disclosure, in response to a database aggregation query request, the data table to be queried, the grouping fields, and the aggregation method are determined; a grouping array is created, and the grouping array is used to store different grouping fields in the data table; the data table is divided into at least two data sub-tables, and the following grouping processing and aggregation processing are performed on each data sub-table in turn.

[0053] When performing group processing, the group processing is divided into two parts to reduce the dependency between data. The first part is to search for matching group fields in the group array for multiple tuples in parallel; that is, for multiple tuples in the data sub-table, the following operations can be performed in parallel: check whether there is a target group field in the group array that is the same as the group field of the tuple; if so, record the array index of the target group field; this parallel process can effectively improve the grouping efficiency. The second part performs group processing on unmatched data, that is, after traversing all tuples in the data sub-table, for tuples to be processed that do not have an array index recorded, the group field of the tuple to be processed is added to the group array, and the array index of the group field of the tuple to be processed is recorded; and because the amount of data in the second part is limited after the group processing in the first part, the data pre-fetching capability of the processor can be fully utilized.

[0054] Finally, the aggregation process is performed, that is, the aggregation calculation is performed according to the array index of each tuple record in the data sub-table, the grouping array and the aggregation method to obtain the aggregate query result. This embodiment can effectively improve the query efficiency of group aggregation by dividing the grouping process into two parts.

[0055] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a schematic diagram of grouping and aggregating data tables in the related art.

[0057] Figure 2 This is a flowchart of an aggregate query method provided by an exemplary embodiment.

[0058] Figure 3 This is a schematic diagram of splitting a data table provided by an exemplary embodiment.

[0059] Figure 4 It is a schematic diagram of grouping data sub-table 1 provided by an exemplary embodiment.

[0060] Figure 5 It is a schematic diagram of grouping data sub-table 2 provided by an exemplary embodiment.

[0061] Figure 6 It is a schematic diagram of the aggregation result of data group 1 provided by an exemplary embodiment.

[0062] Figure 7 It is a schematic diagram of the aggregation result of data group 2 provided by an exemplary embodiment.

[0063] Figure 8 It is a structural diagram of a device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0065] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0066] To facilitate understanding of the embodiments of this specification, the relevant terms are first explained here:

[0067] Tuple: A fundamental concept in relational databases. A relation is a table, each row in the table (i.e., each record in the database) is a tuple, and each column is an attribute. In a two-dimensional table, a tuple is also called a row.

[0068] Database aggregate functions, also known as group functions, are usually used to count and calculate data in a table. They are generally used in conjunction with group by to count and calculate grouped data.

[0069] Commonly used aggregate functions include, but are not limited to: (1) count(col): calculates the total number of rows in a specified column (aggregate functions do not count null values); (2) max(col): calculates the maximum value in a specified column; (3) min(col): calculates the minimum value in a specified column; (4) sum(col): calculates the sum of a specified column; (5) avg(col): calculates the average value in a specified column. These functions are not limited to the following.

[0070] Assume that for the data table described in Table 1, when the user wants to query the total score of each person, he needs to use the name as the grouping field and sum the scores. The corresponding database aggregation query statement is: SELECT name, sum(score) FROM table GROUP BY name; this database aggregation query statement specifies the data table to be queried, the grouping field (grouped by name), and the aggregation method (summing the scores) to obtain the aggregation query results described in Table 2.

[0071] Table 1

[0072] Name Score Zhang San 95 Li Si 80 Zhang San 88 Wang Wu 65 Li Si 78

[0073] Table 2

[0074] Name Total Score Zhang San 183 Li Si 158 Wang Wu 65

[0075] When implementing group-and-aggregate queries, common database engines currently create a grouping array and an aggregation array. The grouping array is used to store the different grouping fields in the data table, and the aggregation array is used to store the aggregated results of the different grouping fields in the grouping array. For each tuple to be grouped in the data table, a check is first performed to determine whether there is a matching grouping field in the grouping array. If so, the aggregated results corresponding to the matching grouping field in the aggregation array are updated accordingly. If not, a new group is created in the grouping array (i.e., the grouping field of the tuple to be grouped is inserted into the grouping array).

[0076] For an example, see Figure 1 , assuming that the numbers in the data table need to be counted to determine the number of different numbers, the corresponding database aggregation query statement is: SELECT Number, Count(*) FROM table GROUP BY Number; for example, if the tuple with ID 5 is found at time T-1 and there is a matching grouping field 14 in the grouping array, the corresponding aggregation result in the aggregation array will be changed from 1 to 2; for example, if the tuple with ID 6 is found at time T and there is no matching grouping field in the grouping array, then "26" needs to be added to the grouping array, and the corresponding aggregation result in the aggregation array will be updated to 1. It should be noted that Figure 1 The grouping array and aggregation array in are the grouping aggregation results at time T. The arrays with ID 7 and later have not been traversed yet.

[0077] Considering the possibility that different tuples belong to the same group, for example, for two tuples in the same group, the latter tuple needs to depend on the group created by the former tuple. In other words, the latter tuple needs to be aggregated based on the group created by the former tuple, such as Figure 1The two tuples with IDs 4 and 7 need to depend on the tuple with ID 2 (the grouping field 15 of these three tuples belongs to the same group). Therefore, the fields that need to be aggregated can only be grouped sequentially and then aggregated, resulting in low aggregation query efficiency.

[0078] In response to the problems in the related art, the embodiments of this specification provide an aggregate query method, which responds to a database aggregate query request, determines the data table to be queried, the grouping field, and the aggregation method; creates a grouping array, which is used to store different grouping fields in the data table; divides the data table into at least two data sub-tables, and performs the following grouping processing and aggregation processing on each data sub-table in turn.

[0079] When performing group processing, the group processing is divided into two parts to reduce the dependency between data. The first part is to search for matching group fields in the group array for multiple tuples in parallel; that is, for multiple tuples in the data sub-table, the following operations can be performed in parallel: check whether there is a target group field in the group array that is the same as the group field of the tuple; if so, record the array index of the target group field; this parallel process can effectively improve the grouping efficiency. The second part performs group processing on unmatched data, that is, after traversing all tuples in the data sub-table, for tuples to be processed that do not have an array index recorded, the group field of the tuple to be processed is added to the group array, and the array index of the group field of the tuple to be processed is recorded; and because the amount of data in the second part is limited after the group processing in the first part, the data pre-fetching capability of the processor can be fully utilized.

[0080] Finally, the aggregation process is performed, that is, the aggregation calculation is performed according to the array index of each tuple record in the data sub-table, the grouping array and the aggregation method to obtain the aggregate query result. This embodiment can effectively improve the query efficiency of group aggregation by dividing the grouping process into two parts.

[0081] The aggregate query method provided in the embodiments of this specification can be executed by an electronic device equipped with a database engine, and the electronic device includes but is not limited to a server, a cloud server, a smart phone / mobile phone, a tablet computer, a personal digital assistant (PDA), a laptop computer, a desktop computer or any other device with computing capabilities. Among them, the database engine is a core service for storing, processing and protecting data. The database engine can be used to control access rights and process transactions quickly, thereby meeting the requirements of most applications within the enterprise that need to process large amounts of data. Use the database engine to create a relational database for online transaction processing or online analytical processing data. This includes creating tables for storing data and database objects (such as indexes, views and stored procedures) for viewing, managing and protecting data security.

[0082] Exemplarily, the electronic device includes a processor and a memory, the memory stores executable instructions that can be run on the processor, and the processor implements the aggregate query method provided in the embodiments of this specification when executing the executable instructions.

[0083] Exemplarily, the electronic device integrates a computer program product, and when the electronic device executes the computer program product, the aggregate query method provided in the embodiments of this specification is implemented.

[0084] Exemplarily, the aggregate query method provided in the embodiments of this specification can be used to perform aggregate queries on data stored in a cloud-native database.

[0085] In some embodiments, see Figure 2 , Figure 2 A schematic diagram of a process flow of an aggregate query method is shown. The method can be executed by an electronic device and includes:

[0086] In S101 , in response to a database aggregate query request, the data table to be queried, the grouping fields, and the aggregation method are determined.

[0087] In S102 , a grouping array is created, where the grouping array is used to store different grouping fields in the data table.

[0088] In S103, the data table is divided into at least two data sub-tables.

[0089] In S104, each data sub-table is processed in turn as follows: for multiple tuples in the data sub-table, the following operations are performed in parallel: check whether there is a target grouping field that is the same as the grouping field of the tuple in the grouping array; if so, record the array index of the target grouping field; after traversing all tuples in the data sub-table, for tuples to be processed whose array index is not recorded, add the grouping field of the tuple to be processed to the grouping array, and record the array index of the grouping field of the tuple to be processed; perform aggregation calculation based on the array index of each tuple record in the data sub-table, the grouping array and the aggregation method to obtain an aggregate query result.

[0090] In this embodiment, after the data table is divided into at least two data sub-tables, the following grouping and aggregation processing are performed on each data sub-table in turn. When performing the grouping processing, the grouping processing is divided into two parts to reduce the dependencies between the data. The first part is to search for matching grouping fields in the grouping array for multiple tuples in parallel. The parallel process is conducive to improving processing efficiency; the second part performs grouping processing on unmatched data. Since the amount of data in the second part is limited after the grouping processing in the first part, the data pre-fetching capability of the processor can be fully utilized. Finally, the aggregation processing process is performed to obtain the aggregated query result. By dividing the grouping process into two parts, the query efficiency of group aggregation can be effectively improved.

[0091] In a possible implementation, the database aggregation query request may carry a database aggregation query statement. The electronic device may determine the data table to be queried, the grouping field, and the aggregation method by parsing the database aggregation query statement.

[0092] In another possible implementation, the database aggregate query request may also be a result obtained by parsing a database aggregate query statement, and the database aggregate query request may directly carry the data table to be queried, the grouping field, and the aggregation method. This embodiment does not impose any restrictions on this.

[0093] In some embodiments, the electronic device may create a grouping array for storing different grouping fields in the data table. Before grouping is performed, the grouping array is empty, that is, the grouping array does not store any grouping fields.

[0094] In some embodiments, the electronic device may split the data table to be queried into at least two data sub-tables. Exemplarily, the electronic device may split the data table into at least two data sub-tables based on the number of preset tuples contained in the data sub-table; in one example, the number of preset tuples may be determined based on the parallel processing capability of the processor, i.e., the higher the parallel processing capability of the processor, the more preset tuples. Of course, it may also be set based on other factors (such as hardware prefetch capability) or by the developer based on their own experience, and this embodiment does not impose any restrictions on this.

[0095] For example, if there are 20 tuples in the data table and the preset number of tuples in the data sub-table is 5, the data table can be divided into 4 data sub-tables, each with 5 tuples.

[0096] In some embodiments, after dividing the data table into at least two data sub-tables, the electronic device may perform the following grouping and aggregation processing on each data sub-table in sequence.

[0097] Here is an example of the grouping process:

[0098] In one possible implementation, for multiple tuples in a data partition table, the electronic device can perform the following operations in parallel: traverse all grouping fields in the grouping array to determine whether the grouping array contains a target grouping field identical to the grouping field of the tuple; if so, indicating that the grouping field of the tuple and the target grouping field in the grouping array belong to the same group, the electronic device records the array index of the target grouping field; if not, the electronic device continues processing the next tuple. Then, after traversing all tuples in the data partition table, for any tuple to be processed for which the array index is not recorded, meaning that the grouping array does not store the grouping field of the tuple to be processed, the electronic device adds the grouping field of the tuple to be processed to the grouping array and records the array index of the grouping field of the tuple to be processed.

[0099] In another possible implementation, considering that the above implementation method needs to traverse all the grouping fields in the grouping array in the process of searching whether there is a target grouping field that is the same as the grouping field of the tuple, when the data volume is large, the search time consumed by this process is long.

[0100] Therefore, to improve search efficiency, this embodiment implements the creation of a hash table before sequentially performing grouping and aggregation processing on each data sub-table. The hash table includes hash buckets, with different hash buckets being used to store array indexes pointing to different grouping fields in the grouping array. As the electronic device sequentially processes each data sub-table, it can perform the following operations in parallel for multiple tuples in the data sub-table: performing a hash operation based on the grouping field in the tuple to obtain a hash value; determining the target hash bucket corresponding to the tuple in the hash table based on the hash value; determining whether the grouping field of the tuple is the same as the target grouping field indicated by the array index stored in the target hash bucket; if so, recording the array index of the target grouping field; if not, continuing to process the next tuple. After traversing all tuples in the data sub-table, for tuples to be processed that do not have an array index recorded, indicating that the grouping array does not store the grouping field of the tuple to be processed, the electronic device adds the grouping field of the tuple to be processed to the grouping array, stores the array index of the grouping field of the tuple to be processed in the target hash bucket corresponding to the tuple to be processed, and records the array index of the grouping field of the tuple to be processed. This embodiment uses a hash table to reduce the number of searches and improve the efficiency of searching for matching target group fields.

[0101] It is understood that the embodiments of this specification do not impose any restrictions on the construction method of the hash table, including but not limited to direct addressing method, square middle method, folding method, division with remainder method and random number method, etc. (1) Direct addressing method: For example, there is a population statistics table from 1 to 100 years old, where age is used as the keyword, and the hash function takes the keyword itself. (2) Square middle method: The middle few bits after the keyword is squared are taken as the hash address. (3) Folding method: The keyword is divided into several parts with the same number of bits, and the number of bits in the last part can be different, and then the sum of these parts is taken as the hash address. (4) Division with remainder method: The remainder obtained after the keyword is divided by a number p that is not greater than the length of the hash table m is taken as the hash address; H(key) = key MOD p (p <= m). (5) Random number method: Select a random function and take the random function value of the keyword as its hash address, that is, H(key) = random(key), where random is the random function.

[0102] For an example, see Figure 3 、 Figure 4 as well as Figure 5 , Figure 3 Shows the Figure 1 The data table in is divided into two data sub-tables. Assume that the construction method of the hash table is the division and remainder method, the table length of the hash table is 3, the divisor is 3, that is, the hash function is H(key) = key mod 3, where H(key) indicates the hash bucket, key indicates the grouping field, and mod is the remainder function, such as Figure 4 As shown, after performing the above grouping process on the data table 1, the following can be obtained: Figure 4 The hash table and grouping array shown in the figure take 14 as an example, 14 modulo 3 is 2, which points to hash bucket 2 (i.e. Figure 4 The array index 3 stored in bucket 2 points to 14 in the grouping array.

[0103] Next, we group the data in Table 2 and get the following: Figure 5 The hash table and grouping array are first processed in parallel on the 6 tuples in the data table 2, and the search is performed in Figure 4 Is there a matching target group array in the group array shown? For example, 10 is divided by 3 to get 1, which points to hash bucket 1 (i.e. Figure 41 in the data sub-table), the array index 0 stored in the hash bucket 1 (bucket 1) points to 10 in the grouping array. If the two are the same, the array index 0 of the target grouping field is recorded. After traversing all the tuples in the data sub-table, for the tuples to be processed whose array index is not recorded (4 groups of tuples with IDs of 0, 3, 4 and 5 in the data sub-table 2), the grouping field of the tuple to be processed is added to the grouping array, and the array index of the grouping field of the tuple to be processed is stored in the target hash bucket corresponding to the tuple to be processed, and the array index of the grouping field of the tuple to be processed is recorded; taking 26 (the tuple with ID 0 in the data sub-table 2) as an example, 26 is added to the grouping array, and 26 is modulo 3 to obtain 2, which points to hash bucket 2 (i.e. Figure 4 , stores the array index 4 of 26 in hash bucket 2 (bucket2) and records the array index 4.

[0104] Among them, considering the hash conflict problem, all the array indexes corresponding to the conflicting group fields are stored in the same linear linked list, such as Figure 4 and Figure 5 As shown, there is a hash conflict between 9 and 15, and the hash values are both 0. The corresponding array indexes are stored in the form of a linked list in hash bucket 0 (bucket0); for example, there is a hash conflict between 10 and 25, and the hash values are both 1. The corresponding array indexes are stored in the form of a linked list in hash bucket 1 (bucket1).

[0105] Exemplarily, in order to improve matching efficiency, when creating a hash table, the electronic device can determine the number of hash buckets in the hash table based on the number of tuples in the data table, and the two are positively correlated, that is, the more tuples in the data table, the more hash buckets there are, which is conducive to reducing or avoiding hash conflicts and improving matching efficiency.

[0106] In one example, the number of hash buckets in the hash table is at least twice the number of tuples in the data table, for example, the number of hash buckets in the hash table is a power of 2.

[0107] Exemplarily, the hash bucket in the hash table is used to store at least one array index using a linked list structure; after determining the target hash bucket corresponding to a tuple in the hash table, if the target hash bucket stores more than one array index, it is necessary to compare the grouping field of the tuple with the target grouping fields indicated by multiple array indexes in the target hash bucket until a successful match is found or all matches fail.

[0108] Exemplarily, for any tuple in a data partition table, the electronic device can perform a hash operation based on the grouping field in the tuple to obtain a hash value; determine the target hash bucket corresponding to the tuple in the hash table based on the hash value; if the target hash bucket is not empty, read the first target grouping field from the grouping array based on the first array index stored in the linked list structure of the target hash bucket; compare the grouping field of the tuple with the read target grouping field value; if the two are the same, record the array index of the read target grouping field; if the two are different, read the next target grouping field value from the grouping array based on the next array index stored in the linked list structure of the target hash bucket, and repeat the step of comparing the grouping field of the tuple with the read target grouping field value until a successful match is found or all matches fail.

[0109] For tuples to be processed whose array index is not recorded, if there are multiple tuples to be processed whose array index is not recorded, the electronic device can perform the following processing on the multiple tuples to be processed in sequence: for each tuple to be processed, determine the target hash bucket corresponding to the tuple to be processed in the hash table; if the target hash bucket is empty, or the grouping field of the tuple is different from the target grouping field indicated by the array index stored in the target hash bucket, add the grouping field of the tuple to be processed to the grouping array, store the corresponding array index in the target hash bucket corresponding to the tuple to be processed, and record the array index; if the grouping field of the tuple to be processed is the same as the target grouping field indicated by the array index stored in the target hash bucket, record the array index of the target grouping field.

[0110] For an example, see Figure 5 After traversing all tuples in the data sub-table 2, there are 4 groups of tuples with IDs 0, 3, 4, and 5 for which the array index is not recorded. Taking the two groups of tuples with IDs 4 and 5 as an example, for the tuple with ID 4, the group field 5 is modulo 3 to get 2, which points to hash bucket 2 (i.e. Figure 5 At this time, array indexes 3 and 4 are stored in hash bucket 2 (bucket 2), and the grouping fields pointed to by the two are 14 and 26 respectively, which are different from 5. Therefore, 5 is added to the grouping array, and the array index 6 of grouping field 5 in the grouping array is added to the end of the linked list of hash bucket 2, and the array index 6 is recorded; then, for the tuple with ID 5, the grouping field 5 is modulo 3 to get 2, which points to hash bucket 2 (that is, Figure 5 At this time, array indexes 3, 4, and 6 are stored in hash bucket 2 (bucket 2). After comparison, it is determined that group field 5 exists in the group array, and the array index 6 is recorded.

[0111] Here is an example of the aggregation process:

[0112] In some embodiments, before sequentially performing grouping and aggregation processing on each data sub-table, an aggregation array is first created. The aggregation array is used to store the aggregation results of different grouping fields in the grouping array; in other words, there is a one-to-one correspondence between the grouping fields in the grouping array and the aggregation results in the aggregation array. After the grouping process determines the array index for each tuple in the data sub-table, the electronic device can perform an aggregation calculation based on the array index of each tuple record in the data sub-table, the grouping array, and the aggregation method to obtain an aggregate query result.

[0113] Exemplarily, for each tuple in the data partition table, the electronic device reads the grouping field in the grouping array according to the array index, and determines whether there is an aggregation result of the grouping field in the aggregation array; if so, the aggregation result is updated using the aggregation method and the tuple; if not, the aggregation result corresponding to the grouping field of the tuple is calculated using the aggregation method and the tuple, and the aggregation result corresponding to the grouping field of the tuple is saved in the aggregation array.

[0114] For an example, see Figure 6 as well as Figure 7 , Figure 6 Show the Figure 4 The aggregate array after the data sub-table 1 is aggregated, Figure 7 Shown in the Figure 5 The data shown in Table 2 is aggregated into an aggregate array after aggregation. Figure 5 Taking the two tuples with IDs of 4 and 5 in the data sub-table 2 as an example, the array index of the two tuple records is 6. For the tuple with ID 4, after determining that the aggregation result of the grouping field 5 does not exist in the aggregation array, the aggregation result 1 is calculated using the aggregation method and the tuple, and 1 is saved in the aggregation array, and a correspondence is established between the aggregation result 1 and the grouping field 5 in the grouping array; then, for the tuple with ID 5, it is determined by searching that the aggregation result of the grouping field 5 exists in the aggregation array, and the aggregation result is updated using the aggregation method and the tuple, and the updated aggregation result is 2.

[0115] In some embodiments, after processing all the data sub-tables, the electronic device may return the grouping array and the aggregation array as the aggregate query result of the database aggregate query request, so as to display the aggregate query result in a display interface.

[0116] The various technical features in the above embodiments can be combined arbitrarily as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of the various technical features in the above embodiments also falls within the scope of disclosure of this specification.

[0117] Figure 8 This is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 8 At the hardware level, the device includes a processor 802, an internal bus 804, a network interface 806, a memory 808, and a non-volatile memory 810. Of course, it may also include hardware required for other services. One or more embodiments of this specification can be implemented based on software, such as the processor 802 reading the corresponding computer program from the non-volatile memory 810 into the memory 808 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0118] In some embodiments, the present disclosure further provides an electronic device, including:

[0119] processor;

[0120] a memory for storing processor-executable instructions;

[0121] The processor executes the following steps by running the executable instructions:

[0122] In response to a database aggregate query request, determine the data table to be queried, the grouping fields, and the aggregation method;

[0123] Creating a grouping array, wherein the grouping array is used to store different grouping fields in the data table;

[0124] The data table is divided into at least two data sub-tables, and each data sub-table is processed as follows:

[0125] For multiple tuples in the data table, the following operations are performed in parallel: checking whether there is a target grouping field in the grouping array that is the same as the grouping field of the tuple; if so, recording the array index of the target grouping field;

[0126] After traversing all tuples in the data sub-table, for tuples to be processed whose array index is not recorded, a grouping field of the tuple to be processed is added to the grouping array, and the array index of the grouping field of the tuple to be processed is recorded;

[0127] Aggregation calculation is performed according to the array index of each tuple record in the data sub-table, the grouping array and the aggregation method to obtain an aggregate query result.

[0128] In some embodiments, the processor is further configured to create a hash table; the hash table includes hash buckets, and different hash buckets are configured to store array indexes pointing to different grouping fields in the grouping array.

[0129] When searching for a target grouping field in the grouping array that is identical to the grouping field of the tuple, the processor is further configured to perform a hash operation based on the grouping field in the tuple to obtain a hash value; determine a target hash bucket corresponding to the tuple in the hash table based on the hash value; and determine whether the grouping field of the tuple is identical to the target grouping field indicated by the array index stored in the target hash bucket.

[0130] For a tuple to be processed whose array index is not recorded, the processor is further configured to store the array index of the grouping field of the tuple to be processed in a target hash bucket corresponding to the tuple to be processed.

[0131] In some embodiments, the hash bucket in the hash table is used to store at least one array index using a linked list structure. The processor is specifically configured to, if the target hash bucket is not empty, read the first target grouping field from the grouping array according to the first array index stored in the linked list structure in the target hash bucket; compare the grouping field of the tuple with the read target grouping field value; if the two are the same, record the array index of the read target grouping field; if the two are different, read the next target grouping field value from the grouping array according to the next array index stored in the linked list structure in the target hash bucket, and repeat the steps of comparing the grouping field of the tuple with the read target grouping field value.

[0132] In some embodiments, the processor is specifically used to: when there are multiple tuples to be processed whose array indexes are not recorded, perform the following processing on the multiple tuples to be processed in sequence: for each tuple to be processed, determine the target hash bucket corresponding to the tuple to be processed in the hash table; if the target hash bucket is empty, or the grouping field of the tuple is different from the target grouping field indicated by the array index stored in the target hash bucket, add the grouping field of the tuple to be processed to the grouping array, store the corresponding array index in the target hash bucket corresponding to the tuple to be processed, and record the array index; if the grouping field of the tuple to be processed is the same as the target grouping field indicated by the array index stored in the target hash bucket, record the array index of the target grouping field.

[0133] In some embodiments, the processor is further configured to create an aggregation array, wherein the aggregation array is configured to store aggregation results of different grouping fields in the grouping array.

[0134] When performing aggregation calculation based on the array index of each tuple record in the data sub-table, the grouping array and the aggregation method, the processor is specifically used to: for each tuple in the data sub-table, read the grouping field in the grouping array according to the array index, and determine whether there is an aggregation result of the grouping field in the aggregation array; if so, update the aggregation result using the aggregation method and the tuple; if not, use the aggregation method and the tuple to calculate the aggregation result corresponding to the grouping field of the tuple, and save the aggregation result corresponding to the grouping field of the tuple in the aggregation array.

[0135] In some embodiments, the processor is further configured to return the grouping array and the aggregation array as the aggregate query result of the database aggregate query request after processing all the data sub-tables, so as to display the aggregate query result in a display interface.

[0136] In some embodiments, the number of hash buckets in the hash table is determined according to the number of tuples in the data table, and the two are positively correlated.

[0137] In some embodiments, the number of hash buckets in the hash table is at least twice the number of tuples in the data table.

[0138] In some embodiments, the processor is further configured to divide the data table into at least two data sub-tables according to a preset number of tuples contained in the data sub-table; wherein the preset number of tuples is determined according to the parallel processing capability of the processor.

[0139] The implementation process of the functions and effects of each component in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0140] In some embodiments, the embodiments of this specification further provide a resource scheduling system, which includes:

[0141] A target resource node-side scheduler corresponding to any target resource node in the resource node cluster, configured to execute the steps of any one of the above methods;

[0142] The central scheduler corresponding to the resource node cluster is used to execute the steps of any one of the above methods.

[0143] In some embodiments, the embodiments of this specification further provide a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any of the methods described above.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0145] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0146] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0147] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0148] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0149] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0150] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0151] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0152] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."

[0153] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. An aggregate query method, comprising: In response to a database aggregate query request, determine the data table to be queried, the grouping fields, and the aggregation method; Creating a grouping array, wherein the grouping array is used to store different grouping fields in the data table; The data table is divided into at least two data sub-tables, and each data sub-table is processed as follows: For multiple tuples in the data table, the following operations are performed in parallel: checking whether there is a target grouping field in the grouping array that is the same as the grouping field of the tuple; if so, recording the array index of the target grouping field; After traversing all tuples in the data sub-table, for tuples to be processed whose array index is not recorded, a grouping field of the tuple to be processed is added to the grouping array, and the array index of the grouping field of the tuple to be processed is recorded; Aggregation calculation is performed according to the array index of each tuple record in the data sub-table, the grouping array and the aggregation method to obtain an aggregate query result.

2. The method according to claim 1, further comprising: Create a hash table; The hash table includes hash buckets, and different hash buckets are used to store array indexes pointing to different packet fields in the packet array; The step of searching whether there is a target grouping field in the grouping array that is the same as the grouping field of the tuple includes: Performing a hash operation on the grouping field in the tuple to obtain a hash value; Determine, according to the hash value, a target hash bucket corresponding to the tuple in the hash table; Determine whether the grouping field of the tuple is the same as the target grouping field indicated by the array index stored in the target hash bucket; For a tuple to be processed whose array index is not recorded, the method further includes: The array index of the grouping field of the tuple to be processed is stored in the target hash bucket corresponding to the tuple to be processed.

3. The method according to claim 2, wherein the hash bucket in the hash table is used to store at least one array index using a linked list structure; Determining whether the grouping field of the tuple is the same as the target grouping field indicated by the array index stored in the target hash bucket includes: If the target hash bucket is not empty, read the first target group field from the group array according to the first array index stored in the linked list structure of the target hash bucket; Compare the grouping field of the tuple with the target grouping field value read; If the two are the same, record the array index of the target group field to be read; If the two are different, read the next target grouping field value from the grouping array according to the next array index stored in the linked list structure of the target hash bucket, and repeat the step of comparing the grouping field of the tuple with the read target grouping field value.

4. The method according to claim 2, wherein for a tuple to be processed whose array index is not recorded, a grouping field of the tuple to be processed is added to the grouping array, the array index of the grouping field of the tuple to be processed is stored in the target hash bucket corresponding to the tuple to be processed, and the array index of the grouping field of the tuple to be processed is recorded, comprising: When there are multiple tuples to be processed whose array indexes are not recorded, the multiple tuples to be processed are processed in sequence as follows: For each tuple to be processed, determine the target hash bucket corresponding to the tuple to be processed in the hash table; If the target hash bucket is empty, or the grouping field of the tuple is different from the target grouping field indicated by the array index stored in the target hash bucket, the grouping field of the tuple to be processed is added to the grouping array, and the corresponding array index is stored in the target hash bucket corresponding to the tuple to be processed, and the array index is recorded; If the grouping field of the tuple to be processed is the same as the target grouping field indicated by the array index stored in the target hash bucket, the array index of the target grouping field is recorded.

5. The method according to claim 1, further comprising: Creating an aggregation array, wherein the aggregation array is used to store the aggregation results of different grouping fields in the grouping array; The performing aggregation calculation according to the array index of each tuple record in the data sub-table, the grouping array and the aggregation method includes: For each tuple in the data sub-table, read the grouping field in the grouping array according to the array index, and determine whether there is an aggregation result of the grouping field in the aggregation array; If it exists, update the aggregation result using the aggregation method and the tuple; If it does not exist, the aggregation result corresponding to the grouping field of the tuple is calculated using the aggregation method and the tuple, and the aggregation result corresponding to the grouping field of the tuple is saved in the aggregation array.

6. The method according to claim 5, further comprising: After processing all the data sub-tables, the grouping array and the aggregation array are returned as the aggregation query result of the database aggregation query request, so as to display the aggregation query result in a display interface.

7. The method according to any one of claims 2 to 4, wherein the number of hash buckets in the hash table is determined according to the number of tuples in the data table, and the two are positively correlated.

8. The method according to claim 7, wherein the number of hash buckets in the hash table is at least twice the number of tuples in the data table.

9. The method according to claim 1, wherein dividing the data table into at least two data sub-tables comprises: According to the preset number of tuples contained in the data sub-table, the data table is divided into at least two data sub-tables; wherein the preset number of tuples is determined according to the parallel processing capability of the processor.

10. A resource scheduling system, comprising: A target resource node-side scheduler corresponding to any target resource node in the resource node cluster, configured to execute the steps of the method according to any one of claims 1 to 9; The central scheduler corresponding to the resource node cluster is used to execute the steps of the method described in any one of claims 1 to 9.

11. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 9 by running the executable instructions.

12. A computer-readable storage medium having computer instructions stored thereon, which implement the steps of the method according to any one of claims 1 to 9 when executed by a processor.

Images