A data grouping method, device, medium, and program product of a database

Abstract

The application discloses a database data grouping method, device, medium and program product in the computer technology field. The application can use a preset TOP number of rows to constrain a hash grouping process, so that a grouping stage can perceive a TOP number of rows upper limit. When the number of groups in the first hash table reaches the capacity, newly appearing grouping key data is no longer directly inserted into the hash table, but is respectively classified into two types of overflow data according to whether the grouping key data is likely to be repeated, so as to avoid allocating memory for groups exceeding the TOP requirement. When the number of groups in the first hash table has met the TOP number of rows, new data that cannot be combined is directly discarded, and no additional memory and processing overhead is generated. For the overflow data, a second hash table is used for processing, and the upper limit of memory occupation is further controlled. Thus, the grouping operation and the TOP operation are synchronously performed, the hash table only needs to maintain groups meeting the TOP requirement, and the occupation of memory resources by the grouping operation and the TOP operation is reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data grouping method, device, medium and program product for a database. Background Technology

[0002] In database query processing, grouping and TOP operations are typically executed sequentially as two independent phases. The grouping phase aggregates all input data by the grouping key. With hash grouping, when the data volume is large or the cardinality of the grouping key is high, the hash table needs to maintain a large number of groups and their set function values, resulting in significant memory consumption. After all grouping is complete, the top few rows are selected as the TOP result based on sorting or filtering conditions. During this process, the grouping phase cannot know the final upper limit of the required number of rows. Even if only a small number of groups are needed later, memory space must still be allocated and maintained for all possible groups in the early stages, leading to a large number of ultimately discarded groups consuming limited memory resources.

[0003] Therefore, how to reduce the memory resource consumption of grouping and TOP operations is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a data grouping method, device, medium and program product for a database, so that the grouping operation and the TOP operation are performed synchronously, thereby reducing the memory resource consumption of the grouping operation and the TOP operation.

[0005] Firstly, this application provides a data grouping method for a database, including: Retrieve data to be grouped in batches; The data to be grouped is grouped using a preset first hash table. If the number of groups in the first hash table is less than the preset number of top rows and less than the capacity of the first hash table, then if there is an existing group in the first hash table with the same key value as the current row, the set function value of the existing group is updated according to the current row; if there is no existing group, the current row is inserted as a new group into the first hash table; wherein, the capacity is the minimum value between the preset physical upper limit and the preset number of top rows. If the number of groups is not less than the preset number of TOP rows, then the set function value is updated when the existing group exists, and the current row is discarded when the existing group does not exist. If the number of groups reaches the capacity but does not reach the preset TOP row number, then the set function value is updated if an existing group exists; if no existing group exists, and if the current row belongs to a non-repeating group and the sum of the current group number and the number of rows in the current first overflow data does not reach the preset TOP row number, then the current row is assigned to the first overflow data; if the current row belongs to a non-repeating group and the sum of the current group number and the number of rows in the current first overflow data reaches the preset TOP row number, then the current row is discarded; if the current row cannot be determined to be a non-repeating group, then the current row is assigned to the second overflow data. Once all data to be grouped has been processed, output the latest first hash table; The first overflow data and the second overflow data are grouped and processed using a preset second hash table, and finally the latest second hash table is output; the capacity of the second hash table is the minimum value between the preset physical upper limit and the target remaining number of rows; the target remaining number of rows is the difference between the preset TOP number of rows and the total number of rows in the latest first hash table; Based on the latest first hash table and the latest second hash table, grouping results that meet the preset TOP row number are obtained.

[0006] Optionally, it also includes: If the number of groups reaches the capacity, does not reach the preset TOP row number, and the sum of the current group number and the current first overflow data row number reaches the preset TOP row number, then if there is no existing group, if the current row belongs to a non-duplicate group, the current row is discarded; if the current row cannot be determined to be a non-duplicate group, the current row is assigned to the second overflow data.

[0007] Optionally, a composite filter can be used to detect whether the current row belongs to a non-repeating group; Accordingly, a composite filter is used to detect whether the current row belongs to a non-repeating group, including: If any sub-filter in the composite filter determines that the grouping key of the current row is not repeated, then the current row is determined to belong to a non-repeating group; otherwise, it is impossible to determine whether the current row is a non-repeating group.

[0008] Optionally, the step of grouping the first overflow data and the second overflow data using a preset second hash table, and finally outputting the latest second hash table, includes: Data is retrieved in batches from the second overflow data; The second hash table is used to group the acquired rows of data. If the current number of groups in the second hash table is less than the target number of remaining rows and less than the capacity of the second hash table, then if there is an existing group in the second hash table with the same key value as the current row, the set function value of the group is updated according to the current row and the current row is marked as removed; if there is no existing group, the current row is inserted into the second hash table as a new group and the current row is marked as removed. If the current number of groups reaches the target number of remaining rows, then if there is an existing group, update the set function value of that group; otherwise, discard the current row. If the current number of groups reaches the capacity of the second hash table but does not reach the target number of remaining rows, then an overflow handling operation is performed; If the second overflow data is processed and the current group number reaches the target remaining row number, then data is obtained batch by batch from the first overflow data. For each obtained row of data, if there is an existing group with the same key value in the second hash table, the set function value of the group is updated according to the current row and the current row is discarded; if there is no existing group, the current row is discarded. If the first overflow data has been processed, the current second hash table is output as the latest second hash table.

[0009] Optionally, it also includes: If the second overflow data is processed and the current group number has not reached the target remaining number of rows, then data is obtained from the first overflow data in batches, and for each obtained row of data; If an existing group with the same key value exists in the second hash table, the set function value of the group is updated according to the current row and the current row is discarded; if it does not exist, the current row is output and the number of output rows of the first overflow data is accumulated; wherein, the number of output rows is merged into the second hash table after the first overflow data is processed; When the sum of the current group number, the output row number, and the total row number reaches the preset TOP row number, for the subsequently acquired first overflow data, if there is an existing group with the same key value, the set function value is updated and the current row is discarded; otherwise, the current row is discarded and no output operation is performed.

[0010] Optionally, the overflow handling operation includes: Traverse the first overflow data and the second overflow data; after merging the row data that can be merged with the existing group in the second hash table, remove the mark; if the sum of the total number of rows and the current number of groups reaches the preset TOP row number, output the current second hash table as the latest second hash table; otherwise, output the current second hash table, record the cumulative output row number of the second hash table, and reinitialize the second hash table; Traverse the unremoved rows in the second overflow data; merge the rows that can be merged with existing groups in the second hash table and mark them as removed; otherwise, insert the current row as a new group into the second hash table and mark it as removed. Traverse the unremoved rows in the first overflow data, merge the rows that can be merged with the existing groups in the second hash table, and mark them as removed; if the sum of the total number of rows, the current number of groups, and the cumulative number of output rows in the second hash table reaches the preset TOP number of rows, then output the current second hash table; otherwise, output the current second hash table, update the cumulative number of output rows in the second hash table, reinitialize the second hash table, and continue processing the unremoved rows.

[0011] Optionally, it also includes: When the preset TOP row number is less than or equal to the total number of rows, the calculation of the target remaining row number and the grouping processing of the first overflow data and the second overflow data are skipped, and the grouping result that meets the preset TOP row number is obtained based on the latest first hash table.

[0012] Secondly, this application provides a data grouping device for a database, comprising: The acquisition module is used to acquire data to be grouped in batches; The first grouping module is used to group the rows of data in the data to be grouped using a preset first hash table. If the number of groups in the first hash table is less than the preset number of top rows and less than the capacity of the first hash table, then if there is an existing group with the same key value as the current row in the first hash table, the set function value of the existing group is updated according to the current row; if there is no existing group, the current row is inserted as a new group into the first hash table; wherein, the capacity is the minimum value between a preset physical upper limit and a preset number of top rows; if the number of groups is not less than the preset number of top rows, then if there is an existing group, the set function is updated. If no existing group exists, the current row is discarded; if the number of groups reaches the capacity but does not reach the preset TOP row number, the set function value is updated if an existing group exists; if no existing group exists, and the current row belongs to a non-repeating group, and the sum of the current group number and the number of rows in the current first overflow data does not reach the preset TOP row number, the current row is assigned to the first overflow data; if the current row belongs to a non-repeating group, and the sum of the current group number and the number of rows in the current first overflow data reaches the preset TOP row number, the current row is discarded; if the current row cannot be determined to be a non-repeating group, the current row is assigned to the second overflow data. The first output module is used to output the latest first hash table after all the data to be grouped has been processed. The second grouping module is used to group the first overflow data and the second overflow data using a preset second hash table, and finally output the latest second hash table; the capacity of the second hash table is the minimum of the preset physical upper limit and the target remaining number of rows; the target remaining number of rows is the difference between the preset TOP number of rows and the total number of rows in the latest first hash table; The second output module is used to obtain grouping results that meet the preset number of TOP rows based on the latest first hash table and the latest second hash table.

[0013] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the aforementioned disclosed database data grouping method.

[0014] Fourthly, this application provides a non-volatile storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned database data grouping method.

[0015] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned disclosed database data grouping method.

[0016] As can be seen, this application can constrain the hash grouping process by using a preset TOP row count, enabling the grouping phase to be aware of the TOP row count limit. Specifically, the capacity of the first hash table is the smaller of a preset physical limit and a preset TOP row count. When the number of groups in the first hash table reaches this capacity, newly appearing group key data is no longer directly inserted into the hash table. Instead, it is categorized into two types of overflow data based on whether it might be a duplicate group, thus avoiding continuous memory allocation for groups exceeding the TOP requirement. When the number of groups in the first hash table already meets the TOP row count, subsequent new data that cannot be merged is directly discarded, preventing additional memory and processing overhead. For overflow data, a second hash table with a capacity bound to the target remaining row count is used for processing, further controlling the upper limit of memory usage. Therefore, useless data can be identified and discarded as early as possible during the grouping process. The hash table only needs to maintain a number of groups equivalent to the TOP requirement, effectively alleviating memory pressure. This allows grouping operations and TOP operations to be performed synchronously, reducing the memory resource consumption of grouping and TOP operations.

[0017] Correspondingly, the database data grouping device, equipment, medium, and program product provided in this application also have the above-mentioned technical effects. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0019] Figure 1 This is a flowchart of the data grouping method for the first database disclosed in this application; Figure 2 This is a flowchart of the data grouping method for the second database disclosed in this application; Figure 3 This is a flowchart of an overflow handling operation method disclosed in this application; Figure 4 This is a schematic diagram of a data grouping device for a database disclosed in this application; Figure 5 This is a schematic diagram of an electronic device disclosed in this application; Figure 6 A server architecture diagram provided in this application; Figure 7 A terminal structure diagram provided for this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0021] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0022] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Currently, the maximum number of rows required at the grouping stage is unknown. Even if only a small number of groups are needed later, memory space still needs to be allocated and maintained for all possible groups in the early stages, resulting in a large number of ultimately discarded groups occupying limited memory resources. To address this, this application provides a database data grouping scheme that enables grouping operations to be performed synchronously with TOP operations, reducing the memory resource consumption of both grouping and TOP operations.

[0024] This application discloses a data grouping method for a database. This method introduces the TOP row number constraint into the hash grouping process in advance. By dynamically adjusting the hash table capacity and diverting overflowing data, data exceeding the requirement is discarded during the grouping stage, which can reduce memory pressure.

[0025] See Figure 1 As shown, the method includes: S101. Obtain the data to be grouped in batches.

[0026] The data to be grouped typically originates from child nodes in the database query execution plan and is returned in batches. Each batch contains several rows of data, and each row contains at least a grouping key and a measure. The grouping key is the column value used to divide the data into groups, such as grouping by user ID or region; the measure is the numerical value that needs to be aggregated, such as the amount spent or the number of visits.

[0027] S102. Use the preset first hash table to group the data of each row of the data to be grouped.

[0028] Before performing specific grouping operations using the first hash table, a preset TOP row number can be set. This TOP row number represents the maximum number of groups required to be returned in the query, and its value can be derived from the upper-level operators. Simultaneously, the maximum physical capacity of the hash table, denoted as `max_rowcnt_org`, is determined based on available system memory or configuration parameters. The first hash table is initialized, and its capacity `max_rowcnt` is set to the smaller of the preset physical upper limit and the preset TOP row number; that is, `max_rowcnt` equals the minimum of `max_rowcnt_org` and `top_num`, ensuring that the first hash table does not exceed the required memory space. Furthermore, a composite filter is initialized to quickly determine whether a grouping key may have already appeared. This composite filter can specifically include multiple filters such as a maximum value filter, a minimum value filter, and a Bloom filter.

[0029] In one example, for each row of data, the following different branches of processing logic are executed based on the current number of groups and capacity status of the first hash table. Therefore, step S102 can specifically include: S1021. If the number of groups in the first hash table is less than the preset number of TOP rows and less than the capacity of the first hash table, then if there is an existing group in the first hash table with the same key value as the current row, the set function value of the existing group is updated according to the current row; if there is no existing group, the current row is inserted as a new group into the first hash table. The capacity of the first hash table is the minimum of the preset physical upper limit and the preset number of TOP rows.

[0030] As can be seen, if the current number of groups in the first hash table is less than the preset TOP row number and also less than the capacity of the first hash table, then a regular grouping operation is performed. Specifically, it checks whether there is already an existing group in the first hash table with the same grouping key as the current row. If it exists, the set function value of the existing group is updated according to the current row. The set function is an aggregation calculation function, such as summation, counting, averaging, etc. The update method can be to add the measure value of the current row to the cumulative value of the existing group. If it does not exist, the current row is inserted as a new group into the first hash table, and the set function value of the group is initialized according to the measure value of the current row. After processing a batch of data, if the number of groups in the first hash table still has not reached the TOP row number and has not reached the capacity, then the next batch of data is obtained and processed according to this rule; if the TOP row number has been reached, then proceed to S1022; if the capacity has been reached, then proceed to S1023.

[0031] S1022. If the number of groups is not less than the preset number of TOP rows, update the set function value when there are existing groups, and discard the current row when there are no existing groups. This achieves the goal of merging rows without adding new rows after the TOP demand is saturated.

[0032] As can be seen, if the current number of groups in the first hash table is already not less than the preset number of TOP rows, meaning the TOP requirement has been met or exceeded, there is no need to add new groups. In this case, for each subsequent row of data, if an existing group with the same grouping key can be found in the first hash table, the set function value of that group is also updated; if not found, the current row is discarded without allocating any memory for it. This rule is continued to process all subsequent batches of data until all data has been acquired. This ensures that when the TOP requirement is met, only existing groups are merged and updated, avoiding exceeding the result limit and memory usage caused by adding new groups.

[0033] S1023. If the number of groups reaches the capacity but does not reach the preset TOP row number, the set function value is updated if an existing group exists. If no existing group exists, if the current row belongs to a non-duplicate group and the sum of the current group number and the current first overflow data row number does not reach the preset TOP row number, the current row is assigned to the first overflow data. If the current row belongs to a non-duplicate group and the sum of the current group number and the current first overflow data row number reaches the preset TOP row number, the current row is discarded. If the current row cannot be determined to be a non-duplicate group, the current row is assigned to the second overflow data. Thus, after the first hash table is full, a filter is used to split new data into first overflow data or second overflow data, and non-duplicate data is discarded when the total amount of overflow data reaches the TOP.

[0034] As can be seen, if the current number of groups in the first hash table has reached its capacity but has not yet reached the preset TOP row count, it means that the hash table memory is full but the number of groups is still insufficient for the TOP requirement, and a filter is needed to distribute the subsequent data. For the current row, if an existing group with the same key value can be found in the first hash table, the set function value of that group is updated. If no existing group is found, the filter is used to determine whether the group key of the current row belongs to a non-repeating group, and the filter state is updated to record the appearance of the group key. The filter can be a composite filter, that is, containing multiple sub-filters. The determination rule is that if any sub-filter in the composite filter determines that the group key of the current row has never appeared, then the overall filter determines that the group key is non-repeating, and the current row belongs to a non-repeating group; if all sub-filters cannot determine that the group key is non-repeating, then the overall filter cannot determine that it is non-repeating.

[0035] Therefore, in one implementation, a composite filter is used to detect whether the current row belongs to a non-repeating group; correspondingly, detecting whether the current row belongs to a non-repeating group using a composite filter includes: if any sub-filter in the composite filter determines that the grouping key of the current row is non-repeating, then the current row is determined to belong to a non-repeating group; otherwise, the current row cannot be determined to be a non-repeating group.

[0036] When the filter determines that the current row belongs to a non-duplicate group, it further checks whether the sum of the number of groups in the current first hash table and the number of rows already included in the first overflow data reaches the preset TOP row number. If the sum does not reach the preset TOP row number, the current row is included in the first overflow data, which is used to temporarily store rows that have been identified as new groups but whose final need is uncertain. If the sum has reached the preset TOP row number, the current row is discarded. When the filter cannot determine that the row is non-duplicate, the current row is included in the second overflow data, which is used to temporarily store rows that may be merged with existing groups. After processing a batch of data, if the sum of the number of groups in the first hash table and the number of rows in the first overflow data still does not exceed the TOP row number, the next batch of data is obtained and processed according to this rule; once the sum exceeds the preset TOP row number, for each subsequent row of data, if there is no existing group with the same key value in the first hash table, if the filter determines that the row is non-duplicate, the current row is discarded and the filter is updated; if the filter cannot determine that the row is non-duplicate, it is included in the second overflow data and the filter is updated, and it is no longer included in the first overflow data. This avoids accumulating the first overflow data after the total number of potential groups has met the TOP requirement, thus controlling memory usage. The above process continues, with each batch of data being processed before the next batch is acquired and processed according to the same rules, until all data to be grouped has been acquired.

[0037] Therefore, in one implementation, it further includes: if the number of groups reaches the capacity, does not reach the preset TOP row number, and the sum of the current number of groups and the number of rows of the current first overflow data reaches the preset TOP row number, then if there are no existing groups, if the current row belongs to a non-duplicate group, the current row is discarded; if the current row cannot be determined to be a non-duplicate group, the current row is assigned to the second overflow data.

[0038] S103. After all the data to be grouped has been processed, output the latest first hash table.

[0039] In other words, after all the data to be grouped has been processed, all groups in the first hash table are output, and this first hash table becomes the latest first hash table. At this time, the total number of rows in the first hash table can be recorded, denoted as the number of output rows of the first hash table, m1_end.

[0040] S104. The first overflow data and the second overflow data are grouped and processed using a preset second hash table, and the latest second hash table is finally output.

[0041] As can be seen, after the first hash table outputs the data, a second hash table with limited capacity is used to process the overflow data to obtain the final result. The capacity of the second hash table is the minimum of the preset physical upper limit and the target remaining number of rows; the target remaining number of rows is the difference between the preset top row count and the total number of rows in the latest first hash table, representing the maximum number of groups that can be added to the second hash table. When initializing the second hash table, its capacity is taken as the smaller of the preset physical upper limit and the target remaining number of rows to strictly control subsequent memory usage.

[0042] It should be noted that when the preset physical upper limit is greater than or equal to the target remaining number of rows, the capacity of the second hash table is equal to the target remaining number of rows. At this time, "the current number of groups has reached the capacity" and "the target remaining number of rows has been reached" are the same condition. The two branches mentioned later will naturally merge in actual execution, and there will be no situation where the capacity is full but the target remaining number of rows is not full. Only when the preset physical upper limit is less than the target remaining number of rows is it necessary to handle the overflow operation after the capacity is full separately.

[0043] Overflow data processing prioritizes the second overflow data set, as it is more likely to be merged with existing groups. Rows are retrieved batch by batch from the second overflow data set, and the following operations are performed on each row: If the current number of groups in the second hash table is less than the target remaining row count and less than the capacity of the second hash table, normal processing is performed. If an existing group with the same key value exists in the second hash table, the set function value of that group is updated based on the current row, and the current row is marked as removed, indicating that the row data has been absorbed. If no such group exists, the current row is inserted into the second hash table as a new group and marked as removed. If the current number of groups in the second hash table has reached the target remaining row count, subsequent updates only apply to rows that can be merged with existing groups; new group rows that cannot be merged are discarded. If the current number of groups in the second hash table reaches its capacity but has not yet reached the target remaining row count, an overflow processing operation is triggered. This overflow processing operation is used to recycle limited memory space and extract as many groups as possible from the overflow data as possible that meet the TOP requirements.

[0044] Therefore, in one implementation, a preset second hash table is used to group the first overflow data and the second overflow data, and finally output the latest second hash table. This includes: obtaining data in batches from the second overflow data; grouping the obtained rows of data using the second hash table; if the current number of groups in the second hash table is less than the target remaining number of rows and less than the capacity of the second hash table, then if there is an existing group in the second hash table with the same key value as the current row, the set function value of the group is updated according to the current row and the current row is marked as removed; if there is no existing group, the current row is inserted as a new group into the second hash table and the current row is marked as removed; if the current number of groups reaches... If the target number of remaining rows is reached, and an existing group exists, update the set function value of that group; otherwise, discard the current row. If the current number of groups reaches the capacity of the second hash table but does not reach the target number of remaining rows, perform an overflow handling operation. If the second overflow data processing is complete and the current number of groups reaches the target number of remaining rows, retrieve data batch by batch from the first overflow data. For each retrieved row, if an existing group with the same key exists in the second hash table, update the set function value of that group based on the current row and discard the current row; otherwise, discard the current row. If the first overflow data processing is complete, output the current second hash table as the latest second hash table.

[0045] Once the second overflow data has been processed, the first overflow data is processed based on whether the second hash table is full. If the current number of groups in the second hash table has reached the target number of remaining rows, data is retrieved batch by batch from the first overflow data. For each row of data, if an existing group with the same key value exists in the second hash table, the set function value of that group is updated and the current row is discarded; otherwise, the current row is discarded directly. After processing all the first overflow data, the current second hash table is output as the latest second hash table.

[0046] If the number of current groups in the second hash table has not yet reached the target number of remaining rows when the second overflow data is processed, data is retrieved batch by batch from the first overflow data, and the following operations are performed on each row. If an existing group with the same key value exists, its set function value is updated and the current row is discarded; otherwise, the current row is directly output, and the number of output rows of the first overflow data is accumulated. When the sum of the current number of groups in the second hash table, the accumulated number of output rows of the first overflow data, and the number of output rows of the first hash table reaches the preset TOP row number, for the subsequently retrieved first overflow data, the set function value is updated and the current row is discarded only if an existing group with the same key value exists; otherwise, the current row is discarded, and no further output operations are performed. After all the first overflow data has been processed, this accumulated number of output rows is considered part of the output result of the second hash table, and together with the groups in the second hash table, constitutes the output content of the second hash table.

[0047] Therefore, in one implementation, if the second overflow data has been processed and the current group number has not reached the target remaining row number, data is obtained batch by batch from the first overflow data. For each obtained row of data, if there is an existing group with the same key value in the second hash table, the set function value of the group is updated according to the current row and the current row is discarded; if there is no existing group, the current row is output and the number of output rows of the first overflow data is accumulated. The number of output rows is added to the second hash table after the first overflow data has been processed. When the sum of the current group number, the number of output rows and the total number of rows reaches the preset TOP row number, for the subsequently obtained first overflow data, if there is an existing group with the same key value, the set function value is updated and the current row is discarded; if there is no existing group, the current row is discarded and no output operation is performed.

[0048] The overflow handling operation described above is triggered when the second hash table is full but the target number of remaining rows has not yet been reached. The specific process can include the following stages: In the first stage, the second and first overflow data are traversed. Rows that can be merged with existing groups in the second hash table are merged and marked as removed. Rows that cannot be merged are skipped. Rows skipped in the first stage remain unremoved and will not be discarded during this stage. They will be re-traversed and participate in further grouping in the subsequent second stage. After traversal, it is checked whether the sum of the number of output rows from the first hash table and the number of groups in the current second hash table reaches the preset TOP row count. If it does, all groups in the current second hash table are output and the overflow handling ends. If it does not, all groups in the current second hash table are output, the cumulative number of output rows in the second hash table is recorded, the second hash table is reinitialized, and rows marked as removed in the second and first overflow data are treated as non-existent. The traversal progress is reset, and the second stage begins. In the second phase, the remaining rows in the second overflow data are traversed. Rows that can be merged with existing groups in the second hash table are merged and marked as removed. Rows that cannot be merged are inserted into the second hash table as new groups and marked as removed. Then, the remaining rows in the first overflow data are traversed. Rows that can be merged are merged and marked as removed. Rows that cannot be merged are skipped. Next, the sum of the number of output rows in the first hash table, the current number of groups in the second hash table, and the cumulative number of output rows in the second hash table is checked to see if it reaches the preset TOP row count. If it does, all groups in the current second hash table are output and the process ends. If it does not, all groups in the current second hash table are output, the cumulative output row count is updated, the second hash table is reinitialized, and the traversal progress is reset. The second phase is repeated until the TOP row count requirement is met or all data has been traversed. This two-phase loop processing method maximizes the group merging effect within limited memory by absorbing data from existing groups in the first phase and allowing new groups only in the second phase. This reduces the gap between the number of output rows and the TOP row count.

[0049] Therefore, in one embodiment, the overflow handling operation includes: traversing the first overflow data and the second overflow data; performing a merging operation on row data that can be merged with existing groups in the second hash table, and marking them as removed; if the sum of the total number of rows and the current number of groups reaches a preset TOP row number, then outputting the current second hash table as the latest second hash table; otherwise, outputting the current second hash table, recording the cumulative output row number of the second hash table, and reinitializing the second hash table; traversing the unremoved rows in the second overflow data; performing a merging operation on row data that can be merged with existing groups in the second hash table, and marking them as removed; otherwise, inserting the current row data as a new group into the second hash table, and marking it as removed; traversing the unremoved rows in the first overflow data, performing a merging operation on row data that can be merged with existing groups in the second hash table, and marking them as removed; if the sum of the total number of rows, the current number of groups, and the cumulative output row number of the second hash table reaches a preset TOP row number, then outputting the current second hash table; otherwise, outputting the current second hash table, updating the cumulative output row number of the second hash table, reinitializing the second hash table, and continuing to process the unremoved rows.

[0050] S105. Based on the latest first hash table and the latest second hash table, obtain the grouping results that meet the preset number of TOP rows.

[0051] That is, the groups output by the first hash table and the groups output by the second hash table are combined together, and the total number does not exceed the preset TOP row number, which is used as the final result of the query.

[0052] Furthermore, in a special case, if the number of output rows of the first hash table has reached or exceeded the preset TOP row count after processing all the data to be grouped, the calculation of the target remaining row count and all processing steps for the first and second overflow data can be skipped, and the grouping in the latest first hash table can be directly used as the grouping result that meets the preset TOP row count, thereby saving subsequent processing overhead. Therefore, in one embodiment, it further includes: when the preset TOP row count is less than or equal to the total number of rows, skipping the calculation of the target remaining row count and the grouping processing for the first and second overflow data, and obtaining the grouping result that meets the preset TOP row count based on the latest first hash table.

[0053] In this embodiment, if any step fails to acquire data in batches, including when no data is returned from the child node or the data source is exhausted prematurely, all grouped data already present in the first and second hash tables will be output directly, and processing will terminate. These acquisition steps include the initial stage of acquiring data to be grouped from the child node, subsequent acquisition of data from the second overflow data, and acquisition of data from the first overflow data. This mechanism ensures that the processed grouping results will not be lost in the event of an abnormal interruption of the data flow.

[0054] As can be seen, this embodiment introduces a TOP constraint during the grouping phase. The capacity of the first hash table is constrained by the number of TOP rows. Once the number of groups reaches the capacity or the number of TOP rows, subsequent useless data is discarded or distributed as early as possible, no longer occupying hash table memory. The capacity of the second hash table is further bound to the remaining available number of groups, ensuring that overall memory consumption remains controllable. Simultaneously, a filter is used to divide data into two categories: non-duplicate and potentially duplicate data. Potentially duplicate data is processed first to merge groups, improving grouping efficiency. This method is suitable for database query scenarios that require grouping and aggregation with a limit on the number of returned rows, especially in cases with large data volumes and a small number of required groups, significantly reducing memory pressure and improving execution speed.

[0055] The following provides a method flow for pushing down the TOP operation to the data grouping process of the database in the hash grouping process.

[0056] First, let's introduce the main parameters involved in this method flow. The preset TOP row count (top_num) is the maximum number of groups the query requires to return, derived from the upper-level operator. The physical maximum capacity of the first hash table (max_rowcnt_org) is the upper limit of the hash table during processing, determined by available system memory or configuration parameters. The design capacity of the first hash table (max_rowcnt) is the smaller of the physical maximum capacity and the preset TOP row count. Segment A (first overflow data) is used to store new data determined to be non-duplicate, and segment B (second overflow data) is used to store data that cannot be determined to be non-duplicate, i.e., data that may be duplicated. Filters are used to quickly determine whether a group key may have already appeared. Their characteristic is that the result of determining non-duplicate is definite, while the result of determining possible duplicates may contain false positives. For ease of description, the number of groups will be referred to as the "row count" in the following text.

[0057] This embodiment achieves unified decision-making on TOP limitation and memory control by explicitly setting the capacity of the first hash table to the minimum of the preset physical upper limit and the preset TOP row count. When the preset TOP row count is less than or equal to the preset physical upper limit, the capacity of the first hash table is directly equal to the preset TOP row count. Therefore, when the number of groups in the first hash table grows to its capacity, it means that the TOP requirement has been met. At this point, there is no need to maintain additional counters or perform complex branch judgments; the optimization strategy of discarding subsequent new group data can be directly triggered, simplifying the judgment logic and shortening the execution path. Simultaneously, since the hash table capacity is limited by the TOP row count, the memory space occupied by the hash table is naturally constrained to the minimum range required to meet query needs, avoiding the waste of reserving excess memory to accommodate more groups. In typical online analytical query scenarios where the preset TOP row count is much smaller than the physical upper limit, the memory usage of the hash table is significantly reduced, allowing more memory to be used for concurrent queries, caching, or other operators, improving the overall throughput of the database system. This capacity setting method also makes the expansion overhead of the hash table controllable. Once the hash table reaches its capacity, it stops adding new groups, eliminating the need to rehash existing data due to expansion, thus reducing CPU overhead.

[0058] Please see Figure 2 The following are the detailed execution steps.

[0059] Step 1: Set initial conditions and initialize relevant structures.

[0060] Determine the preset number of top rows, `top_num`, based on the specific execution plan. Determine the maximum physical capacity of the first hash table, `max_rowcnt_org`, based on factors such as memory limitations. Initialize the first hash table, setting its capacity `max_rowcnt` to the smaller of `max_rowcnt_org` and `top_num`, i.e., `max_rowcnt` equals `MIN(max_rowcnt_org, top_num)`. Simultaneously initialize the filter. After completion, proceed to step 2.

[0061] Step 2: Obtain the next batch of data to be grouped from the child nodes.

[0062] If the data retrieval is successful and returned, proceed to step 3 for processing. If the retrieval fails and no data is returned, proceed to step 22 to perform a fallback output.

[0063] Step 3: Based on the first hash table, process the data to be grouped in this batch row by row.

[0064] For any row of data, if no existing group with the same grouping key is found in the first hash table, the row is inserted as a new group into the first hash table; if an existing group with the same grouping key is found, the set function value of the existing group is updated according to the data of the row, that is, group merging is performed.

[0065] After processing this batch of data, the current row number of the first hash table is recorded as m1. If m1 has reached or exceeded the preset TOP row number (i.e., m1 is greater than or equal to top_num), proceed to step 4. If m1 has not yet reached top_num but has reached or exceeded the set capacity of the first hash table (i.e., m1 is greater than or equal to max_rowcnt), proceed to step 6. If neither of the above two conditions is met, return to step 2 and continue to obtain the next batch of data.

[0066] Step 4: Retrieve the next batch of data to be grouped from the child nodes. If successful, proceed to step 5. Otherwise, proceed to step 22.

[0067] Step 5: Based on the first hash table, process each row of the data to be grouped in this batch. For any row of data, if an existing group with the same grouping key can be found in the first hash table, update the set function value of the existing group according to the row of data; if no existing group with the same grouping key can be found, skip processing and discard the row of data directly.

[0068] After processing this batch of data, return to step 4 to retrieve the next batch of data. This step and step 4 form a loop, continuously retrieving data and merging and updating existing groups until all data has been retrieved.

[0069] Step 6: Retrieve the next batch of data to be grouped from the child nodes. If successful, proceed to step 7. Otherwise, proceed to step 10.

[0070] Step 7: Based on the first hash table and the filter, process the data to be grouped in this batch row by row.

[0071] For any given row of data, first attempt to find an existing group with the same grouping key in the first hash table. If found, update the set function value of the existing group based on that row. If not found, use a filter to determine the grouping key of the row and update the filter's state. If the filter determines it is non-duplicate, the row is assigned to segment A (first overflow data) and the filter is updated; if the filter cannot determine it is non-duplicate, the row is assigned to segment B (second overflow data) and the filter is updated.

[0072] After processing this batch of data, record the current row number of segment A as p1 and the current row number of segment B as p2. If the sum of the row number of segment A and the row number of the first hash table exceeds the preset TOP row number (i.e., the sum of p1 and m1 is greater than top_num), then proceed to step 8. Otherwise, return to step 6 to continue obtaining the next batch of data.

[0073] Step 8: Retrieve the next batch of data to be grouped from the child nodes. If successful, proceed to step 9. Otherwise, proceed to step 10.

[0074] Step 9: Based on the first hash table and the filter, process the data to be grouped in this batch row by row.

[0075] For any row of data, if an existing group with the same grouping key can be found in the first hash table, then the set function value of the existing group is updated based on that row of data. If no existing group can be found, and the filter determines that it is non-duplicate, then the row of data is discarded and the filter is updated. If no existing group can be found, and the filter cannot determine that it is non-duplicate, then the row of data is assigned to segment B and the filter is updated.

[0076] After processing this batch of data, update the data volume p2 for segment B. Then return to step 8 to continue obtaining the next batch of data. This step and step 8 form a loop. Once the total number of potential groups exceeds the TOP requirement, no more data will be added to segment A; only the ability to merge and distribute data to segment B will be retained.

[0077] Step 10: Output all data in the first hash table and record the final row number of the first hash table as m1_end.

[0078] Destroy the first hash table. Initialize the second hash table, setting its capacity `max_rowcnt_level_2` to the smaller of `max_rowcnt_org` and `(top_num minus m1_end)`, i.e., `max_rowcnt_level_2` equals `MIN(max_rowcnt_org, top_num minus m1_end)`. After completion, proceed to step 11.

[0079] Step 11: Retrieve the next batch of data to be grouped from segment B. If successful, proceed to step 12. Otherwise, proceed to step 17 to begin processing segment A data.

[0080] Step 12: Based on the second hash table, process each row of the data to be grouped in this batch. For any row of data, if no existing group with the same grouping key can be found in the second hash table, insert the row as a new group into the second hash table and mark the row as removed; if an existing group with the same grouping key can be found, update the set function value of the existing group according to the row of data, and also mark the row as removed.

[0081] After processing this batch of data, the current row number of the second hash table is recorded as m2. If m2 has reached or exceeded the target remaining row number (top_num minus m1_end), proceed to step 13. If m2 has reached or exceeded the set capacity of the second hash table, max_rowcnt_level_2 (and this capacity is equal to max_rowcnt_org), proceed to step 19 to perform subsequent overflow processing. Otherwise, return to step 11 and continue retrieving data from segment B.

[0082] Step 13: Retrieve the next batch of data to be grouped from segment B. If successful, proceed to step 14. Otherwise, proceed to step 15.

[0083] Step 14: Based on the second hash table, process each row of the data to be grouped in this batch. For any row of data, if an existing group with the same grouping key can be found in the second hash table, update the set function value of the existing group according to the row of data; if no existing group with the same grouping key can be found, discard the row of data directly.

[0084] After processing this batch of data, update the row count m2 of the second hash table. Then return to step 13 to continue retrieving data segment B. This step and step 13 form a loop. When the number of groups in the second hash table reaches the target number of remaining rows, only merge data segment B without adding new rows.

[0085] Step 15: Retrieve the next batch of data to be output from segment A. If successful, proceed to step 16. Otherwise, proceed to step 22.

[0086] Step 16: Based on the second hash table, merge the data in the same group in the second hash table according to the data to be output in segment A of this batch, and process them row by row. For any row of data, if an existing group with the same grouping key can be found in the second hash table, update the set function value of the existing group according to the data of that row, and discard the data of that row without outputting it; if no existing group with the same grouping key can be found, discard the data of that row directly.

[0087] After processing this batch of data, update the row count m2 of the second hash table. Then proceed to step 22. This step corresponds to the finalization of the A segment data after the second hash table is full and the B segment data has been processed.

[0088] Step 17: Retrieve the next batch of data to be output from segment A. If successful, proceed to step 18. Otherwise, proceed to step 22.

[0089] Step 18: Based on the second hash table, merge the same group data in the second hash table according to the data to be output in segment A of this batch, and process them row by row.

[0090] For any row of data, if an existing group with the same grouping key can be found in the second hash table, the set function value of the existing group is updated according to the row of data, and the row of data is discarded and not output again; if no existing group with the same grouping key can be found, the row of data is output directly.

[0091] After processing this batch of data, update the number of rows in the second hash table to m2, and record the cumulative number of output rows in segment A as q1. If the sum of the number of rows in the second hash table m2, the cumulative number of output rows in segment A q1, and the final number of rows in the first hash table m1_end has reached or exceeded the preset TOP number of rows (i.e., m2 + q1 + m1_end is greater than or equal to top_num), then proceed to step 20. Otherwise, return to step 17 to continue retrieving segment A data.

[0092] Step 19: Perform subsequent overflow handling (TOP_AFTER process) until completion. The specific handling method of this process will be detailed later. After the TOP_AFTER process is completed, proceed to step 23.

[0093] Step 20: Retrieve the next batch of data to be grouped from segment A. If successful, proceed to step 21. Otherwise, proceed to step 22.

[0094] Step 21: Based on the second hash table, merge the data in the same group in the second hash table according to the data to be merged in this batch A segment, and process them row by row. For any row of data, if an existing group with the same grouping key can be found in the second hash table, update the set function value of the existing group according to the data of that row, and discard the data of that row without outputting it; if no existing group with the same grouping key can be found, discard the data of that row directly.

[0095] After processing this batch of data, return to step 20 to continue retrieving data from segment A. This step and step 20 form a loop. Once the total number of output rows has reached the TOP requirement, the remaining data in segment A is only merged and not output.

[0096] Step 22: Directly output all data in the first hash table or the second hash table (if it exists), and stop accepting and processing new data. Then proceed to step 23.

[0097] Step 23, hash grouping ends.

[0098] As can be seen, this embodiment sets up an output mechanism for failures in any data acquisition step. This ensures that even in unexpected situations such as premature exhaustion of the data source or abnormal interruption of the data flow, the method can still output the grouped data in the processed first or second hash table completely, avoiding the loss of all previously processed grouped results due to anomalies. Compared to schemes that require all data to be acquired before unified output, this embodiment has stronger robustness and fault tolerance. Simultaneously, this mechanism works in conjunction with the termination conditions of each branch in the main process. When the main process normally meets the TOP row count condition and actively terminates, it also completes result collection by outputting existing hash table data, unifying the result output path for normal and abnormal terminations, simplifying the implementation logic, and reducing code maintenance costs. During database query execution, child nodes may stop returning data prematurely due to data distribution imbalance, node failure, or resource contention. This fallback output mechanism ensures that queries can still return some valid results in such scenarios, rather than failing completely, improving the availability of the query service and user experience.

[0099] The following details a feasible implementation of the subsequent overflow handling (TOP_AFTER process) in step 19. This process is triggered when the second hash table is full but the number of groups has not yet reached the target number of remaining rows. The goal is to make full use of the limited memory space to extract as many groups as possible from the data in segments A and B that meet the TOP row count requirement.

[0100] Please see Figure 3 The overflow handling process includes the following sub-steps.

[0101] Sub-step 1: Retrieve the next batch of data to be grouped from segment B. If successful, proceed to sub-step 2. Otherwise, proceed to sub-step 3.

[0102] Sub-step 2: Based on the second hash table, process each row of the data to be grouped in this batch. For any row of data, if an existing group with the same grouping key can be found in the second hash table, update the set function value of the existing group according to the row of data and mark the row of data as removed; if no existing group with the same grouping key can be found, skip the process.

[0103] After processing this batch of data, record the current data volume of the second hash table as m3. Then return to sub-step 1 to continue retrieving data segment B. This step and sub-step 1 form a loop, traversing data segment B and merging and absorbing existing groups.

[0104] Sub-step 3: Retrieve the next batch of data to be merged from segment A. If successful, proceed to sub-step 4. Otherwise, proceed to sub-step 5.

[0105] Sub-step 4: Based on the second hash table, process each row of the data to be merged in this batch. For any row of data, if an existing group with the same grouping key can be found in the second hash table, update the set function value of the existing group according to the row of data and mark the row of data as removed; if no existing group with the same grouping key can be found, skip the process.

[0106] After processing this batch of data, update the remaining data volume p1 of segment A. Then proceed to sub-step 5.

[0107] Sub-step 5: Check if the sum of the number of rows already output from the first hash table (m1_end) and the current data volume (m3) in the second hash table exceeds the preset TOP number of rows (top_num). If it does, i.e., m1_end plus m3 is greater than top_num, then directly output all data in the second hash table and proceed to sub-step 13 to end. If it does not exceed, proceed to sub-step 6.

[0108] Sub-step 6: Output all data in the second hash table, record the number of rows output by the second hash table in this round, and accumulate this number of rows into the accumulated output row count m4 of the second hash table. Then reinitialize the second hash table (clear it) and reset the traversal progress of data in segments A and B. Data marked as to be removed is considered non-existent. After completion, proceed to sub-step 7.

[0109] Sub-step 7: Obtain the next batch of data to be grouped from segment B. If successful, proceed to sub-step 8. Otherwise, proceed to sub-step 9.

[0110] Sub-step 8: Based on the second hash table, process each row of the data to be grouped in this batch. For any row of data, if it has already been marked as removed, no processing is performed; if it has not been removed, check if there is an existing group with the same grouping key in the second hash table. If it exists, update the set function value of the existing group according to the row of data and mark the row of data as removed. If it does not exist, insert the row of data as a new group into the second hash table and mark it as removed.

[0111] After processing this batch of data, record the data volume of the second hash table as m5. Then return to sub-step 7 to continue retrieving data from segment B. This step, together with sub-step 7, forms a loop to process the data in segment B that was not previously removed. At this point, it is permissible to insert new groups into the second hash table.

[0112] Sub-step 9: Retrieve the next batch of data to be merged from segment A. If successful, proceed to sub-step 10. Otherwise, proceed to sub-step 11.

[0113] Sub-step 10 processes the batch of data to be merged row by row based on the second hash table. For any row, if it has been marked as removed, it is left unprocessed; if it has not been removed, it checks if there is an existing group with the same grouping key in the second hash table. If it exists, the set function value of the existing group is updated according to the row, and the row is marked as removed; if it does not exist, it is skipped. After processing the batch of data, the remaining data volume p1 of segment A is updated. Then, it returns to sub-step 9 to continue obtaining segment A data. This step and sub-step 9 form a loop, merging only the data in segment A that has not been removed, without inserting new groups.

[0114] Sub-step 11 checks whether the sum of the number of output rows m1_end of the first hash table, the current data volume m5 of the second hash table, and the cumulative number of output rows m4 of the second hash table exceeds the preset TOP number of rows top_num. If it does, i.e., m1_end plus m5 plus m4 is greater than top_num, then all data in the second hash table is output directly, and the process proceeds to sub-step 13 to end. If it does not exceed the preset TOP number of rows, the process proceeds to sub-step 12.

[0115] Sub-step 12: Output all data in the second hash table, record the number of output rows in this round, and update the cumulative number of output rows m4. Then reinitialize the second hash table and reset the traversal progress of segments A and B. After completion, return to sub-step 7 to start a new round of loop.

[0116] Sub-step 13, subsequent processing ends.

[0117] In this embodiment, overflow handling can be accomplished using alternative methods, preserving implementation flexibility for the TOP_AFTER process. In practical applications, the scale and distribution characteristics of overflow data vary depending on the specific query scenario, and using a fixed overflow handling strategy may be inefficient under certain data distributions. By allowing alternative handling methods, this method can dynamically select or switch different overflow handling strategies based on factors such as data duplication rate, remaining memory, or query priority. For example, it can employ external sorting and merging, reuse the first hash table for multiple rounds of processing, or use parallel processing based on data sharding, thereby achieving superior processing performance in different scenarios. Furthermore, this open design reduces the method's dependence on specific overflow handling implementation details, allowing the overflow handling module to be independently optimized and replaced when the database system version is upgraded or the hardware environment changes, without modifying the overall grouping framework, thus improving the method's scalability and adaptability.

[0118] It should be noted that the above algorithm is a simplified version based on the premise that segments A and B are not fragmented; that is, no additional fragmentation is performed on the data during the preprocessing stage. As an optional extension, the data can also be fragmented during the preprocessing stage according to the available memory size, allowing each fragment to fit in memory and each fragment to execute the aforementioned grouping processing flow independently. This fragmentation method can further support grouping operations on larger-scale data.

[0119] For example, before grouping, the data to be grouped is preprocessed by sharding based on the available memory size. This ensures that each shard can be fully contained within the available memory, transforming the grouping operation of a large dataset into a grouping operation of multiple small shards. This avoids disk overflow or repeated swapping due to data volume exceeding memory capacity, minimizing the read / write overhead of grouping processing. When each shard independently executes the hash grouping process described in this method, the capacity settings of its first and second hash tables can be dynamically adjusted based on the shard size and the current available memory, rather than depending on the global data size, resulting in finer-grained and more adaptable memory control. Simultaneously, shard preprocessing reduces the false positive rate of filters such as Bloom filters. Because each shard contains only a subset of the global data, the number of grouping keys that the filter needs to record is reduced, leading to higher accuracy within the same filter space. This reduces the amount of data that is incorrectly classified as non-repeating data by the filter and thus reduces the processing burden and memory pressure of the subsequent second hash table.

[0120] As can be seen from the above algorithm steps, the row limit of the TOP operation can be pushed down to the hash grouping process. By binding the capacity of the first hash table to the number of TOP rows, using filters to divert overflow data, and using a second hash table with limited capacity to merge overflow data, useless data is discarded as early as possible, memory pressure is effectively controlled, and the grouping processing speed is improved.

[0121] The following describes a data grouping device for a database provided in an embodiment of this application. The data grouping device for a database described below can be referred to in conjunction with other embodiments described herein.

[0122] See Figure 4 As shown in the figure, this application discloses a data grouping device for a database, including: Module 401 is used to acquire data to be grouped in batches; The first grouping module 402 is used to group the rows of data to be grouped using a preset first hash table. If the number of groups in the first hash table is less than the preset TOP row number and less than the capacity of the first hash table, then if there is an existing group with the same key value as the current row in the first hash table, the set function value of the existing group is updated according to the current row; if there is no existing group, the current row is inserted as a new group into the first hash table. The capacity is the minimum value between the preset physical upper limit and the preset TOP row number. If the number of groups is not less than the preset TOP row number, then the set function value is updated when an existing group exists. If no existing group exists, discard the current row; if the number of groups reaches the capacity but does not reach the preset TOP row number, update the set function value if an existing group exists; if no existing group exists, and the current row belongs to a non-duplicate group, and the sum of the current group number and the number of rows in the current first overflow data does not reach the preset TOP row number, then the current row is assigned to the first overflow data; if the current row belongs to a non-duplicate group, and the sum of the current group number and the number of rows in the current first overflow data reaches the preset TOP row number, then discard the current row; if the current row cannot be determined to be a non-duplicate group, then the current row is assigned to the second overflow data. The first output module 403 is used to output the latest first hash table after all the data to be grouped has been processed. The second grouping module 404 is used to group the first overflow data and the second overflow data using a preset second hash table, and finally output the latest second hash table; the capacity of the second hash table is the minimum of the preset physical upper limit and the target remaining number of rows; the target remaining number of rows is the difference between the preset TOP number of rows and the total number of rows in the latest first hash table; The second output module 405 is used to obtain grouping results that meet the preset number of TOP rows based on the latest first hash table and the latest second hash table.

[0123] In one implementation, the first grouping module is used for: If the number of groups reaches the capacity, does not reach the preset TOP row number, and the sum of the current number of groups and the number of rows of the current first overflow data reaches the preset TOP row number, then if there are no existing groups, if the current row belongs to a non-duplicate group, the current row is discarded; if the current row cannot be determined to be a non-duplicate group, the current row is assigned to the second overflow data.

[0124] In one implementation, the first grouping module is used for: Use a composite filter to detect whether the current row belongs to a non-repeating group; Accordingly, the first grouping module is used for: If any sub-filter in the composite filter determines that the grouping key of the current row is not a duplicate, then the current row is determined to belong to a non-duplicate group; otherwise, the current row cannot be determined to be a non-duplicate group.

[0125] In one implementation, the second grouping module is used for: Retrieve data in batches from the second overflow data; The second hash table is used to group the acquired rows of data. If the current number of groups in the second hash table is less than the target number of remaining rows and less than the capacity of the second hash table, then if there is an existing group in the second hash table with the same key value as the current row, the set function value of the group is updated according to the current row and the current row is marked as removed; if there is no existing group, the current row is inserted into the second hash table as a new group and the current row is marked as removed. If the current number of groups reaches the target number of remaining rows, update the set function value of the group if there is an existing group, and discard the current row if there is no existing group. If the current number of groups reaches the capacity of the second hash table but does not reach the target number of remaining rows, then an overflow handling operation is performed; If the second overflow data has been processed and the current number of groups has reached the target number of remaining rows, then data is retrieved batch by batch from the first overflow data. For each retrieved row of data, if there is an existing group with the same key value in the second hash table, the set function value of the group is updated according to the current row and the current row is discarded; if there is no existing group, the current row is discarded. If the first overflow data has been processed, the current second hash table is output as the latest second hash table.

[0126] In one implementation, the second grouping module is used for: If the second overflow data has been processed and the current number of groups has not reached the target number of remaining rows, then data is obtained from the first overflow data in batches, and for each row of data obtained; If an existing group with the same key value exists in the second hash table, update the set function value of the group based on the current row and discard the current row; if it does not exist, output the current row and accumulate the number of output rows of the first overflow data; the number of output rows is added to the second hash table after the first overflow data has been processed. When the sum of the current group number, the output row number, and the total row number reaches the preset TOP row number, for the first overflow data obtained subsequently, if there is an existing group with the same key value, the set function value is updated and the current row is discarded; otherwise, the current row is discarded and no output operation is performed.

[0127] In one implementation, the second grouping module is used for: Iterate through the first and second overflow data; merge rows that can be merged with existing groups in the second hash table, and remove the markers; if the sum of the total number of rows and the current number of groups reaches the preset TOP row number, output the current second hash table as the latest second hash table; otherwise, output the current second hash table, record the cumulative number of output rows of the second hash table, and reinitialize the second hash table; Traverse the unremoved rows in the second overflow data; merge the rows that can be merged with existing groups in the second hash table and mark them as removed; otherwise, insert the current row as a new group into the second hash table and mark it as removed. Iterate through the unremoved rows in the first overflow data. After merging the rows that can be merged with the existing groups in the second hash table, mark them as removed. If the sum of the total number of rows, the current number of groups, and the cumulative number of output rows in the second hash table reaches the preset TOP number of rows, output the current second hash table. Otherwise, output the current second hash table, update the cumulative number of output rows in the second hash table, reinitialize the second hash table, and continue processing the unremoved rows.

[0128] In one implementation, the first output module is used for: When the preset TOP row count is less than or equal to the total number of rows, skip the calculation of the target remaining row count and the grouping processing of the first overflow data and the second overflow data, and obtain the grouping result that meets the preset TOP row count based on the latest first hash table.

[0129] For more detailed information on the working process of each module and unit in this embodiment, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0130] As can be seen, this embodiment provides a data grouping device for a database, which enables grouping operations to be performed synchronously with TOP operations, thereby reducing the memory resource consumption of grouping and TOP operations.

[0131] The following describes an electronic device provided by an embodiment of this application. The electronic device described below can be referred to in conjunction with other embodiments described herein.

[0132] See Figure 5 As shown in the figure, an embodiment of this application discloses an electronic device, including: Memory 501 is used to store computer programs; Processor 502 is configured to execute the computer program to implement the method disclosed in any of the above embodiments.

[0133] Furthermore, embodiments of this application also provide an electronic device. The aforementioned electronic device can be, for example,... Figure 6 The server shown can also be as follows: Figure 7 The terminal shown. Figure 6 and Figure 7 These are all diagrams illustrating the structure of an electronic device according to an exemplary embodiment. The content in the diagrams should not be considered as any limitation on the scope of this application.

[0134] Figure 6This is a schematic diagram of a server structure provided in an embodiment of this application. The server may specifically include: at least one processor, at least one memory, a power supply, a communication interface, an input / output interface, and a communication bus. The memory stores a computer program, which is loaded and executed by the processor to implement relevant steps in the data grouping of the database disclosed in any of the foregoing embodiments.

[0135] In this embodiment, the power supply is used to provide operating voltage for each hardware device on the server; the communication interface can create a data transmission channel between the server and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0136] In addition, the memory, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system, computer programs and data, etc., and the storage method can be temporary storage or permanent storage.

[0137] The operating system manages and controls the various hardware devices and computer programs on the server to enable the processor to perform operations and processes on the data in the memory. It can be Windows Server, Netware, Unix, Linux, etc. The computer programs, in addition to those capable of performing the data grouping method for the database disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data may include application update information and application developer information.

[0138] Figure 7 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal may include, but is not limited to, a smartphone, tablet computer, laptop computer, or desktop computer.

[0139] Typically, the terminal in this embodiment includes a processor and a memory.

[0140] The processor may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor can be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor may also include a main processor and coprocessors. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which handles computational operations related to machine learning.

[0141] The memory may include one or more computer non-volatile storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory is used to store at least the following computer program, which, after being loaded and executed by the processor, is capable of implementing the relevant steps in the database data grouping method executed on the terminal side as disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory may also include operating systems and data, and the storage method may be temporary or permanent storage. The operating system may include Windows, Unix, Linux, etc. The data may include, but is not limited to, application update information.

[0142] In some embodiments, the terminal may further include a display screen, an input / output interface, a communication interface, a sensor, a power supply, and a communication bus.

[0143] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the terminal and may include more or fewer components than illustrated.

[0144] The following describes a non-volatile storage medium provided in an embodiment of this application. The non-volatile storage medium described below can be referred to in conjunction with other embodiments described herein.

[0145] A non-volatile storage medium is provided for storing a computer program, wherein the computer program, when executed by a processor, implements the database data grouping method disclosed in the foregoing embodiments. The non-volatile storage medium is a computer-readable non-volatile storage medium, which, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon include an operating system, computer programs, and data, and the storage method can be temporary storage or permanent storage.

[0146] The following describes a computer program product provided by an embodiment of this application. The computer program product described below can be referred to in conjunction with other embodiments described herein.

[0147] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned disclosed database data grouping method.

[0148] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the steps in any of the above embodiments.

[0149] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0150] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of non-volatile storage medium known in the art.

[0151] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data grouping method for a database, characterized in that, include: Retrieve data to be grouped in batches; The data to be grouped is grouped using a preset first hash table. If the number of groups in the first hash table is less than the preset number of top rows and less than the capacity of the first hash table, then if there is an existing group in the first hash table with the same key value as the current row, the set function value of the existing group is updated according to the current row; if there is no existing group, the current row is inserted as a new group into the first hash table; wherein, the capacity is the minimum value between the preset physical upper limit and the preset number of top rows. If the number of groups is not less than the preset number of TOP rows, then the set function value is updated when the existing group exists, and the current row is discarded when the existing group does not exist. If the number of groups reaches the capacity but does not reach the preset TOP row number, then the set function value is updated if an existing group exists; if no existing group exists, and if the current row belongs to a non-repeating group and the sum of the current group number and the number of rows in the current first overflow data does not reach the preset TOP row number, then the current row is assigned to the first overflow data; if the current row belongs to a non-repeating group and the sum of the current group number and the number of rows in the current first overflow data reaches the preset TOP row number, then the current row is discarded; if the current row cannot be determined to be a non-repeating group, then the current row is assigned to the second overflow data. Once all data to be grouped has been processed, output the latest first hash table; The first overflow data and the second overflow data are grouped and processed using a preset second hash table, and finally the latest second hash table is output; the capacity of the second hash table is the minimum value between the preset physical upper limit and the target remaining number of rows; the target remaining number of rows is the difference between the preset TOP number of rows and the total number of rows in the latest first hash table; Based on the latest first hash table and the latest second hash table, grouping results that meet the preset TOP row number are obtained.

2. The method according to claim 1, characterized in that, Also includes: If the number of groups reaches the capacity, does not reach the preset TOP row number, and the sum of the current group number and the current first overflow data row number reaches the preset TOP row number, then if there is no existing group, if the current row belongs to a non-duplicate group, the current row is discarded; if the current row cannot be determined to be a non-duplicate group, the current row is assigned to the second overflow data.

3. The method according to claim 1, characterized in that, Use a composite filter to detect whether the current row belongs to a non-repeating group; Accordingly, a composite filter is used to detect whether the current row belongs to a non-repeating group, including: If any sub-filter in the composite filter determines that the grouping key of the current row is not repeated, then the current row is determined to belong to a non-repeating group; otherwise, it is impossible to determine whether the current row is a non-repeating group.

4. The method according to claim 1, characterized in that, The step of grouping the first overflow data and the second overflow data using a preset second hash table, and finally outputting the latest second hash table, includes: Data is retrieved in batches from the second overflow data; The second hash table is used to group the acquired rows of data. If the current number of groups in the second hash table is less than the target number of remaining rows and less than the capacity of the second hash table, then if there is an existing group in the second hash table with the same key value as the current row, the set function value of the group is updated according to the current row and the current row is marked as removed; if there is no existing group, the current row is inserted into the second hash table as a new group and the current row is marked as removed. If the current number of groups reaches the target number of remaining rows, then if there is an existing group, update the set function value of that group; otherwise, discard the current row. If the current number of groups reaches the capacity of the second hash table but does not reach the target number of remaining rows, then an overflow handling operation is performed; If the second overflow data is processed and the current group number reaches the target remaining row number, then data is obtained batch by batch from the first overflow data. For each obtained row of data, if there is an existing group with the same key value in the second hash table, the set function value of the group is updated according to the current row and the current row is discarded; if there is no existing group, the current row is discarded. If the first overflow data has been processed, the current second hash table is output as the latest second hash table.

5. The method according to claim 4, characterized in that, Also includes: If the second overflow data is processed and the current group number has not reached the target remaining number of rows, then data is obtained from the first overflow data in batches, and for each obtained row of data; If an existing group with the same key value exists in the second hash table, update the set function value of that group based on the current row and discard the current row; If it does not exist, output the current line and accumulate the number of output lines of the first overflow data; wherein, the number of output lines is added to the second hash table after the first overflow data has been processed; When the sum of the current group number, the output row number, and the total row number reaches the preset TOP row number, for the subsequently acquired first overflow data, if there is an existing group with the same key value, the set function value is updated and the current row is discarded; otherwise, the current row is discarded and no output operation is performed.

6. The method according to claim 4, characterized in that, The overflow handling operation includes: Traverse the first overflow data and the second overflow data; after merging the row data that can be merged with the existing group in the second hash table, remove the mark; if the sum of the total number of rows and the current number of groups reaches the preset TOP row number, output the current second hash table as the latest second hash table; otherwise, output the current second hash table, record the cumulative output row number of the second hash table, and reinitialize the second hash table; Traverse the unremoved rows in the second overflow data; merge the rows that can be merged with existing groups in the second hash table and mark them as removed; otherwise, insert the current row as a new group into the second hash table and mark it as removed. Traverse the unremoved rows in the first overflow data, merge the rows that can be merged with the existing groups in the second hash table, and mark them as removed; if the sum of the total number of rows, the current number of groups, and the cumulative number of output rows in the second hash table reaches the preset TOP number of rows, then output the current second hash table; otherwise, output the current second hash table, update the cumulative number of output rows in the second hash table, reinitialize the second hash table, and continue processing the unremoved rows.

7. The method according to any one of claims 1 to 6, characterized in that, Also includes: When the preset TOP row number is less than or equal to the total number of rows, the calculation of the target remaining row number and the grouping processing of the first overflow data and the second overflow data are skipped, and the grouping result that meets the preset TOP row number is obtained based on the latest first hash table.

8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method as described in any one of claims 1 to 7.

9. A non-volatile storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images