Data grouping method, device, equipment, medium and product

Abstract

The invention discloses a data grouping method and device, equipment, a medium and a product. The method comprises the steps that to-be-grouped data are processed according to a preset hash table, so that row data are inserted into the preset hash table, and the preset hash table at least comprises a first hash table; outputting data of the first hash table; for the second hash table, whether data of the second hash table is output or not is determined according to the state of the second hash table, the data of the second hash table is brushed to the corresponding partitions and segments based on the filter according to the set state, the segments are the first segments or the second segments, and the first segments do not comprise repeated data. According to the technical scheme, the scale and overhead of single data hash grouping can be reduced by partitioning the data, duplicated data and non-duplicated data can be distinguished by segmenting the data through the filter, and actual hash detection is reduced; hash grouping is carried out by using double Hash tables, so that the performance influence caused by continuous distribution of repeated data is effectively reduced, and the efficiency and the performance of data grouping are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a data grouping method, apparatus, device, medium, and product. Background Art

[0002] In relational databases, grouping is a common data processing operation. A common implementation of data grouping algorithms is through hash tables. With the advancement of database technology, the amount of data that needs to be processed has increased dramatically. This approach has the following drawbacks: When grouping the entire data set, the data size is large, and the hash processing overhead is high. For example, the hash detection workload, data input and output (I / O), and memory usage are all high, which affects the performance of the hash grouping process. This is especially true in scenarios where duplicate data is continuously distributed, where the performance impact of the hash grouping process is more significant. Summary of the Invention

[0003] The present application provides a data grouping method, apparatus, device, medium and product to improve the efficiency and performance of data grouping.

[0004] In a first aspect, an embodiment of the present application provides a data grouping method, including:

[0005] For each row of data to be grouped, process the row of data according to a preset hash table to insert the row of data into the preset hash table, where the preset hash table is a set of hash tables existing in the memory, each hash table is used to store non-repeated grouped data with a grouping key as the key, and groups in different hash tables are non-repeated, and the preset hash tables include at least a first hash table;

[0006] Outputting the data of the first hash table;

[0007] In a case where the preset hash table also includes a second hash table, whether to output the data of the second hash table is determined according to the state of the second hash table, and, in a case where the second hash table is in a set state, the data of the second hash table is flushed to the corresponding partition and segment based on the filter, wherein the segment is the first segment or the second segment, and the filter is used to filter duplicate data so that the first segment does not include duplicate data.

[0008] In a second aspect, an embodiment of the present application further provides a data grouping device, including:

[0009] a first processing module, configured to process each row of data in the data to be grouped according to a preset hash table to insert the row of data into the preset hash table, wherein the preset hash table is a set of hash tables existing in a memory, each hash table is configured to store non-repeated grouped data using a grouping key as a key, and groups in different hash tables are non-repeated, and the preset hash tables include at least a first hash table;

[0010] An output module, configured to output data of the first hash table;

[0011] A second processing module is configured to determine whether to output data of a second hash table according to a state of the second hash table when the preset hash table also includes the second hash table, and to flush the data of the second hash table to corresponding partitions and segments based on a filter when the second hash table is in a set state, wherein the segment is the first segment or the second segment, and the filter is configured to filter duplicate data so that duplicate data is not included in the first segment.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0013] one or more processors;

[0014] a storage device for storing one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the data grouping method as described in the first aspect.

[0016] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, which, when executed by a processor, implements the data grouping method as described in the first aspect.

[0017] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising a computer program and / or instructions, which, when executed by a processor, implements the data grouping method as described in any of the above embodiments.

[0018] Embodiments of the present application provide a data grouping method, apparatus, device, medium, and product, the data grouping method comprising: for each row of data in the data to be grouped, processing the row of data according to a preset hash table to insert the row of data into the preset hash table, the preset hash table being a set of hash tables existing in the memory, each hash table being used to store non-repeated grouped data with a grouping key as Key, and groups of different hash tables being non-repeated, the preset hash table comprising at least a first hash table; outputting the data of the first hash table; when the preset hash table also includes a second hash table, determining whether to output the data of the second hash table according to the state of the second hash table, and, when the second hash table is in a set state, flushing the data of the second hash table to the corresponding partition and segment based on a filter, wherein the segment is the first segment or the second segment, and the filter being used to filter duplicate data so that duplicate data is not included in the first segment. The above technical solution can reduce the scale and overhead of single data hash grouping by partitioning the data. By using filter segmentation, it can distinguish between duplicate data and non-duplicate data, reducing actual hash detection; using dual hash tables for hash grouping can effectively reduce the performance impact caused by the continuous distribution of duplicate data, and improve the efficiency and performance of data grouping. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A flowchart of a data grouping method provided in an embodiment of the present application;

[0021] Figure 2 A flowchart of the first grouping and partitioning provided in an embodiment of the present application;

[0022] Figure 3 A flowchart of a first-level partition processing stage provided in an embodiment of the present application;

[0023] Figure 4 A flowchart of a secondary partition processing stage provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the structure of a data grouping device provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present application and are not intended to limit the present application. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions of the present application, not all of the structures.

[0027] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0028] It should be noted that the concepts of "first" and "second" mentioned in the embodiments of this application are only used to distinguish different devices, modules, units or other objects, and are not used to limit the order or interdependence of the functions performed by these devices, modules, units or other objects.

[0029] In addition, the embodiments and features in the embodiments of the present application may be combined with each other unless there is any conflict.

[0030] The acquisition, storage, use, and processing of data in this application's technical solution comply with relevant national laws and regulations.

[0031] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the relevant content of the solution.

[0032] Figure 1A flowchart of a data grouping method provided in an embodiment of the present application, which can be applied to the case of processing table data or row data. Specifically, the data grouping method can be performed by a data grouping device, which can be implemented by software and / or hardware and integrated into an electronic device. Electronic devices include but are not limited to computers, smart phones, servers, and other devices with computing functions, and can also be central processing units (CPUs), system-on-chip (SoCs), computers, field-programmable gate arrays (FPGAs), or microcontrollers (MCUs).

[0033] like Figure 1 As shown, the method specifically includes the following steps:

[0034] S110. For each row of data in the data to be grouped, process the row of data according to a preset hash table to insert the row of data into the preset hash table. The preset hash table is a set of hash tables existing in the memory. Each hash table is used to store non-repeated grouped data with a grouping key as the Key, and groups in different hash tables are non-repeated. The preset hash tables include at least a first hash table.

[0035] The preset hash table can be used to group data to be grouped. The main process of using the hash table for grouping includes: checking whether a row of data is mapped to the hash table. If the row of data does not exist in the hash table, the row is inserted into the hash table and a new group is created for it based on the grouping column values. If the row of data exists in the hash table, the row is inserted (merged) into the existing group. Based on this, the hash table can be used to store the data of each group.

[0036] In this embodiment, the preset hash table may include two hash tables: a first hash table (also referred to as a main hash table) and a second hash table (also referred to as a secondary hash table). In the process of batch processing of the grouped data, for the current batch of data to be grouped, if the first hash table can accommodate this batch of data to be grouped, then the first hash table can be used for grouping; if the first hash table cannot accommodate this batch of data to be grouped and the second hash table can accommodate this batch of data to be grouped, then the second hash table needs to be used for grouping; if neither the first hash table nor the second hash table can accommodate this batch of data to be grouped, then for some of the rows therein, it is necessary to flush the disk. It can be understood that the initial state of the preset hash table is to create an empty first hash table. When the first hash table cannot accommodate new data, a second hash table can be created.

[0037] The disk flushing process can be understood as writing these rows of data to the corresponding partitions and segments on the disk based on the filter to achieve data grouping. Each partition can include two segments. Using the filter, a batch of data can be divided into a first segment (also called segment A) and a second segment (also called segment B). In this embodiment, the filter has the following characteristics: it prevents data in segment A from being repeated.

[0038] Exemplarily, the process of segmenting data using filters mainly includes:

[0039] Initialize the filter;

[0040] Traverse the current batch of data line by line. If the filter determines that the data in the current row is definitely not duplicate data, the data in the current row can be divided into segment A. Otherwise, the new data is divided into segment B. Each time a row of data is processed, the filter is updated according to the segmentation result of the current row of data.

[0041] The filter in this embodiment can be understood as a total filter, which can be a composite structure composed of a single or multiple types of sub-filters. The types of sub-filters include but are not limited to maximum value filters, minimum value filters, Bloom filters, etc.

[0042] When the overall filter determines data duplication, it can combine the results of the subfilters to make a comprehensive determination. For example, if at least one subfilter determines that a row of data is definitely not duplicate data, the overall filter can determine that the row of data is definitely not duplicate data. Otherwise, since all subfilters cannot determine that the row of data is definitely not duplicate data, the overall filter cannot determine that the row of data is definitely not duplicate data either.

[0043] Exemplarily, the mechanism by which a subfilter determines whether a row of data is duplicate data may be determined by the filter characteristics. For example, for a maximum value subfilter, if the value of a row of data is greater than the maximum value recorded by the subfilter, the subfilter determines that the row of data is definitely not duplicate data. Otherwise, the subfilter cannot determine that the row of data is definitely not duplicate data.

[0044] It's understood that updating the master filter requires updating all subfilters simultaneously. The method for updating subfilters can be determined by the filter's characteristics. For example, for a maximum subfilter, if the value of a row of data exceeds the maximum value recorded in the subfilter, the maximum value recorded in the subfilter will be updated to the value of that row of data; otherwise, the subfilter will not be updated.

[0045] S120: Output the data of the first hash table.

[0046] The data of each group in the first hash table can be directly output.

[0047] S130. When the preset hash table also includes a second hash table, determine whether to output the data of the second hash table according to the state of the second hash table, and when the second hash table is in a set state, flush the data of the second hash table to the corresponding partition and segment based on the filter, wherein the segment is the first segment or the second segment, and the filter is used to filter duplicate data so that the first segment does not include duplicate data.

[0048] Among them, whether the data of each group in the second hash table can be output needs to be determined according to the state of the second hash table. The state of the second hash table includes, for example, whether the second hash table is initialized and / or whether the disk flushing has been triggered. In addition, when the second hash table meets the set state (such as reaching the capacity limit, triggering the disk flushing and / or there is no data to be processed, etc.), the data in the second hash table will be flushed to the corresponding partitions and segments. In this case, it is also necessary to flush the data of the second hash table to the corresponding partitions and segments based on the filter, and further grouping and partitioning can be performed on the data in each partition. It can be understood that any data that enters the second hash table will be flushed to the first-level partition afterwards; any data will first enter the second hash table before being flushed to the partition; the second hash table can be regarded as a buffer before the data is flushed to the disk. As long as the second hash table is full or there is no new data to be processed, the data in the second hash table can be flushed to the disk in sections and segments at one time.

[0049] The embodiment of the present application provides an improved hash grouping algorithm based on data partitioning and filters, which aims to reduce the resource consumption of the hash grouping algorithm and improve the performance of the hash grouping algorithm. Specifically, a double hash table is used for hash grouping. The introduction of the main hash table can make a part of the non-repeated data groups resident in the memory. The introduction of the secondary hash table can achieve deduplication for some data that needs to be flushed to the disk, improve the performance of data grouping, and effectively reduce the performance impact caused by the continuous distribution of duplicate data; based on the application of the filter, the data to be flushed to the disk can be divided into two segments, ensuring that the data in the first segment is non-repeated, making its processing logic simpler and more efficient; based on data partition processing, the scale and overhead of a single data hash grouping can be reduced; the actual hash detection is reduced, and the overall performance is improved; and, when memory space is prioritized, this method can also reduce the processing volume of data I / O and reduce resource consumption in this scenario.

[0050] In one embodiment, for each row of data in the data to be grouped, processing the row of data according to a preset hash table to insert the row of data into the preset hash table includes:

[0051] When the first hash table can accommodate a batch of data to be grouped, for each row of data in the batch of data to be grouped: if there is no group in the first hash table with the same grouping column value as that of the row of data, then the row of data is inserted into the first hash table as a new group; if there is a group in the first hash table with the same grouping column value as that of the row of data, then the row of data is merged into the group with the same grouping column value as that of the row of data.

[0052] Specifically, the grouped data is processed in batches. For the current batch of data to be grouped, if the first hash table can accommodate this batch of data to be grouped, the first hash table can be used for grouping. The current batch of data to be grouped is processed row by row. For any row, a hash algorithm is used to calculate its grouping column to obtain the grouping column value. Then, depending on whether the column value already exists in the first hash table, the row can be inserted into the first hash table as a new group, or it can be merged into the existing group in the first hash table. On this basis, if the capacity of the first hash table is sufficient, a preset hash table can be used to achieve data grouping.

[0053] In one embodiment, when the first hash table cannot accommodate a batch of data to be grouped, the preset hash table further includes a second hash table;

[0054] The method further comprises:

[0055] Initialize the second hash table;

[0056] For each row of data in the data to be grouped, the row of data is processed according to the preset hash table to insert the row of data into the preset hash table, including:

[0057] For each row of data in the batch of data to be grouped: if there is a group with the same grouping column value as the row of data in the first hash table, then merge the row of data into the group with the same grouping column value as the row of data; if there is no group with the same grouping column value as the row of data in the first hash table, and there is a group with the same grouping column value as the row of data in the second hash table, then merge the row of data into the group with the same grouping column value as the row of data; if there is no group with the same grouping column value as the row of data in both the first hash table and the second hash table, then insert the row of data into the second hash table as a new group.

[0058] Specifically, the grouped data is processed in batches. For the current batch of data to be grouped, if the first hash table cannot accommodate this batch of data to be grouped (i.e., the number of groups accommodated by the first hash table has reached the upper limit), the first hash table hash group can be converted to a second hash table hash group, and the second hash table can be initialized. The current batch of data to be grouped is processed row by row. For any row, a hash algorithm is used to calculate the column value. If the column value already exists in the first or second hash table, it means that the row of data belongs to an existing group. The row can be merged into the group in the first or second hash table. Merging into an existing group can be understood as updating the set function value of the existing group. If the column value does not exist in either the first or second hash table, it means that the row of data does not belong to an existing group. It is then inserted into the second hash table as a new group. The method of determining whether a row of data belongs to an existing group is mainly based on whether the group key (i.e., the column value of the group column, also known as the group key) conflicts with the existing groups in the first and second hash tables. On this basis, two hash tables can be used to implement data grouping if the capacity is sufficient.

[0059] In one embodiment, the preset hash table further includes a second hash table;

[0060] When the second hash table is in a set state, flushing the data of the second hash table to corresponding partitions and segments based on the filter includes:

[0061] When the second hash table cannot accommodate a batch of data to be grouped and the second hash table is in a set state, for each row of data in the second hash table, the row of data is flushed to the corresponding partition and segment based on the filter.

[0062] Specifically, the grouped data is processed in batches. For the current batch of data to be grouped, if neither the first hash table nor the second hash table can accommodate this batch of data, it is necessary to process this batch of data line by line to flush the disk, write this batch of data to different partitions and segments, and use filters to ensure that the data in segment A of each partition is not repeated.

[0063] It should be noted that the partitions here are the first partitions, and each partition can be called a first-level partition. In the subsequent processing process, the data in each first-level partition may be re-flushed to different partitions in multiple levels. For example, the data in each first-level partition may be written to different second-level partitions during a disk flushing operation.

[0064] In one embodiment, the setting state is that the second hash table triggers a disk flush;

[0065] Determining whether to output data from the second hash table according to a state of the second hash table, and, when the second hash table is in a set state, flushing the data from the second hash table to corresponding partitions and segments based on a filter, including:

[0066] If the state of the second hash table is uninitialized, the data of the second hash table is not output;

[0067] If the state of the second hash table is that the disk flush has not been triggered, output the data of the second hash table;

[0068] If the status of the second hash table is that the disk has been triggered, the data in the second hash table is flushed to the corresponding first-level partitions and segments based on the filter, and the data of each first-level partition is partitioned according to the third hash table.

[0069] Specifically, after using the preset hash table to group the data to be grouped (and possibly flushing the data that cannot be accommodated by both hash tables), it can be determined how to process the data in the second hash table according to the state of the second hash table. If the state of the second hash table is uninitialized, it means that the first hash table can accommodate all the data to be grouped and the second hash table is not used, then only the data of the first hash table needs to be output, and there is no need to output the data of the second hash table as the result of data grouping; if the state of the second hash table is initialized but the flushing has not been triggered, it means that the first hash table and the second hash table can accommodate all the data to be grouped and there is no data that has been flushed, then it is necessary to output the data of the first hash table and the second hash table as the result of data grouping; if the state of the second hash table is triggered to flush the disk, it means that the first hash table and the second hash table cannot accommodate all the data to be grouped, then it is necessary to flush the data of the second hash table and write it to different partitions and segments, and use filters to ensure that the data in segment A of each partition is not repeated. It should be noted that each partition here can be called a first-level partition. During subsequent processing, the data in each first-level partition may be re-flushed to different partitions according to the multi-level partitions. For example, the data in each first-level partition may be written to different second-level partitions during a flushing operation.

[0070] In one embodiment, grouping and partitioning the data of each first-level partition according to the third hash table includes:

[0071] For any first-level partition, if the second segment of the first-level partition has data, then for each row of data in the second segment of the first-level partition, the row of data is inserted into or merged into the third hash table, or flushed to the corresponding second-level partition and segment.

[0072] Specifically, for the second segment of data in each first-level partition, since there may be duplicate data in the second segment of data, a new hash table (referred to as the third hash table) can be initialized to further process it row by row for grouping. For example, for the current row of data, it may be inserted into or merged into the third hash table as a new group, or it may be flushed to disk and written to a more specific partition (i.e., the second-level partition).

[0073] In one embodiment, for each row of data in the second segment of data in the first-level partition, inserting the row of data into or merging it into the third hash table, or flushing it to the corresponding second-level partition and segment, includes:

[0074] In the case that the third hash table can accommodate a batch of second-segment data of the first-level partition, for each row of data in the batch of second-segment data: if there is a group in the third hash table with the same grouping column value as that of the row of data, then the row of data is merged into the group with the same grouping column value as that of the row of data; if there is no group in the third hash table with the same grouping column value as that of the row of data, then the row of data is inserted into the third hash table as a new group.

[0075] Specifically, the data in each first-level partition is processed row by row. The processing of this batch of segment B data is determined based on whether the third hash table can accommodate the current batch of segment B data (in the first-level partition) and whether the segment B data of the first-level partition has previously triggered a hash table flush. If the third hash table can accommodate the batch of second segment data of the first-level partition, then for any row of data in the batch of second segment data: if the same group cannot be found in the third hash table, it will be inserted into the third hash table; if the same group can be found in the third hash table, it will be merged into that group. On this basis, the grouping of data in the first-level partition is achieved.

[0076] In one embodiment, for each row of data in the second segment of data in the first-level partition, inserting the row of data into or merging it into the third hash table, or flushing it to the corresponding second-level partition and segment, includes:

[0077] When the third hash table cannot accommodate a batch of second segment data of the first-level partition, for each row of data in the batch of second segment data, the row of data is flushed to the corresponding second-level partition and segment based on the filter.

[0078] Specifically, the data in each first-level partition is processed row by row. If the third hash table cannot accommodate the batch of second-segment data of the first-level partition, then for any row of data in the batch of second-segment data, it can be flushed to the corresponding second-level segment and segment based on the filter, wherein the first segment (segment A) data in each second-level partition is not repeated.

[0079] In one embodiment, the data of each first-level partition is grouped and partitioned according to the third hash table, including: if there is no data in the second segment of the first-level partition, or the processing of the second segment of the first-level partition has been completed, then for each row of data in the first segment of the first-level partition, according to the status of the second segment of the first-level partition and the third hash table, the row of data is output, or the row of data is merged into the third hash table, or the row of data is flushed to the corresponding second-level partition and segment.

[0080] Specifically, if there is no data in the second segment of a first-level partition, or the second segment has been processed, the corresponding grouping result for the first segment of data can be determined based on the status of the second segment of data and whether there is data in the third hash table.

[0081] In one embodiment, based on the status of the second segment of data in the first-level partition and the third hash table, outputting the row of data, merging the row of data into the third hash table, or flushing the row of data to the corresponding second-level partition and segment includes:

[0082] If the status of the second segment of data in the first-level partition is that the disk flush has not been triggered and there is no data in the third hash table, output the row of data;

[0083] If the status of the second segment of data in the first-level partition is that no disk flush has been triggered, and there is data in the third hash table:

[0084] If there is a group in the third hash table with the same grouping column value as the row of data, then merge the row of data into the group with the same grouping column value as the row of data;

[0085] If there is no group in the third hash table with the same grouping column value as the row of data, then output the row of data;

[0086] When the status of the second segment of data in the first-level partition is that it has triggered a disk flush, based on the third hash table and filter, the row of data is output, or the row of data is merged into the third hash table, or the row of data is flushed to the corresponding second-level partition and segment.

[0087] Specifically, for a first-level partition, the first segment of data is processed row by row, and the corresponding grouping result can be determined according to the status of the second segment of data of the first-level partition and whether there is data in the third hash table. If the second segment of data of the first-level partition has not triggered a disk flush and there is no data in the third hash table, it means that the second segment of the first-level partition has no data, and the first segment of data of the current row can be directly output; if the second segment of data of the first-level partition has not triggered a disk flush and there is data in the third hash table, it means that the second segment of data of the first-level partition has been grouped in the third hash table, and the first segment of data of the current row can be inserted into the third hash table as a new group or merged into the same group in the third hash table; if the second segment of data of the first-level partition has triggered a disk flush, it means that the third hash table cannot accommodate all the second segment of data of the first-level partition, and then it is necessary to determine how to process the first segment of data of the row, which may be to output it, to merge it into the third hash table, or to flush it to the second-level partition and segment.

[0088] In one embodiment, grouping and partitioning the data of each first-level partition according to the third hash table further includes:

[0089] After processing the first segment of data in the first-level partition, output the data in the third hash table;

[0090] If the second segment of data in the first-level partition has triggered the flushing of the third hash table, the filter is reset and the data in each second-level partition is processed.

[0091] Specifically, after the first segment of data in a first-level partition is processed, the third hash table can be output, and if the second segment of data in the partition triggers a disk flush (has been flushed to the second-level partition), the filter is reset and the data in the second-level partition is further grouped and partitioned.

[0092] In one embodiment, the data in each secondary partition is processed, including:

[0093] For a secondary partition:

[0094] If the second segment of data in the secondary partition triggers a disk flush, the data in the fourth hash table is flushed to the disk in order and the fourth hash table is cleared;

[0095] If the second segment of data in the secondary partition has not triggered a disk flush, then for a batch of second segment data in the secondary partition:

[0096] In a case where the fourth hash table can accommodate the batch of second segment data, merging each row of data in the batch of second segment data into or inserting it into the fourth hash table;

[0097] When the fourth hash table cannot accommodate the batch of second segment data, the data in the fourth hash table is flushed to disk in an orderly manner.

[0098] In one embodiment, processing the data in each secondary partition includes:

[0099] For the first segment of a batch of secondary partitions:

[0100] If the second segment of data in the secondary partition has not triggered a disk flush and there is no data in the fourth hash table, the first segment of data in the batch is output;

[0101] If the second segment of data in the secondary partition has not triggered a disk flush, and there is data in the fourth hash table, each row of data in the first segment of data is merged into the fourth hash table or output according to the fourth hash table;

[0102] If the second segment of data in the secondary partition triggers a disk flush, and the fourth hash table can accommodate the first segment of data, each row of data in the first segment of data is merged or inserted into the fourth hash table according to the fourth hash table;

[0103] If the second segment of data in the secondary partition triggers a disk flush, and the fourth hash table cannot accommodate the first segment of data, the data in the fourth hash table will be flushed in order, and each row of data in the first segment of data will be merged or inserted into the fourth hash table according to the fourth hash table.

[0104] In one embodiment, processing the data in each secondary partition includes:

[0105] If there is no data in the first segment of the secondary partition, or if the second segment of data in the secondary partition has triggered a disk flush and the fourth hash table has been flushed and cleared in an orderly manner, then if the second segment of data in the corresponding primary partition has triggered a disk flush in the corresponding hash table, the ordered segments generated by the orderly disk flush of the data in the secondary partition are grouped and output in an orderly manner;

[0106] When the second segment of data of the corresponding first-level partition has not triggered a disk refresh, the data of the fourth hash table is output.

[0107] In one embodiment, the process of orderly flushing a hash table includes: determining the flushing order based on the order of the hash table, and the flushing order satisfies: data between different hash slots are ordered according to the hash slot sequence number, data in the same hash slot are ordered according to the hash value, and data with the same hash value in the same hash slot are ordered according to the size relationship of the grouping column value; writing the data in the hash table to the disk according to the flushing order.

[0108] On this basis, the algorithm can be expanded to multiple layers and different levels of algorithm implementation. By performing partitioning at different levels, multi-level data grouping can be achieved, which can further divide the data volume.

[0109] Exemplarily, for the case of N-level partitioning, the process of data grouping and partitioning can be divided into three stages: 1) Double hash table deduplication, that is, using the preset hash table to group and deduplicate data to obtain multiple non-repeating data groups; 2) Intermediate-level partitioning, which can be understood as a nested combination of several completely isomorphic first-level partitioning, second-level partitioning,...N-1-level partitioning processing processes. The partitioning process from each level to the next level is isomorphic to the above-mentioned first partitioning process. The main purpose of the disk flushing process is to perform the next level of partitioning; 3) Last-level partitioning, the main purpose of the disk flushing process is to merge and group, and no longer perform the next level of partitioning, which is isomorphic to the above-mentioned processing process for the second-level partition.

[0110] The data grouping process is described below in an exemplary manner.

[0111] First, the preset hash tables (the first hash table and the second hash table, also called the main hash table and the secondary hash table) are used for grouping and initial partitioning, such as Figure 2 As shown, this process includes the following steps:

[0112] Step 1) The hash grouping operation begins. Initialize an empty main hash table. Go to step 2).

[0113] Step 2) Get a batch of data to be grouped. If there is no data to be grouped, go to step 8), otherwise go to step 3).

[0114] Step 3) Determine whether the secondary hash table has been initialized. If so, go to step 5), otherwise go to step 4).

[0115] Step 4) determines whether the main hash table can accommodate a new batch of data to be grouped. If yes, go to step 6), otherwise go to step 5).

[0116] Step 5) Initialize an empty secondary hash table, and determine how to process the data to be grouped based on whether the secondary hash table can accommodate a new batch of data to be grouped.

[0117] If the secondary hash table can accommodate a new batch of data to be grouped, go to step 7).

[0118] If the secondary hash table cannot accommodate a new batch of data to be grouped, and the secondary hash table has triggered a disk flush, the secondary hash table data is flushed in sections and segments, and the process goes to step 7).

[0119] If the secondary hash table cannot accommodate a new batch of data to be grouped and the secondary hash table has not triggered a disk flush, a filter is initialized for the secondary hash table and the secondary hash table data is flushed in sections and segments, and then the process goes to step 7).

[0120] Step 6) Process the grouped data row by row based on the main hash table. For any row of data: if the same group cannot be found in the main hash table, insert it into the main hash table; if the same group can be found in the hash table, merge the groups. Go to step 2).

[0121] Step 7) Process the grouped data row by row based on the primary hash table and the secondary hash table. For any row of data:

[0122] If the same group can be found in the main hash table, the groups are merged;

[0123] If the same group cannot be found in the primary hash table, continue to look for the same group in the secondary hash table;

[0124] If the same group still cannot be found in the secondary hash table, insert it into the secondary hash table;

[0125] If the same groups are found in the secondary hash table, the groups are merged and the process goes to step 2).

[0126] Step 8) After all the grouped data are retrieved, how to process the data is determined based on whether the secondary hash table is initialized and whether a disk flush has been triggered.

[0127] If the secondary hash table is not initialized, output all data in the primary hash table and go to step 10).

[0128] If the secondary hash table has not triggered a disk flush, all data in the primary hash table and the secondary hash table are output, and the process goes to step 10).

[0129] If the secondary hash table triggers a disk flush, all data in the primary hash table is output, and all data in the secondary hash table is flushed in sections and segments, and then go to step 9).

[0130] Step 9) Enter the first-level partition processing stage.

[0131] Step 10) The hash grouping operation ends.

[0132] It should be noted that, in the above process, the capacity of the primary hash table and the secondary hash table should be larger than the data volume of a single batch of data to be grouped.

[0133] In the above process, if the values ​​of the grouping columns of two rows of data are the same, the two rows of data are said to belong to the same group.

[0134] In the above process, the process of merging groups can be understood as: using new data rows belonging to the same group to update the set function value of the old data rows in the hash table, and then discarding the new data rows.

[0135] In the above process, the process of flushing the partitions and segments of the secondary hash table is as follows:

[0136] i) Get a batch of data to be flushed from the secondary hash table and go to step ii).

[0137] ii) Based on the filter, the data to be flushed is processed row by row. For any row of data, the specified partitioning algorithm is used to determine its partition and the partition number is set. If the filter determines that the row does not exist, it is inserted into the filter and marked as segment A data; otherwise, it is marked as segment B data. Go to step iii).

[0138] iii) Based on the partition sequence number mark and segment mark set for each row of data, the data to be flushed to disk is flushed to different first-level partitions and segments A or B of the first-level partitions. Go to step i).

[0139] iiii) The partition and segment flushing is completed.

[0140] Then, for the first-level partition processing stage, such as Figure 3 As shown, the following steps are included:

[0141] Step 1) The first-level partition processing phase begins. Initialize an empty third hash table. Go to step 2).

[0142] Step 2) Take the next non-empty first-level partition. If there are no non-empty first-level partitions to be processed, go to step 10), otherwise go to step 3).

[0143] Step 3) If there is data in segment B of the first-level partition, go to step 4), otherwise go to step 6).

[0144] Step 4) Read a batch of data from the disk in the B segment of the first-level partition. If the data has been read, go to step 6). Otherwise, go to step 5).

[0145] Step 5) Determine how to process the new batch of segment B data based on whether the third hash table can accommodate the new batch of segment B data and whether the segment B data of the current first-level partition has previously triggered the flushing of the third hash table.

[0146] If it can fit, the new batch of segment B data is processed row by row based on the third hash table. For any row of data: if the same group cannot be found in the third hash table, it is inserted into the third hash table; if the same group can be found in the third hash table, the groups are merged. Go to step 4).

[0147] If it cannot be accommodated and the data of segment B in the current first-level partition has not triggered a disk flush, a filter is initialized first, and then a new batch of segment B data is partitioned and flushed based on the third hash table. Then go to step 4).

[0148] If it cannot be accommodated and the data of segment B of the current first-level partition has triggered a disk flush, a new batch of segment B data is partitioned based on the third hash table and the filter. Then go to step 4).

[0149] Step 6) If there is data in the first-level partition A, go to step 7), otherwise go to step 9).

[0150] Step 7) Read a batch of data of the first-level partition A segment from the disk. If the data has been read, go to step 9), otherwise go to step 8).

[0151] Step 8) Determine how to process segment A data based on whether segment B data in the first-level partition has previously triggered a disk flush and whether there is data in the third hash table.

[0152] If the data in segment B of the first-level partition has not triggered a disk flush before, and there is no data in the third hash table, the data in segment A of the current batch is directly output. Go to step 7).

[0153] If the first-level partition's segment B data has not previously triggered a disk flush, and the third hash table is not empty, the current batch of segment A data is processed row by row based on the third hash table. For any row: if the same group is found in the third hash table, the group is merged; if the same group cannot be found in the third hash table, it is directly output. Go to step 7).

[0154] If the first-level partition has previously triggered a disk flush for the segment B data, the current batch of segment A data is partitioned based on the third hash table and the filter. Go to step 7).

[0155] Step 9) Determine how to continue processing the loaded segment B data based on whether the segment B of the first-level partition has triggered the third hash table flush.

[0156] If the third hash table flush has already been triggered for this first-level partition, all data in the third hash table is output, the filter is reset, and the second-level partition processing phase for this first-level partition is entered. When the second-level partition processing phase ends, the first-level partition processing is complete. Then go to step 2).

[0157] If the third hash table flush has not been triggered in the first-level partition B, all data in the third hash table is output. The first-level partition processing is completed. Go to step 2).

[0158] Step 10) The hash grouping operation ends.

[0159] It should be noted that the partition processing process involved in step 5) of the first-level partition processing stage is as follows:

[0160] i) Based on the third hash table and the filter, process the partitioned data row by row. For any row of data: if the third hash table contains identical groups, merge the groups. If the third hash table does not contain identical groups, determine the partition to which it belongs using the specified partitioning algorithm (different from the hash function used for the initial partitioning and segmentation), assign a partition number, and insert the data into the filter. Go to step ii).

[0161] ii) Based on the partition serial number tag set for each row of data, flush the data to be partitioned to the B segments of each secondary partition to which the primary partition belongs. Go to step iii).

[0162] iii) Partition flashing is completed.

[0163] The partition processing flow in step 8) of the first-level partition processing stage is as follows:

[0164] i) Based on the third hash table and the filter, the partitioned data is processed row by row. For any row of data: if the third hash table contains identical groups, a flag is set to indicate that the groups need to be merged. If the third hash table does not contain identical groups, the filter is tested again. If the filter determines that the data does not exist, a flag is set to indicate that the data is directly output. If the filter cannot determine that the data does not exist, the partition to which it belongs is determined according to the specified partitioning algorithm, and the partition number flag is set. Then, the process proceeds to step ii).

[0165] ii) Based on the flags set for each row of data (need to merge groups / directly output) and the partition number flag, the data to be partitioned and the corresponding groups are grouped and merged, or directly output, or flushed to the disk to the A segments of the second-level partitions to which the first-level partition belongs. Go to step iii).

[0166] iii) Partition flashing is completed.

[0167] Finally, for the secondary partition processing process, such as Figure 4 Shown, including:

[0168] Step 1) The secondary partition processing phase begins. Initialize an empty fourth hash table. Go to step 2).

[0169] Step 2) Take the next non-empty secondary partition. If there are no non-empty secondary partitions to be processed, go to step 10), otherwise go to step 3).

[0170] Step 3) If there is data in segment B of the secondary partition, go to step 4), otherwise go to step 6).

[0171] Step 4) Read a batch of data of the B segment of the secondary partition from the disk.

[0172] If the data has been read out and the data of segment B of the secondary partition has previously triggered a disk flush, the fourth hash table data is flushed to the disk in an orderly manner and the fourth hash table is cleared. Go to step 6).

[0173] If the data has been read and the data in segment B of this secondary partition has never been refreshed, go to step 6).

[0174] If the data has not been read completely, go to step 5).

[0175] Step 5) Determine how to process the new batch of segment B data based on whether the fourth hash table can accommodate the new batch of segment B data.

[0176] If it can fit, the new batch of segment B data is processed row by row based on the fourth hash table. For any row of data: if the same group cannot be found in the fourth hash table, it is inserted into the fourth hash table; if the same group can be found in the fourth hash table, the groups are merged. Go to step 4).

[0177] If the data in the fourth hash table cannot be accommodated, the data in the fourth hash table is flushed to the disk in an orderly manner. Go to step 5).

[0178] Step 6) If there is data in the second-level partition A, go to step 7), otherwise go to step 9).

[0179] Step 7) Read a batch of data of the A segment of the secondary partition from the disk.

[0180] If the data has been read and the data in segment B of the secondary partition has triggered a disk flush, the fourth hash table data is flushed in order and the fourth hash table is cleared. Go to step 9).

[0181] If the data has been read and the data in segment B of this secondary partition has never triggered a disk refresh before, go to step 9).

[0182] If the data has not been read completely, go to step 8).

[0183] Step 8) Determine how to process a new batch of segment A data based on the status of the segment B data being flushed to disk and the status of the fourth hash table.

[0184] If the data of segment B in the current secondary partition has not triggered a disk flush before, and there is no data in the fourth hash table, the data of segment A in this batch is directly output. Go to step 7).

[0185] If the data in segment B of this secondary partition has not previously triggered a disk flush, and the fourth hash table is not empty, the current batch of segment A data is processed row by row based on the fourth hash table. For any row: if the same group is found in the fourth hash table, the group is merged; if the same group cannot be found in the fourth hash table, it is directly output. Go to step 7).

[0186] If the data in segment B of this secondary partition has previously triggered a disk flush, and the fourth hash table can accommodate the new batch of data, the new batch of segment A data is processed row by row based on the fourth hash table. For any row of data: if the same group cannot be found in the fourth hash table, it is inserted into the fourth hash table; if the same group can be found in the fourth hash table, the groups are merged. Go to step 7).

[0187] If the data in segment B of this secondary partition has already triggered a disk flush, and the fourth hash table cannot accommodate the new batch of data, the fourth hash table data is first flushed in order, and then the new batch of segment A data is processed row by row based on the fourth hash table. For any row of data: if the same group cannot be found in the fourth hash table, it is inserted into the fourth hash table; if the same group can be found in the fourth hash table, the groups are merged. Go to step 7).

[0188] Step 9) Determine how to continue processing the loaded segment B data based on whether the segment B of the first-level partition has triggered the third hash table flush.

[0189] If the third hash table flush has already been triggered in the first-level partition B, all ordered segments generated by the ordered flush in the second-level partition are grouped and merged in order, and the result is directly output. The processing of the second-level partition is completed. Go to step 2).

[0190] If the third hash table flush has not been triggered in the first-level partition B, all data in the fourth hash table is output. The second-level partition processing is completed. Go to step 2).

[0191] Step 10) The secondary partition processing phase ends.

[0192] It should be noted that the orderly disk flushing process involved in steps 4), 7), and 8) of the secondary partition processing stage is as follows:

[0193] i) Based on the order of the fourth hash table, determine the data to be flushed from the fourth hash table. The flushing order has the following characteristics: data between different hash slots is ordered by hash slot number, data within a hash slot is ordered by hash value, and data within a hash slot with the same hash value is ordered based on the size of the grouped column data. Go to step ii).

[0194] ii) Perform an orderly disk flush based on the determined disk flush data, and write an independent orderly data segment on the disk. The orderly disk flush is completed.

[0195] Figure 5 This is a schematic diagram of the structure of a data grouping device provided in an embodiment of the present application. The data grouping device provided in this embodiment includes:

[0196] A first processing module 210 is configured to process each row of data in the data to be grouped according to a preset hash table to insert the row of data into the preset hash table, wherein the preset hash table is a set of hash tables existing in the memory, each hash table is configured to store non-repeated grouped data using a grouping key as a key, and groups in different hash tables are non-repeated, and the preset hash tables include at least a first hash table;

[0197] An output module 220, configured to output the data of the first hash table;

[0198] The second processing module 230 is used to determine whether to output the data of the second hash table according to the state of the second hash table when the preset hash table also includes the second hash table, and, when the second hash table is in a set state, flush the data of the second hash table to the corresponding partition and segment based on the filter, wherein the segment is the first segment or the second segment, and the filter is used to filter duplicate data so that the first segment does not include duplicate data.

[0199] By partitioning data, the device can reduce the scale and overhead of single data hash grouping. By using filter segmentation, it can distinguish duplicate data from non-duplicate data, reducing actual hash detection. By using dual hash tables for hash grouping, it effectively reduces the performance impact caused by the continuous distribution of duplicate data and improves the efficiency and performance of data grouping.

[0200] Based on any of the above embodiments, the first processing module 210 is specifically configured to:

[0201] When the first hash table can accommodate a batch of data to be grouped, for each row of data in the batch of data to be grouped:

[0202] If there is no group in the first hash table with the same grouping column value as the row of data, insert the row of data into the first hash table as a new group;

[0203] If there is a group in the first hash table whose column value is consistent with the grouping column value of the row of data, the row of data is merged into the group whose column value is consistent with the grouping column value of the row of data.

[0204] Based on any of the above embodiments, when the first hash table cannot accommodate a batch of data to be grouped, the preset hash table further includes a second hash table;

[0205] The device further comprises:

[0206] An initialization module, used for initializing the second hash table;

[0207] The first processing module 210 is specifically configured to: for each row of data in the batch of data to be grouped:

[0208] If there is a group in the first hash table with the same grouping column value as the row of data, then merge the row of data into the group with the same grouping column value as the row of data;

[0209] If there is no group with the same grouping column value as the row data in the first hash table, and there is a group with the same grouping column value as the row data in the second hash table, merge the row data into the group with the same grouping column value as the row data;

[0210] If there is no group with the same grouping column value as the row data in either the first hash table or the second hash table, the row data is inserted into the second hash table as a new group.

[0211] Based on any of the above embodiments, the preset hash table further includes a second hash table;

[0212] The first processing module 210 is specifically configured to:

[0213] When the second hash table cannot accommodate a batch of data to be grouped and the second hash table is in a set state, for each row of data in the second hash table, the row of data is flushed to the corresponding first-level partition and segment based on the filter.

[0214] Based on any of the above embodiments, the set state is that the second hash table triggers a disk refresh;

[0215] The second processing module 230 includes: an output unit, configured to: if the state of the second hash table is uninitialized, not output the data of the second hash table; if the state of the second hash table is that the disk has not been triggered, output the data of the second hash table;

[0216] a flushing unit, configured to flush the data of the second hash table to corresponding first-level partitions and segments based on a filter if the state of the second hash table is that the flushing has been triggered;

[0217] The partitioning unit is used to group and partition the data of each first-level partition according to the third hash table.

[0218] Based on any of the above embodiments, the partitioning unit is specifically used to: for any first-level partition, if the second segment of the first-level partition has data, then for each row of data in the second segment of the first-level partition, the row of data is inserted into or merged into the third hash table, or flushed to the corresponding second-level partition and segment.

[0219] Based on any of the above embodiments, for each row of data in the second segment of data of the first-level partition, inserting the row of data into or merging it into the third hash table, or flushing it to the corresponding second-level partition and segment, includes:

[0220] When the third hash table can accommodate a batch of second segment data of the first-level partition, for each row of data in the batch of second segment data:

[0221] If a group with the same grouping column value as the row of data exists in the third hash table, merging the row of data into the group with the same grouping column value as the row of data;

[0222] If there is no group in the third hash table that has the same column value as the grouping column of the row of data, the row of data is inserted into the third hash table as a new group.

[0223] Based on any of the above embodiments, for each row of data in the second segment of data of the first-level partition, inserting the row of data into or merging it into the third hash table, or flushing it to the corresponding second-level partition and segment, includes:

[0224] When the third hash table cannot accommodate a batch of second segment data of the first-level partition, for each row of data in the batch of second segment data, the row of data is flushed to the corresponding second-level partition and segment based on the filter.

[0225] Based on any of the above embodiments, the partition unit is specifically configured to:

[0226] If there is no data in the second segment of the first-level partition, or the processing of the second segment of the data of the first-level partition has been completed, then for each row of data in the first segment of the data of the first-level partition, according to the status of the second segment of the data of the first-level partition and the third hash table, the row of data is output, or the row of data is merged into the third hash table, or the row of data is flushed to the corresponding second-level partition and segment.

[0227] Based on any of the above embodiments, according to the status of the second segment of data of the first-level partition and the third hash table, outputting the row of data, or merging the row of data into the third hash table, or flushing the row of data to the corresponding second-level partition and segment, includes:

[0228] When the state of the second segment of data in the first-level partition is that disk flushing has not been triggered and there is no data in the third hash table, output the row of data;

[0229] When the state of the second segment of data in the first-level partition is that no disk flushing has been triggered, and there is data in the third hash table:

[0230] If a group with the same grouping column value as the row of data exists in the third hash table, the row of data is merged into the group with the same grouping column value as the row of data;

[0231] If there is no group in the third hash table with the same grouping column value as the row of data, output the row of data;

[0232] When the status of the second segment of data in the first-level partition is that it has triggered a disk flush, based on the third hash table and filter, the row of data is output, or the row of data is merged into the third hash table, or the row of data is flushed to the corresponding second-level partition and segment.

[0233] Based on any of the above embodiments, the partitioning unit is further configured to:

[0234] After processing the first segment of data of the first-level partition, outputting the data of the third hash table;

[0235] If the second segment of data in the first-level partition has triggered the flushing of the third hash table, the filter is reset and the data in each second-level partition is processed.

[0236] Based on any of the above embodiments, processing the data in each secondary partition includes:

[0237] For a secondary partition:

[0238] If the second segment of data in the secondary partition triggers a disk flush, the data in the fourth hash table is flushed to the disk in order, and the fourth hash table is cleared;

[0239] If the second segment of data in the secondary partition has not triggered a disk flush, then for a batch of second segment data in the secondary partition:

[0240] In a case where the fourth hash table can accommodate the batch of second segment data, merging or inserting each row of data in the batch of second segment data into the fourth hash table;

[0241] When the fourth hash table cannot accommodate the batch of second segment data, the data in the fourth hash table is flushed to disk in an orderly manner.

[0242] Based on any of the above embodiments, processing the data in each secondary partition includes:

[0243] For the first batch of data in the secondary partition:

[0244] If the second segment of data in the secondary partition has not triggered a disk refresh, and there is no data in the fourth hash table, output the first segment of data in the batch;

[0245] If the second segment of data in the secondary partition has not triggered a disk refresh, and the fourth hash table has data, then each row of data in the first segment of data is merged into the fourth hash table or output according to the fourth hash table;

[0246] If the second segment of data in the secondary partition triggers a disk flush, and the fourth hash table can accommodate the first segment of data, each row of data in the first segment of data is merged or inserted into the fourth hash table according to the fourth hash table;

[0247] If the second segment of data in the secondary partition triggers a disk flush, and the fourth hash table cannot accommodate the first segment of data, the data in the fourth hash table will be flushed in order, and each row of data in the first segment of data will be merged into or inserted into the fourth hash table according to the fourth hash table.

[0248] Based on any of the above embodiments, processing the data in each secondary partition includes:

[0249] If the first segment of the secondary partition has no data, or the second segment of data of the secondary partition has triggered a disk flush and the fourth hash table has been flushed and cleared in an orderly manner, then if the second segment of data of the corresponding primary partition has triggered a disk flush of the corresponding hash table, the ordered segments generated by the orderly disk flush of the data of the secondary partition are grouped in an orderly manner and output;

[0250] When the second segment of data of the corresponding first-level partition has not triggered a disk flush, the data of the fourth hash table is output.

[0251] Based on any of the above embodiments, the process of orderly flushing a hash table includes:

[0252] The flushing order is determined based on the order of the hash table. The flushing order satisfies the following conditions: data between different hash slots are ordered according to the hash slot sequence number, data within the same hash slot are ordered according to the hash value, and data with the same hash value within the same hash slot are ordered according to the size of the grouping column value;

[0253] The data in the hash table is written to the disk according to the disk flushing order.

[0254] The data grouping device provided in the embodiments of the present application can be used to execute the data grouping method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0255] Figure 6A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 10 can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, user equipment, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided for example only and are not intended to limit the implementation of the present application as described and / or claimed herein.

[0256] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0257] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks and wireless networks.

[0258] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above.

[0259] In some embodiments, the methods of the above embodiments may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform any of the above-described methods in any suitable manner (e.g., by means of firmware).

[0260] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0261] Computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0262] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0263] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device 10 having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device 10. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0264] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0265] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0266] An embodiment of the present application further provides a computer program product, comprising a computer program and / or instructions, which, when executed by a processor, implements the data grouping method as described in any of the above embodiments.

[0267] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of this application can be achieved. This is not limited herein.

[0268] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included within the scope of protection of this application.

Claims

1. A data grouping method, characterized in that: include: For each row of data to be grouped, process the row of data according to a preset hash table to insert the row of data into the preset hash table, where the preset hash table is a set of hash tables existing in the memory, each hash table is used to store non-repeated grouped data with a grouping key as the key, and groups in different hash tables are non-repeated, and the preset hash tables include at least a first hash table; Outputting the data of the first hash table; In a case where the preset hash table also includes a second hash table, whether to output the data of the second hash table is determined according to the state of the second hash table, and, in a case where the second hash table is in a set state, the data of the second hash table is flushed to the corresponding partition and segment based on the filter, wherein the segment is the first segment or the second segment, and the filter is used to filter duplicate data so that the first segment does not include duplicate data.

2. The method according to claim 1, characterized in that The processing of each row of data in the data to be grouped according to a preset hash table to insert the row of data into the preset hash table includes: When the first hash table can accommodate a batch of data to be grouped, for each row of data in the batch of data to be grouped: If there is no group in the first hash table with the same grouping column value as the row of data, insert the row of data into the first hash table as a new group; If there is a group in the first hash table whose column value is consistent with the grouping column value of the row of data, the row of data is merged into the group whose column value is consistent with the grouping column value of the row of data.

3. The method according to claim 1, characterized in that In the case that the first hash table cannot accommodate a batch of data to be grouped, the preset hash table further includes a second hash table; The method further includes: initializing a second hash table; The processing of each row of data in the data to be grouped according to a preset hash table to insert the row of data into the preset hash table includes: For each row of data in the batch of data to be grouped: If there is a group in the first hash table with the same grouping column value as the row of data, then merge the row of data into the group with the same grouping column value as the row of data; If there is no group with the same grouping column value as the row data in the first hash table, and there is a group with the same grouping column value as the row data in the second hash table, merge the row data into the group with the same grouping column value as the row data; If there is no group with the same grouping column value as the row of data in either the first hash table or the second hash table, the row of data is inserted into the second hash table as a new group.

4. The method according to claim 1, wherein The preset hash table also includes a second hash table; When the second hash table is in a set state, flushing the data of the second hash table to corresponding partitions and segments based on the filter includes: When the second hash table cannot accommodate a batch of data to be grouped and the second hash table is in a set state, for each row of data in the second hash table, the row of data is flushed to the corresponding partition and segment based on the filter.

5. The method according to claim 4, characterized in that The setting state is that the second hash table triggers a disk flush; The determining whether to output the data of the second hash table according to the state of the second hash table, and, when the second hash table is in a set state, flushing the data of the second hash table to the corresponding partition and segment based on the filter, includes: If the state of the second hash table is uninitialized, the data of the second hash table is not output; If the state of the second hash table is that the disk flush has not been triggered, output the data of the second hash table; If the state of the second hash table is that the disk has been triggered, the data of the second hash table is flushed to the corresponding first-level partitions and segments based on the filter, and the data of each first-level partition is grouped and partitioned according to the third hash table.

6. The method according to claim 5, characterized in that The data of each first-level partition is grouped and partitioned according to the third hash table, including: For any first-level partition, if the second segment of the first-level partition has data, then for each row of data in the second segment of the first-level partition, the row of data is inserted into or merged into the third hash table, or flushed to the corresponding second-level partition and segment.

7. The method according to claim 6, characterized in that For each row of data in the second segment of data of the first-level partition, inserting the row of data into or merging it into the third hash table, or flushing it to the corresponding second-level partition and segment, includes: When the third hash table can accommodate a batch of second segment data of the first-level partition, for each row of data in the batch of second segment data: If a group with the same grouping column value as the row of data exists in the third hash table, merging the row of data into the group with the same grouping column value as the row of data; If there is no group in the third hash table that has the same column value as the grouping column of the row of data, the row of data is inserted into the third hash table as a new group.

8. The method according to claim 6, characterized in that For each row of data in the second segment of data of the first-level partition, inserting the row of data into or merging it into the third hash table, or flushing it to the corresponding second-level partition and segment, includes: When the third hash table cannot accommodate a batch of second segment data of the first-level partition, for each row of data in the batch of second segment data, the row of data is flushed to the corresponding second-level partition and segment based on the filter.

9. The method according to claim 5, characterized in that The data of each first-level partition is grouped and partitioned according to the third hash table, including: If there is no data in the second segment of the first-level partition, or the processing of the second segment of the data of the first-level partition has been completed, then for each row of data in the first segment of the data of the first-level partition, according to the status of the second segment of the data of the first-level partition and the third hash table, the row of data is output, or the row of data is merged into the third hash table, or the row of data is flushed to the corresponding second-level partition and segment.

10. The method according to claim 9, characterized in that According to the state of the second segment of data of the first-level partition and the third hash table, outputting the row of data, or merging the row of data into the third hash table, or flushing the row of data to the corresponding second-level partition and segment, including: When the state of the second segment of data in the first-level partition is that disk flushing has not been triggered and there is no data in the third hash table, output the row of data; When the state of the second segment of data in the first-level partition is that no disk flushing has been triggered, and there is data in the third hash table: If a group with the same grouping column value as the row of data exists in the third hash table, the row of data is merged into the group with the same grouping column value as the row of data; If there is no group in the third hash table with the same grouping column value as the row of data, output the row of data; When the status of the second segment of data in the first-level partition is that it has triggered a disk flush, based on the third hash table and filter, the row of data is output, or the row of data is merged into the third hash table, or the row of data is flushed to the corresponding second-level partition and segment.

11. The method according to claim 9, characterized in that Grouping and partitioning the data of each first-level partition according to the third hash table further includes: After processing the first segment of data of the first-level partition, outputting the data of the third hash table; If the second segment of data in the first-level partition has triggered the flushing of the third hash table, the filter is reset and the data in each second-level partition is processed.

12. The method according to claim 11, characterized in that Process the data in each secondary partition, including: For a secondary partition: If the second segment of data in the secondary partition triggers a disk flush, the data in the fourth hash table is flushed to the disk in order, and the fourth hash table is cleared; If the second segment of data in the secondary partition has not triggered a disk flush, then for a batch of second segment data in the secondary partition: In a case where the fourth hash table can accommodate the batch of second segment data, merging or inserting each row of data in the batch of second segment data into the fourth hash table; When the fourth hash table cannot accommodate the batch of second segment data, the data in the fourth hash table is flushed to disk in an orderly manner.

13. The method according to claim 12, characterized in that Process the data in each secondary partition, including: For the first batch of data in the secondary partition: If the second segment of data in the secondary partition has not triggered a disk refresh, and there is no data in the fourth hash table, output the first segment of data in the batch; If the second segment of data in the secondary partition has not triggered a disk refresh, and the fourth hash table has data, then each row of data in the first segment of data is merged into the fourth hash table or output according to the fourth hash table; If the second segment of data in the secondary partition triggers a disk flush, and the fourth hash table can accommodate the first segment of data, each row of data in the first segment of data is merged or inserted into the fourth hash table according to the fourth hash table; If the second segment of data in the secondary partition triggers a disk flush, and the fourth hash table cannot accommodate the first segment of data, the data in the fourth hash table will be flushed in order, and each row of data in the first segment of data will be merged into or inserted into the fourth hash table according to the fourth hash table.

14. The method according to claim 12, characterized in that Process the data in each secondary partition, including: If the first segment of the secondary partition has no data, or the second segment of data of the secondary partition has triggered a disk flush and the fourth hash table has been flushed and cleared in an orderly manner, then if the second segment of data of the corresponding primary partition has triggered a disk flush of the corresponding hash table, the ordered segments generated by the orderly disk flush of the data of the secondary partition are grouped in an orderly manner and output; When the second segment of data of the corresponding first-level partition has not triggered a disk flush, the data of the fourth hash table is output.

15. The method according to any one of claims 12 to 14, characterized in that The process of flushing a hash table in an orderly manner includes: The flushing order is determined based on the order of the hash table. The flushing order satisfies the following conditions: data between different hash slots are ordered according to the hash slot sequence number, data within the same hash slot are ordered according to the hash value, and data with the same hash value within the same hash slot are ordered according to the size of the grouping column value; The data in the hash table is written to the disk according to the disk flushing order.

16. A data grouping device, characterized in that: include: a first processing module, configured to process each row of data in the data to be grouped according to a preset hash table to insert the row of data into the preset hash table, wherein the preset hash table is a set of hash tables existing in a memory, each hash table is configured to store non-repeated grouped data using a grouping key as a key, and groups in different hash tables are non-repeated, and the preset hash tables include at least a first hash table; an output module, configured to output data of the first hash table; A second processing module is configured to determine whether to output data of a second hash table according to a state of the second hash table when the preset hash table also includes the second hash table, and to flush the data of the second hash table to corresponding partitions and segments based on a filter when the second hash table is in a set state, wherein the segment is the first segment or the second segment, and the filter is configured to filter duplicate data so that duplicate data is not included in the first segment.

17. An electronic device, characterized in that: include: at least one processor; a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, wherein the computer program is executed by the at least one processor so as to enable the at least one processor to perform the data grouping method according to any one of claims 1 to 15.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data grouping method according to any one of claims 1 to 15 is implemented.

19. A computer program product comprising a computer program and / or instructions, characterized in that When the computer program and / or the instructions are executed by a processor, the data grouping method according to any one of claims 1 to 15 is implemented.

Images