Data processing method and apparatus

Abstract

The one or more embodiments of the specification provide a data processing method and device, including determining a data aggregation rate based on sampling data in a grouping pushdown process, generating a high-frequency value hash table based on the sampling data in a case where the data aggregation rate does not meet an aggregation effect, performing pre-aggregation processing on high-frequency data in to-be-processed data based on the high-frequency value hash table, and distributing a pre-aggregation processing result to a target thread of an aggregation node. In the embodiments of the specification, in a case where the data aggregation effect is detected to be poor in a sampling stage, a high-frequency value hash table is obtained based on hash values of high-frequency data in the data statistics in the sampling stage, so that only the high-frequency data in the high-frequency value hash table is subjected to pre-aggregation processing in a processing stage, the data skew problem of an upper aggregation node is eliminated or alleviated, the aggregation overhead and the data amount of data distribution are effectively balanced, and the data query performance is improved.

Description

Technical Field

[0001] This specification relates to one or more embodiments in the field of terminal technology, and in particular to a data processing method and apparatus. Background Technology

[0002] GPD (Group By Pushdown) is an optimization technique for databases to perform aggregation calculations in parallel scenarios. It refers to pushing the GBY (Group By) operator down to the local machine. Before data transmission, the data is pre-aggregated locally using the GBY operator. Then, the pre-aggregated data is distributed to different worker threads to complete the final data aggregation.

[0003] GPD offers excellent scalability and can reduce data distribution costs, but its data processing performance suffers when the aggregation rate of the GBY pressure operator is low. Therefore, adaptive GBY technology has been introduced, whereby when the GBY pressure operator detects poor aggregation, it directly sends data to the upper-layer node without performing aggregation. However, this can lead to data skew, slowing down the entire data query process. Summary of the Invention

[0004] To eliminate or mitigate the data skew problem that occurs when the aggregation effect is poor during the grouping pushdown process, this specification provides a data processing method, apparatus, electronic device, and storage medium.

[0005] Firstly, one or more embodiments of this specification provide a data processing method, including:

[0006] The data aggregation rate is determined based on the sampled data during the grouping pushdown process;

[0007] If the data aggregation rate does not meet the aggregation effect, a high-frequency value hash table is generated based on the sampled data. The high-frequency value hash table includes the hash values ​​of high-frequency data in the sampled data.

[0008] Based on the high-frequency value hash table, high-frequency data in the data to be processed is pre-aggregated, and the pre-aggregation results are distributed to the target thread of the aggregation node.

[0009] In one or more embodiments of this specification, determining the data aggregation rate based on sampled data during the grouping pushdown process includes:

[0010] The data aggregation rate is determined based on the first data volume of the sampled data during the grouping pushdown process and the second data volume after the sampled data has undergone pre-aggregation processing.

[0011] In one or more embodiments of this specification, determining the data aggregation rate based on sampled data during the grouping pushdown process includes:

[0012] The sampled data is periodically acquired with a first preset data volume, and the process of determining the data aggregation rate based on the sampled data is executed until the data aggregation rate of the current period does not meet the aggregation effect.

[0013] In one or more embodiments of this specification, generating a high-frequency hash table based on the sampled data includes:

[0014] In the sampled data with a total data volume of T, the corresponding hash value is calculated based on the grouping key of each data, and the first data volume with the same hash value as the i-th data is counted, where 1≤i≤T;

[0015] Based on the total data volume, the first data volume, and the number of threads pushing down in groups, determine the second data volume of other data that have different hash values ​​from the i-th data and distribute them evenly to each thread;

[0016] Based on the ratio of the first data volume to the second data volume, determine whether the i-th data is high-frequency data;

[0017] The high-frequency value hash table is generated based on all high-frequency data and their hash values ​​in the sampled data.

[0018] In one or more embodiments of this specification, determining whether the i-th data is high-frequency data based on the ratio of the first data volume to the second data volume includes:

[0019] In response to the ratio being greater than or equal to a preset threshold, the i-th data is determined to be high-frequency data;

[0020] In response to the proportion being less than the preset threshold, the i-th data is determined to be low-frequency data.

[0021] In one or more embodiments of this specification, the step of pre-aggregating high-frequency data in the data to be processed based on the high-frequency value hash table, and distributing the pre-aggregation results to the target thread of the aggregation node, includes:

[0022] For each piece of data in the data to be processed, the corresponding hash value is calculated based on the grouping key of the data;

[0023] The hash value of the data is matched with the hash value in the high-frequency value hash table. If the match is consistent, the data is determined to be high-frequency data.

[0024] The high-frequency data is pre-aggregated to obtain the pre-aggregation result, and the target thread is determined based on the grouping key of the high-frequency data;

[0025] The pre-aggregation processing results are distributed to the target thread.

[0026] In one or more embodiments of this specification, the method further includes:

[0027] The data to be processed is periodically acquired with a second preset data volume, and the frequency value parameter of each high-frequency data in the high-frequency value hash table is determined based on the data to be processed, until the frequency value parameter of all high-frequency data in the high-frequency value hash table does not meet the set requirements. The frequency value parameter represents the frequency of occurrence of the high-frequency data in the data to be processed.

[0028] In response to the presence of at least one high-frequency data whose frequency value parameter meets the set requirements in the high-frequency value hash table, the process of performing pre-aggregation processing on the high-frequency data in the data to be processed based on the high-frequency value hash table and distributing the pre-aggregation processing result to the target thread of the aggregation node is executed.

[0029] If the frequency value parameters of all high-frequency data in the high-frequency value hash table do not meet the set requirements, the data to be processed is directly distributed to the aggregation node.

[0030] Secondly, one or more embodiments of this specification provide a data processing apparatus, including:

[0031] The aggregation rate determination module is configured to determine the data aggregation rate based on the sampled data during the grouping pushdown process;

[0032] The hash table generation module is configured to generate a high-frequency value hash table based on the sampled data when the data aggregation rate does not meet the aggregation effect. The high-frequency value hash table includes the hash values ​​of high-frequency data in the sampled data.

[0033] The pre-aggregation module is configured to perform pre-aggregation processing on high-frequency data in the data to be processed based on the high-frequency value hash table, and distribute the pre-aggregation processing results to the target thread of the aggregation node.

[0034] In one or more embodiments of this specification, the polymerization rate determination module is configured to:

[0035] The data aggregation rate is determined based on the first data volume of the sampled data during the grouping pushdown process and the second data volume after the sampled data has undergone pre-aggregation processing.

[0036] In one or more embodiments of this specification, the polymerization rate determination module is configured to:

[0037] The sampled data is periodically acquired with a first preset data volume, and the process of determining the data aggregation rate based on the sampled data is executed until the data aggregation rate of the current period does not meet the aggregation effect.

[0038] In one or more embodiments of this specification, the hash table generation module is configured to:

[0039] In the sampled data with a total data volume of T, the corresponding hash value is calculated based on the grouping key of each data, and the first data volume with the same hash value as the i-th data is counted, where 1≤i≤T;

[0040] Based on the total data volume, the first data volume, and the number of threads pushing down in groups, determine the second data volume of other data that have different hash values ​​from the i-th data and distribute them evenly to each thread;

[0041] Based on the ratio of the first data volume to the second data volume, determine whether the i-th data is high-frequency data;

[0042] The high-frequency value hash table is generated based on all high-frequency data and their hash values ​​in the sampled data.

[0043] In one or more embodiments of this specification, the hash table generation module is configured to:

[0044] In response to the ratio being greater than or equal to a preset threshold, the i-th data is determined to be high-frequency data;

[0045] In response to the proportion being less than the preset threshold, the i-th data is determined to be low-frequency data.

[0046] In one or more embodiments of this specification, the pre-aggregation module is configured to:

[0047] For each piece of data in the data to be processed, the corresponding hash value is calculated based on the grouping key of the data;

[0048] The hash value of the data is matched with the hash value in the high-frequency value hash table. If the match is consistent, the data is determined to be high-frequency data.

[0049] The high-frequency data is pre-aggregated to obtain the pre-aggregation result, and the target thread is determined based on the grouping key of the high-frequency data;

[0050] The pre-aggregation processing results are distributed to the target thread.

[0051] In one or more embodiments of this specification, the pre-aggregation module is configured to:

[0052] The data to be processed is periodically acquired with a second preset data volume, and the frequency value parameter of each high-frequency data in the high-frequency value hash table is determined based on the data to be processed, until the frequency value parameter of all high-frequency data in the high-frequency value hash table does not meet the set requirements. The frequency value parameter represents the frequency of occurrence of the high-frequency data in the data to be processed.

[0053] In response to the presence of at least one high-frequency data whose frequency value parameter meets the set requirements in the high-frequency value hash table, the process of performing pre-aggregation processing on the high-frequency data in the data to be processed based on the high-frequency value hash table and distributing the pre-aggregation processing result to the target thread of the aggregation node is executed.

[0054] If the frequency value parameters of all high-frequency data in the high-frequency value hash table do not meet the set requirements, the data to be processed is directly distributed to the aggregation node.

[0055] Thirdly, one or more embodiments of this specification provide an electronic device, including:

[0056] processor; and

[0057] A memory storing computer instructions for causing a processor to perform the method according to any embodiment of the first aspect.

[0058] Fourthly, one or more embodiments of this specification provide a storage medium storing computer instructions for causing a computer to perform the method described according to any embodiment of the first aspect.

[0059] This specification describes a data processing method in one or more embodiments, including determining a data aggregation rate based on sampled data during the grouping pushdown process. If the data aggregation rate does not meet the aggregation effect requirements, a high-frequency value hash table is generated based on the sampled data. Pre-aggregation processing is then performed on high-frequency data in the data to be processed based on the high-frequency value hash table, and the pre-aggregation processing results are distributed to the target thread of the aggregation node. In this embodiment, if poor data aggregation is detected during the sampling phase, a high-frequency value hash table is obtained by statistically analyzing the hash values ​​of high-frequency data from the sampling phase. Therefore, during the processing phase, pre-aggregation processing is performed only on high-frequency data in the high-frequency value hash table, eliminating or mitigating the data skew problem of the upper-level aggregation node, effectively balancing aggregation overhead and the amount of data distributed, and improving data query performance. Attached Figure Description

[0060] Figure 1 This is a schematic diagram illustrating the principle of a data processing method according to an exemplary embodiment of this specification.

[0061] Figure 2This is a schematic diagram illustrating the principle of a data processing method according to an exemplary embodiment of this specification.

[0062] Figure 3 This is a flowchart of a data processing method according to an exemplary embodiment of this specification.

[0063] Figure 4 This is a schematic diagram illustrating the principle of a data processing method according to an exemplary embodiment of this specification.

[0064] Figure 5 This is a flowchart of a data processing method according to an exemplary embodiment of this specification.

[0065] Figure 6 This is a flowchart of a data processing method according to an exemplary embodiment of this specification.

[0066] Figure 7 This is a flowchart of a data processing method according to an exemplary embodiment of this specification.

[0067] Figure 8 This is a structural block diagram of a data processing apparatus according to an exemplary embodiment of this specification.

[0068] Figure 9 This is a structural block diagram of an electronic device according to an exemplary embodiment of this specification. Detailed Implementation

[0069] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0070] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0071] In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0072] GPD (Group By Pushdown) is an optimization technique for computational aggregation in parallel scenarios of distributed databases. It refers to pushing the GBY (Group By) operator down to the local machine. Before data transmission, the data is pre-aggregated locally using the GBY operator. Then, the pre-aggregated data is distributed to different worker threads to complete the final data aggregation.

[0073] For example Figure 1 As shown, node 1 is the lower-level node, and nodes 2 and 3 are the upper-level nodes. In GPD technology, the GBY operator can be pushed down to the lower-level node 1. Therefore, before data is transmitted from node 1 to the upper-level nodes 2 and 3, it can be pre-aggregated in node 1 using the pushed-down GBY operator (one-stage GBY). Pre-aggregation refers to grouping the data according to the grouping key (GBY key), performing aggregation operations on each group, and then sending the pre-aggregation results to each thread of the upper-level node according to the grouping key. Each thread of the upper-level node receives data from the lower-level node according to the GBY key; that is, the data processed by each thread of the upper-level node has the same GBY key value. Each thread of the upper-level node performs the final aggregation processing on the data to obtain the corresponding data query results.

[0074] GPD offers excellent scalability and can effectively reduce data distribution costs. However, it adds an extra cost for GBY pressure operator data pre-aggregation, especially when the data aggregation rate of lower-level nodes is low, resulting in poor data processing performance. For example, when lower-level nodes perform data pre-aggregation, they aggregate data based on the same grouping key. If there is very little data with the same grouping key, it not only increases the overhead of data pre-aggregation but also results in a low data aggregation rate, leading to a decline in data processing performance.

[0075] In related technologies, some databases have introduced adaptive GBY technology to solve this problem. That is, during the grouping pushdown process, if the lower-level node detects that the data aggregation effect is not good, it directly distributes the data to the upper-level node and no longer performs the pre-aggregation operation, thereby avoiding the extra overhead of performing the pre-aggregation operation.

[0076] However, combined Figure 1As shown, if a lower-level node directly distributes data to an upper-level node, and if there is a high-frequency data in the lower-level node's data (i.e., a lot of data with the same grouping key), the high-frequency data will be directly distributed to a certain thread of the upper-level node because the pre-aggregation operation is no longer performed. This results in a large amount of data in that thread and a smaller amount of data in other threads, creating a data skew phenomenon, which slows down the entire data query process.

[0077] Based on the deficiencies of the aforementioned related technologies, this specification provides a data processing method, apparatus, electronic device, and storage medium. The aim is to detect when the data aggregation effect is poor during the grouping pushdown process, and to aggregate high-frequency data in subsequent data based on a high-frequency value hash table statistically obtained during the sampling phase, while no longer aggregating low-frequency data. This balances aggregation overhead and data volume, eliminating or mitigating data skew caused by high-frequency data.

[0078] The system architecture of the embodiments described in this specification can be found in [reference]. Figure 1 As shown, the data processing method described in this specification is mainly applied to the lower-level nodes, namely the first-stage GBY nodes. For ease of explanation, the lower-level first-stage GBY nodes are defined as "pre-aggregation nodes," and the GBY operators pushed down to the pre-aggregation nodes are the lower-level GBY operators. At the same time, the upper-level second-stage GBY nodes are defined as "aggregation nodes," and the GBY operators located in the aggregation nodes are the upper-level GBY operators. This specification will be explained according to this definition, and will not be elaborated further.

[0079] In the pre-aggregation node, all data in the grouping pushdown process can be viewed as two stages: the sampling stage and the processing stage.

[0080] See Figure 2 As shown, during the sampling phase, the lower-level pre-aggregation nodes determine whether the aggregation effect is good or bad based on the data from the sampling phase. If the aggregation effect is good, the nodes can remain in the sampling phase; otherwise, if the aggregation effect is bad, the nodes enter the processing phase.

[0081] The sampling phase mainly consists of two parts. First, it assesses the data aggregation effect. If the data aggregation effect is good, the data processing can continue in the sampling phase. If the data aggregation effect is poor, the processing phase can begin. Second, if the data aggregation effect is poor, a high-frequency hash table is created based on the data from the sampling phase. This high-frequency hash table records the hash values ​​of the data that appear most frequently in the sampling phase.

[0082] The processing stage requires using the high-frequency value hash table from the sampling stage to filter the data and determine whether it is high-frequency data. If a data point is high-frequency data, pre-aggregation processing is performed on it; if a data point is low-frequency data, pre-aggregation processing is not required.

[0083] Based on the foregoing, it's understandable that if data aggregation is ineffective, continuing to pre-aggregate the data will not only fail to improve performance but will also reduce query performance due to the additional overhead of pre-aggregation operations. Conversely, distributing data directly without pre-aggregation can easily lead to data skew issues in upper-level aggregation nodes due to subsequent high-frequency data.

[0084] Therefore, in this embodiment, if poor data aggregation is detected during the sampling phase, a high-frequency value hash table is obtained based on the hash values ​​of high-frequency data from the sampling phase. Thus, during the processing phase, pre-aggregation processing is performed only on the high-frequency data in the high-frequency value hash table, while low-frequency data is not pre-aggregated. Because high-frequency data is pre-aggregated, the amount of aggregation result data distributed to the upper-layer aggregation node thread is smaller, eliminating or mitigating the data skew problem of the upper-layer aggregation node. Furthermore, since pre-aggregation processing is only performed on high-frequency data, and low-frequency data is not pre-aggregated, the overhead of the pre-aggregation operation is reduced, effectively balancing aggregation overhead and data volume, and improving data query performance.

[0085] like Figure 3 As shown, in some embodiments, the data processing method exemplified in this specification includes:

[0086] S310. Determine the data aggregation rate based on the sampled data during the grouping pushdown process.

[0087] Combination Figure 1 As shown, during the GPD grouping pushdown process, the data of the lower-level pre-aggregation nodes need to undergo a data pre-aggregation locally based on the lower-level GBY operator. As a result, the amount of data before and after pre-aggregation will change. The aggregation rate can reflect the data compression ratio before and after data pre-aggregation, and thus reflect the data pre-aggregation effect.

[0088] In the embodiments described in this specification, during the sampling phase and data pre-aggregation processing, a certain amount of sampled data can be acquired, and the data volume of the sampled data before and after pre-aggregation processing is statistically analyzed. The corresponding data aggregation rate is then calculated based on the data volume of the sampled data before and after pre-aggregation processing. The specific process for calculating the data aggregation rate is described in the embodiments below.

[0089] In some implementations, during the sampling phase, sampled data can be acquired periodically according to a fixed amount of data. Then, the data aggregation rate for the current period is calculated based on the sampled data. If the data aggregation rate meets the aggregation effect, it indicates that the current data pre-aggregation effect is high, thus continuing the sampling phase. Then, the data aggregation rate is calculated again based on the sampled data of the next period, until the data aggregation rate no longer meets the aggregation effect, indicating that the current data aggregation effect is poor, and then the processing phase begins. The implementation methods described below in this specification will be explained in detail here.

[0090] Furthermore, in the embodiments described in this specification, during the sampling phase, the hash value of each data point can be calculated. For example, a hash operation can be performed based on the group by key for each data point to obtain the corresponding hash value. The purpose of calculating the hash values ​​of the data is to compute a high-frequency hash table, including high-frequency data, based on the sampled data.

[0091] S320. When the data aggregation rate does not meet the aggregation effect, generate a high-frequency value hash table based on the sampled data.

[0092] As discussed above, if the data aggregation rate does not meet the aggregation effect during the sampling phase, it indicates that the data aggregation effect is poor, and the processing phase is required. Simultaneously, a high-frequency hash table needs to be generated based on the sampled data from the sampling phase. This high-frequency hash table records the hash values ​​of frequently occurring data in the sampled data.

[0093] For example, in one sample, during the sampling phase, the hash value of each data point can be counted. If it is determined that the data aggregation rate does not meet the aggregation effect, the hash values ​​of each data point in the sampled data can be used to calculate which data points are high-frequency data that may cause data skew, and the hash values ​​of these high-frequency data points can be recorded to obtain a high-frequency hash table.

[0094] It is worth noting that the high-frequency data recorded in the high-frequency value hash table refers to data that may cause data skew in the upper-level aggregation nodes. Therefore, when calculating whether a certain data is high-frequency data, it is not only necessary to consider the frequency of the hash value of the data in all sampled data, but also to consider the difference in data volume between the data and other data after being distributed to different threads. The greater the difference in data volume, the more serious the data skew program is.

[0095] Therefore, in the embodiments of this specification, when calculating whether a certain data is high-frequency data based on the sampled data, a preset threshold can be set in advance, and then the frequency of the hash value of the data in the sampled data can be counted. After that, the ratio of the frequency of the data to the average amount of data distributed to each thread by other data can be calculated, and the ratio can be compared with the preset threshold. If it exceeds the preset threshold, it means that the data is high-frequency data, otherwise it is low-frequency data. The embodiments described below in this specification will explain this.

[0096] After calculating all the high-frequency data included in the sampled data through the above process, a high-frequency value hash table can be generated based on the hash values ​​of all the high-frequency data.

[0097] S330: Based on a high-frequency value hash table, perform pre-aggregation processing on high-frequency data in the data to be processed, and distribute the pre-aggregation processing results to the target thread of the aggregation node.

[0098] As discussed above, when the data aggregation rate does not meet the aggregation effect, the process enters the processing stage. The goal of this stage is to filter the data to be processed based on the high-frequency data recorded in the high-frequency hash table. If the hash value of a piece of data in the processing stage matches the same hash value in the high-frequency hash table, it indicates that the data is high-frequency, and thus pre-aggregation processing can be performed. Conversely, if the hash value of a piece of data in the processing stage does not match the same hash value in the high-frequency hash table, it indicates that the data is low-frequency, and therefore pre-aggregation processing is unnecessary; the data can be directly distributed to the target thread of the upper-level aggregation node.

[0099] In the embodiments described in this specification, after pre-aggregating the high-frequency data, the pre-aggregation result is obtained. Then, based on the grouping keys of the data, the pre-aggregation result is distributed to the target thread of the upper-level aggregation node. It is worth noting that the aggregation and pre-aggregation described in the embodiments of this specification include, but are not limited to, summation, averaging, counting, maximum value, minimum value, median, variance, standard deviation, etc., and this specification does not limit these.

[0100] In some implementations, considering that the sampled data in the sampling phase cannot accurately represent the true data distribution during the entire GPD group pushdown period, the high-frequency value hash table generated in the sampling phase can be further tested in the processing phase using the data from the processing phase. This determines whether the high-frequency data recorded in the high-frequency value hash table meets the high-frequency index. If all the high-frequency data recorded in the high-frequency value hash table does not meet the high-frequency index, then there is no need to continue the above data filtering process, and the data can be directly distributed to the upper-layer aggregation node. The implementation methods described below in this specification will not be detailed here.

[0101] It is understood that the data processing method of this disclosure, compared with the adaptive GBY scheme in related technologies, does not directly send data to the upper-layer node when the data aggregation effect is not good. Instead, it uses a high-frequency value hash table in the sampling stage to filter high-frequency data in the processing stage and performs pre-aggregation processing on high-frequency data that may cause data skew in the upper-layer node to eliminate or alleviate the data skew problem. Low-frequency data is sent directly to the upper-layer node.

[0102] As can be seen from the above, in the embodiments of this specification, when the data aggregation effect is detected to be poor during the sampling stage, a high-frequency value hash table is obtained based on the hash value of the high-frequency data in the sampling stage. Thus, in the processing stage, only the high-frequency data in the high-frequency value hash table is pre-aggregated, eliminating or alleviating the data skew problem of the upper-layer aggregation node, effectively balancing the aggregation overhead and the amount of data distributed, and improving the data query performance.

[0103] Figure 4 The following describes the grouped pushdown data processing procedure in some embodiments of this specification, in conjunction with... Figure 4 An example is provided to illustrate the data processing method of the embodiments described in this specification.

[0104] In some implementations, during the sampling phase, sampled data can be acquired periodically with a first preset data volume T, and then the data aggregation rate of the current period can be calculated based on the sampled data of the data volume T to determine whether the aggregation effect is met.

[0105] Combination Figure 4 As shown, the amount of sampled data obtained in sampling phase 1 is the first preset data amount T. Then, based on the sampled data of this period, the data aggregation rate Ragg is calculated, expressed as:

[0106]

[0107] In formula (1), R agg Let R represent the data aggregation rate, T represent the first data volume before pre-aggregation of the sampled data, and M represent the second data volume after pre-aggregation of the sampled data. Combining formula (1), we can see that the data aggregation rate R... agg A smaller value indicates a better data pre-aggregation effect, and vice versa. The data aggregation rate R... agg The larger the value, the worse the data aggregation effect.

[0108] Additionally, the data aggregation rate R can be pre-set. agg Set threshold R thre The threshold R thre This indicates the critical value at which the data aggregation rate satisfies the aggregation effect.

[0109] In sampling phase 1, the data aggregation rate R corresponding to sampling phase 1 can be calculated by combining formula (1). agg Then, the data aggregation rate R is... agg With threshold R thre For comparison, assuming the data aggregation rate R in sampling phase 1 is... agg Less than threshold R thre This indicates that the data aggregation effect in sampling phase 1 is good, so the sampling phase of the next cycle, namely sampling phase 2, will continue.

[0110] For sampling phase 2, also using the above formula (1), based on the sampled data of size T collected in sampling phase 2, the data aggregation rate R corresponding to sampling phase 2 can be calculated. agg Then, the data aggregation rate R is... agg With threshold R thr For comparison, assuming the data aggregation rate R in sampling phase 2... agg Greater than or equal to the threshold R thre This indicates that the data aggregation effect in sampling phase 2 is poor.

[0111] In the embodiments described in this specification, if it is determined that the data aggregation rate in the sampling stage does not meet the aggregation effect, the process enters the processing stage. At the same time, it is necessary to generate a high-frequency hash table based on the sampled data from the sampling stage.

[0112] In some implementations, combined Figure 4 As shown, the sampling data used to generate the high-frequency value hash table may include only the sampling data of sampling stage 2 with a data volume of T, or it may include the sampling data of sampling stage 1 and sampling stage 2 with a data volume of 2T. This specification does not impose any restrictions on this.

[0113] In the following text of this manual, the sampling data of sampling phase 2 with a data volume of T is used as an example, combined with... Figure 5 The implementation method describes the process of generating a high-frequency value hash table.

[0114] like Figure 5 As shown, in some embodiments, the data processing method exemplified in this specification, the process of generating a high-frequency value hash table based on sampled data T, includes:

[0115] S510. In the sampled data with a total data volume of T, calculate the corresponding hash value based on the grouping key of each data, and count the first data volume that has the same hash value as the i-th data, where 1≤i≤T.

[0116] S520. Based on the total data volume, the first data volume, and the number of threads pushing down in groups, determine the second data volume of other data whose hash value is different from that of the i-th data and distribute them evenly to each thread.

[0117] S530. Based on the ratio of the first data volume to the second data volume, determine whether the i-th data is high-frequency data.

[0118] S540. Generate a high-frequency value hash table based on all high-frequency data and their hash values ​​in the sampled data.

[0119] In the embodiments described in this specification, the total amount of sampled data is T. For each data in the sampled data, a hash operation can be performed on the group by key of each data to obtain the corresponding hash value, thereby obtaining the hash values ​​of T data.

[0120] In some implementations, the hash value of each data point can be counted during the sampling phase, so that the hash value of each data point in the sampled data can be obtained when it is determined to enter the processing phase. This specification will not elaborate further on this.

[0121] Based on the principle of hash operation, data with the same grouping key should also have the same hash value after hash operation. Therefore, for T data, there may be data with the same grouping key, and thus the hash value obtained will also contain many identical hash values.

[0122] For any data i (1≤i≤T) in the sampled data, we can count the frequency k of the hash value of data i among all the hash values. i The frequency of occurrence k i This can be understood as: the amount of data whose hash value is the same as that of data i among all the hash values ​​of the data, and this amount of data is the first data amount k as described in this specification. i .

[0123] As discussed above, data skew in the upper-level aggregation node refers to the uneven distribution of data across threads. Therefore, when calculating whether the i-th data point is a high-frequency data point, it is necessary to compare the first data point k. i The average amount of data distributed among the threads (i.e., the second data amount L) compared to other data. i The greater the difference between them, the greater the risk of data skew.

[0124] In the embodiments described in this specification, the second data quantity L i Represented as:

[0125]

[0126] In formula (2), T represents the total amount of sampled data, and k i This represents the first data quantity of data that has the same hash value as the i-th data, thus (Tk iThis represents the number of data points in the sampled data whose hash value differs from that of the i-th data point. `dop` represents the number of threads in the upper-level aggregation node, for example... Figure 1 In the example, the thread data of the upper-level aggregation node is 4.

[0127] Therefore, L i This represents the average amount of data that differs from the hash value of the i-th data point, distributed evenly across all threads. This average amount of data is also the second data amount L described in this specification. i .

[0128] After obtaining the first data quantity k i Second data volume L i Then, using the first data quantity k i With the second data volume L i The difference between the two can be expressed as a ratio, as follows:

[0129]

[0130] In formula (3), ratio represents the ratio of the first data quantity ki to the second data quantity L. i The ratio.

[0131] In some implementations, the first data volume ki and the second data volume L can be pre-defined. i The ratio is set to a corresponding preset threshold ratio_thre, which represents the critical value for high-frequency data.

[0132] For the i-th data, if its corresponding first data quantity ki and second data quantity L i If the ratio is greater than or equal to the preset threshold ratio_thre, it indicates that the ratio of the first data volume ki to the second data volume L is greater than or equal to the preset threshold ratio_thre. i The differences between them are large, so after the data is distributed, the amount of data in the thread containing the i-th data will be much greater than the average amount of data in other threads, thus the risk of data skew is very high. Therefore, the i-th data can be determined to be high-frequency data.

[0133] Conversely, if the first data quantity ki corresponding to the i-th data is equal to the second data quantity L... i If the ratio is less than the preset threshold ratio_thre, it indicates that the ratio of the first data volume ki to the second data volume L is less than the preset threshold ratio_thre. i The differences between them are small, so after the data is distributed, the amount of data in the thread containing the i-th data is close to the average amount of data in other threads, and the risk of data skew is low. Therefore, the i-th data can be determined to be low-frequency data.

[0134] By performing high-frequency detection on a sample dataset of size T using the above method, all high-frequency data included in the sample dataset can be obtained. Then, based on these high-frequency data and their hash values, a high-frequency value hash table is generated. That is, the high-frequency value hash table records each high-frequency data in the sample dataset and its corresponding hash value.

[0135] As can be seen from the above, in the embodiments of this specification, high-frequency data is determined based on the difference between the frequency of occurrence of the hash value of data i and the average amount of other data distributed to each thread. The high-frequency hash table generated fully considers the data distribution characteristics of data skew and can provide an efficient data foundation for subsequent data skew processing.

[0136] Continue to refer to Figure 4 In sampling phase 2, if the data aggregation rate does not meet the aggregation effect, the process enters the processing phase. Simultaneously, a high-frequency value hash table is generated using the methods described above. The following section combines... Figure 6 The data processing procedure in the processing phase is described.

[0137] like Figure 6 As shown, in some embodiments, the data processing method exemplified in this specification includes the following data processing steps in the processing phase:

[0138] S610. For each piece of data in the data to be processed, calculate the corresponding hash value based on the grouping key of the data.

[0139] In the embodiments described in this specification, the data to be processed is the data in the processing stage. For each data in the data to be processed, a hash operation can be performed on the group by key to obtain the corresponding hash value.

[0140] S620. Match the hash value of the data with the hash value in the high-frequency hash table. If the match is consistent, determine that the data is high-frequency data.

[0141] In this embodiment of the specification, taking a specific piece of data as an example, after calculating the hash value of the data, the hash value can be matched with the hash values ​​in the high-frequency hash table. If the same hash value is found in the high-frequency hash table, it indicates that the data is high-frequency data. Conversely, if the same hash value is not found in the high-frequency hash table, it indicates that the data is not high-frequency data and is low-frequency data.

[0142] S630. Perform pre-aggregation processing on high-frequency data to obtain the pre-aggregation processing result, and determine the target thread based on the grouping key of the high-frequency data.

[0143] In the embodiments of this specification, after determining the high-frequency data and low-frequency data based on the aforementioned method process, only the high-frequency data can be pre-aggregated. The basis for the pre-aggregation is the grouping key of the data, that is, the data with the same grouping key are pre-aggregated. Those skilled in the art can understand this, and this specification will not elaborate further.

[0144] For high-frequency data, after pre-aggregation processing, the corresponding pre-aggregation result is obtained, and then the pre-aggregation result is sent to the upper-level aggregation node. During data distribution, it is necessary to determine which thread to distribute the pre-aggregation result to based on the data's group by keys, that is, to determine the target thread of the upper-level aggregation node, and then distribute the pre-aggregated data result to the target thread.

[0145] For low-frequency data, there is no need for pre-aggregation processing. The corresponding target thread is determined directly based on the grouping key of the data, and the low-frequency data is distributed to the target thread.

[0146] Combination Figure 1 As shown, after receiving the data, each thread of the upper-level aggregation node can complete the final aggregation processing of the data through the upper-level GBY operator and return the data query results. This manual will not elaborate further on this.

[0147] In some implementations, considering that the sampled data during the sampling phase cannot accurately represent the true data distribution during the entire GPD pushdown period—for example, a certain data may be high-frequency during the sampling phase but appear very infrequently during the processing phase—if all high-frequency data in the high-frequency value hash table generated based on the sampled data no longer meet the high-frequency index, then the process of filtering high-frequency data based on the high-frequency value hash table during the processing phase will incur unnecessary performance overhead, leading to reduced data query performance.

[0148] Therefore, in this embodiment, during the processing stage, the data to be processed can be periodically acquired with a second preset data volume, and the high-frequency data in the high-frequency value hash table can be detected using the data to be processed to determine whether these high-frequency data still meet the high-frequency index. If all high-frequency data do not meet the high-frequency index, the above-mentioned processing stage process is stopped, and data is directly sent to the upper-layer aggregation node without performing pre-aggregation operations. The following is a further explanation... Figure 7 Please provide an explanation.

[0149] like Figure 7 As shown, in some embodiments, the data processing method exemplified in this specification includes:

[0150] S710. Periodically acquire the data to be processed with a second preset data volume, and determine the frequency value parameter of each high-frequency data in the high-frequency value hash table based on the data to be processed, until the frequency value parameter of all high-frequency data in the high-frequency hash table does not meet the set requirements.

[0151] S720, In response to the fact that the frequency value parameter of at least one high-frequency data stored in the high-frequency value hash table meets the set requirements, the process of performing pre-aggregation processing on the high-frequency data in the data to be processed based on the high-frequency value hash table, and distributing the pre-aggregation processing result to the target thread of the aggregation node.

[0152] S730: In response to the fact that the frequency value parameters of all high-frequency data in the high-frequency value hash table do not meet the set requirements, the data to be processed is directly distributed to the aggregation node.

[0153] In the embodiments described in this specification, during the processing stage, the data to be processed can be acquired periodically at a fixed data volume, which is the second preset data volume. For example... Figure 4 In the example, the second preset data volume can be 10T, that is, 10T of data to be processed is acquired in each cycle.

[0154] After obtaining 10T of data to be processed, the frequency parameters of the high-frequency data in the high-frequency hash table can be calculated sequentially based on the data to be processed. The frequency parameter can be the ratio in the aforementioned formula (3). That is, the process of calculating the frequency parameter of a certain high-frequency data is similar to the aforementioned formulas (2) to (3), the difference being that the data for calculating the frequency parameter ratio is 10T of data to be processed. The rest of the process is exactly the same, and this manual will not elaborate further.

[0155] In the embodiments described in this specification, a corresponding threshold can be preset for the frequency parameter, which can be the aforementioned preset threshold ratio_thre.

[0156] It's understandable that for each high-frequency data point in the high-frequency hash table, after calculating its corresponding frequency parameter `ratio` based on 10T of data to be processed, the frequency parameter `ratio` can be compared with a preset threshold `ratio_thre`. If the frequency parameter `ratio` is greater than or equal to the preset threshold `ratio_thre`, it means that the high-frequency data meets the set requirements and is still considered high-frequency data. Conversely, if the calculated frequency parameter `ratio` for a high-frequency data point in the high-frequency hash table is less than the preset threshold `ratio_thre`, it means that the high-frequency data no longer meets the set requirements and has become low-frequency data.

[0157] After determining whether each high-frequency data in the high-frequency hash table meets the set requirements through the above-described process, if at least one high-frequency data in the high-frequency hash table meets the set requirements, it indicates that at least one high-frequency data still exists in the high-frequency hash table. Therefore, the current processing stage can be maintained, and the aforementioned steps can continue. Figure 6 The method and process are shown.

[0158] Conversely, if the frequency parameters of all high-frequency data in the high-frequency hash table do not meet the set requirements, it means that all high-frequency data in the high-frequency hash table has become low-frequency data. Therefore, the process of filtering high-frequency data based on the high-frequency hash table during processing will result in unnecessary performance overhead. Figure 4 As shown, the aforementioned steps can be skipped. Figure 6 The method shown directly distributes data to the various threads of the upper-level aggregation node.

[0159] Of course, those skilled in the art will understand that in other embodiments, the aforementioned process can be stopped if a certain percentage of high-frequency data in the high-frequency hash table does not meet the set requirements. Figure 6 The method shown can be used to, for example, in cases where more than half of the high-frequency data in the high-frequency hash table does not meet the set requirements, the lower-level pre-aggregation node can directly distribute the data to the various threads of the upper-level aggregation node without performing pre-aggregation processing.

[0160] As can be seen from the above, in the embodiments of this specification, high-frequency data in the high-frequency value hash table is periodically detected during the processing stage. When the high-frequency hash table no longer meets the set requirements, the data is directly distributed to the upper-layer aggregation node, and the pre-aggregation operation is no longer performed. This avoids the performance loss caused by the process of filtering high-frequency data based on the high-frequency value hash table and improves the data query performance.

[0161] In some embodiments, this specification provides a data processing apparatus, such as... Figure 8 As shown, the data processing device includes:

[0162] The aggregation rate determination module 10 is configured to determine the data aggregation rate based on the sampled data during the grouping pushdown process;

[0163] The hash table generation module 20 is configured to generate a high-frequency value hash table based on the sampled data when the data aggregation rate does not meet the aggregation effect. The high-frequency value hash table includes the hash values ​​of high-frequency data in the sampled data.

[0164] The pre-aggregation module 30 is configured to perform pre-aggregation processing on high-frequency data in the data to be processed based on the high-frequency value hash table, and distribute the pre-aggregation processing results to the target thread of the aggregation node.

[0165] In one or more embodiments of this specification, the polymerization rate determination module 10 is configured to:

[0166] The data aggregation rate is determined based on the first data volume of the sampled data during the grouping pushdown process and the second data volume after the sampled data has undergone pre-aggregation processing.

[0167] In one or more embodiments of this specification, the polymerization rate determination module 10 is configured to:

[0168] The sampled data is periodically acquired with a first preset data volume, and the process of determining the data aggregation rate based on the sampled data is executed until the data aggregation rate of the current period does not meet the aggregation effect.

[0169] In one or more embodiments of this specification, the hash table generation module 20 is configured to:

[0170] In the sampled data with a total data volume of T, the corresponding hash value is calculated based on the grouping key of each data, and the first data volume with the same hash value as the i-th data is counted, where 1≤i≤T;

[0171] Based on the total data volume, the first data volume, and the number of threads pushing down in groups, determine the second data volume of other data that have different hash values ​​from the i-th data and distribute them evenly to each thread;

[0172] Based on the ratio of the first data volume to the second data volume, determine whether the i-th data is high-frequency data;

[0173] The high-frequency value hash table is generated based on all high-frequency data and their hash values ​​in the sampled data.

[0174] In one or more embodiments of this specification, the hash table generation module 20 is configured to:

[0175] In response to the ratio being greater than or equal to a preset threshold, the i-th data is determined to be high-frequency data;

[0176] In response to the proportion being less than the preset threshold, the i-th data is determined to be low-frequency data.

[0177] In one or more embodiments of this specification, the pre-aggregation module 30 is configured to:

[0178] For each piece of data in the data to be processed, the corresponding hash value is calculated based on the grouping key of the data;

[0179] The hash value of the data is matched with the hash value in the high-frequency value hash table. If the match is consistent, the data is determined to be high-frequency data.

[0180] The high-frequency data is pre-aggregated to obtain the pre-aggregation result, and the target thread is determined based on the grouping key of the high-frequency data;

[0181] The pre-aggregation processing results are distributed to the target thread.

[0182] In one or more embodiments of this specification, the pre-aggregation module 30 is configured to:

[0183] The data to be processed is periodically acquired with a second preset data volume, and the frequency value parameter of each high-frequency data in the high-frequency value hash table is determined based on the data to be processed, until the frequency value parameter of all high-frequency data in the high-frequency value hash table does not meet the set requirements. The frequency value parameter represents the frequency of occurrence of the high-frequency data in the data to be processed.

[0184] In response to the presence of at least one high-frequency data whose frequency value parameter meets the set requirements in the high-frequency value hash table, the process of performing pre-aggregation processing on the high-frequency data in the data to be processed based on the high-frequency value hash table and distributing the pre-aggregation processing result to the target thread of the aggregation node is executed.

[0185] If the frequency value parameters of all high-frequency data in the high-frequency value hash table do not meet the set requirements, the data to be processed is directly distributed to the aggregation node.

[0186] In some embodiments, this specification provides an electronic device comprising:

[0187] processor; and

[0188] The memory stores computer instructions that cause the processor to perform the method described in any of the above embodiments.

[0189] In some embodiments, this specification provides a storage medium storing computer instructions for causing a computer to perform the methods described in any of the above embodiments.

[0190] Figure 9 This is a schematic structural diagram of an electronic device provided as an exemplary embodiment. Please refer to... Figure 9At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, memory 708, and non-volatile memory 710, and may also include other hardware required for different scenarios. One or more embodiments of this specification can be implemented in software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into memory 708 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0191] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0192] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0193] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0194] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0195] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0196] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0197] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” used in one or more embodiments of this specification and in the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0198] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to a determination," or "when," or "in the event of a determination."

[0199] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification shall be included within the scope of protection of one or more embodiments of this specification.

Claims

1. A data processing method, comprising: The data aggregation rate is determined based on the sampled data during the grouping pushdown process; If the data aggregation rate does not meet the aggregation effect, a high-frequency value hash table is generated based on the sampled data. The high-frequency value hash table includes the hash values ​​of high-frequency data in the sampled data. Based on the high-frequency value hash table, high-frequency data in the data to be processed is pre-aggregated, and the pre-aggregation results are distributed to the target thread of the aggregation node. The method further includes: The data to be processed is periodically acquired with a second preset data volume, and the frequency value parameter of each high-frequency data in the high-frequency value hash table is determined based on the data to be processed, until the frequency value parameter of all high-frequency data in the high-frequency value hash table does not meet the set requirements. The data to be processed is then directly distributed to the aggregation node, where the frequency value parameter represents the frequency of occurrence of the high-frequency data in the data to be processed.

2. The data processing method according to claim 1, wherein determining the data aggregation rate based on the sampled data during the grouping pushdown process includes: The data aggregation rate is determined based on the first data volume of the sampled data during the grouping pushdown process and the second data volume after the sampled data has undergone pre-aggregation processing.

3. The data processing method according to claim 1, wherein determining the data aggregation rate based on the sampled data during the grouping pushdown process includes: The sampled data is periodically acquired with a first preset data volume, and the process of determining the data aggregation rate based on the sampled data is executed until the data aggregation rate of the current period does not meet the aggregation effect.

4. The data processing method according to any one of claims 1 to 3, wherein generating a high-frequency hash table based on the sampled data comprises: In the sampled data with a total data volume of T, the corresponding hash value is calculated based on the grouping key of each data, and the first data volume with the same hash value as the i-th data is counted, where 1≤i≤T; Based on the total data volume, the first data volume, and the number of threads pushing down in groups, determine the second data volume of other data that have different hash values ​​from the i-th data and distribute them evenly to each thread; Based on the ratio of the first data volume to the second data volume, determine whether the i-th data is high-frequency data; The high-frequency value hash table is generated based on all high-frequency data and their hash values ​​in the sampled data.

5. The data processing method according to claim 4, wherein determining whether the i-th data is high-frequency data based on the ratio of the first data volume to the second data volume includes: In response to the ratio being greater than or equal to a preset threshold, the i-th data is determined to be high-frequency data; In response to the ratio being less than the preset threshold, the i-th data is determined to be low-frequency data.

6. The data processing method according to claim 1, wherein the step of pre-aggregating high-frequency data in the data to be processed based on the high-frequency value hash table, and distributing the pre-aggregation processing result to the target thread of the aggregation node, comprises: For each piece of data in the data to be processed, the corresponding hash value is calculated based on the grouping key of the data; The hash value of the data is matched with the hash value in the high-frequency value hash table. If the match is consistent, the data is determined to be high-frequency data. The high-frequency data is pre-aggregated to obtain the pre-aggregation result, and the target thread is determined based on the grouping key of the high-frequency data; The pre-aggregation processing results are distributed to the target thread.

7. The data processing method according to claim 1 further includes: In response to the presence of at least one high-frequency data whose frequency value parameter meets the set requirements in the high-frequency value hash table, the process of performing pre-aggregation processing on the high-frequency data in the data to be processed based on the high-frequency value hash table and distributing the pre-aggregation processing result to the target thread of the aggregation node is executed.

8. A data processing apparatus, comprising: The aggregation rate determination module is configured to determine the data aggregation rate based on the sampled data during the grouping pushdown process; The hash table generation module is configured to generate a high-frequency value hash table based on the sampled data when the data aggregation rate does not meet the aggregation effect. The high-frequency value hash table includes the hash values ​​of high-frequency data in the sampled data. The pre-aggregation module is configured to perform pre-aggregation processing on high-frequency data in the data to be processed based on the high-frequency value hash table, and distribute the pre-aggregation processing results to the target thread of the aggregation node; The pre-aggregation module is configured as follows: The data to be processed is periodically acquired with a second preset data volume, and the frequency value parameter of each high-frequency data in the high-frequency value hash table is determined based on the data to be processed, until the frequency value parameter of all high-frequency data in the high-frequency value hash table does not meet the set requirements. The data to be processed is then directly distributed to the aggregation node, where the frequency value parameter represents the frequency of occurrence of the high-frequency data in the data to be processed.

9. An electronic device, comprising: processor; and A memory storing computer instructions for causing a processor to perform the method according to any one of claims 1 to 7.

10. A storage medium storing computer instructions for causing a computer to perform the method according to any one of claims 1 to 7.

Images