Data aggregation method, distributed system, computing device and readable storage medium

Abstract

The embodiments of this specification provide a data aggregation method, a distributed system, a computing device, and a readable storage medium, wherein the data aggregation method is used for a local node of a distributed system, and the distributed system also includes a central node, and the method includes: obtaining a plurality of data to be aggregated, wherein any data to be aggregated corresponds to a first key; aggregating the plurality of data to be aggregated to obtain local aggregated data, and updating the heat weight of the first key; determining a target key from the at least one key according to the heat weight of at least one key corresponding to each data in the local node, wherein the at least one key includes the first key; transmitting the target data corresponding to the target key to the central node for aggregation to obtain global aggregated data, which not only improves the efficiency of local aggregation, but also reduces unnecessary data transmission overhead, and significantly improves the speed and efficiency of data aggregation in the distributed system.

Description

Technical Field

[0001] The embodiments of this specification relate to the technical field of database and big data analysis, and in particular to a data aggregation method, a distributed system, a computing device, and a readable storage medium. Background Art

[0002] In the field of big data, with the rapid growth of data volume, traditional data aggregation methods face huge performance bottlenecks. Especially when performing large-scale data aggregation in distributed systems, the memory access (I / O) overhead of data transmission between nodes becomes one of the performance bottlenecks.

[0003] At present, in order to reduce memory access overhead and network bandwidth usage, data aggregation on distributed systems usually adopts a "two-stage aggregation" strategy, that is, first partially aggregate the data locally on the local node, and then transmit it to the central node for global aggregation. However, such a "two-stage aggregation" strategy can often achieve better aggregation effects when processing high-cardinality (frequency) data to be aggregated. In reality, there are a large number of mixed high-cardinality and low-cardinality data to be aggregated. Too much low-cardinality data to be aggregated will lead to insufficient local aggregation effects, slow down the efficiency of local aggregation, introduce additional aggregation overhead, and cannot effectively reduce data transmission overhead. It is more suitable for a "one-stage aggregation" strategy of direct global aggregation. In the real-time data aggregation scenario on a distributed system, how to dynamically and flexibly distinguish high-cardinality and low-cardinality data to be aggregated and execute appropriate aggregation strategies is an urgent problem to be solved. Summary of the invention

[0004] In view of this, an embodiment of this specification provides a data aggregation method. One or more embodiments of this specification also relate to another data aggregation method, a distributed system, a data aggregation device, another data aggregation device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0005] In one embodiment of the present specification, a data aggregation method is provided, which is applied to a local node of a distributed system, and the distributed system also includes a central node. The method includes:

[0006] Acquire multiple data to be aggregated, wherein any data to be aggregated corresponds to the first key;

[0007] Aggregate multiple data to be aggregated, obtain local aggregated data, and update the heat weight of the first key;

[0008] Determine a target key from the at least one key according to a heat weight of at least one key corresponding to each data in the local node, wherein the at least one key includes a first key;

[0009] The target data corresponding to the target key is transmitted to the central node for aggregation to obtain global aggregated data.

[0010] By introducing the key heat weight mechanism on the local node, it reflects the frequency of key aggregation in a period of time. According to the heat weight of at least one key corresponding to each data in the local node, the target key is determined from at least one key, and the target keys with high and low cardinality are dynamically and flexibly distinguished. The target data corresponding to the target key is transmitted to the central node for global aggregation, the global aggregated data is obtained, and the appropriate aggregation strategy is executed. This not only improves the efficiency of local aggregation and reduces unnecessary data transmission overhead, but also significantly improves the speed and efficiency of data aggregation in distributed systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is one of the schematic diagrams of a data aggregation method;

[0012] Figure 2 This is the second schematic diagram of a data aggregation method;

[0013] Figure 3 This is a third schematic diagram of a data aggregation method;

[0014] Figure 4 is a flow chart of a data aggregation method provided by an embodiment of this specification;

[0015] Figure 5 is a schematic diagram of a data aggregation method provided by an embodiment of this specification;

[0016] Figure 6 is a flow chart of another data aggregation method provided by an embodiment of this specification;

[0017] Figure 7 It is a processing flow chart of a data aggregation method applied to Internet of Things data provided by an embodiment of this specification;

[0018] Figure 8 It is a schematic diagram of the structure of a distributed system provided by an embodiment of this specification;

[0019] Fig. 9 is a structural schematic diagram of a data aggregation device provided by an embodiment of this specification;

[0020] Fig.10 is a schematic diagram of the structure of another data aggregation device provided by an embodiment of this specification;

[0021] Fig.11 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0022] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0023] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0024] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0025] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0026] First, the terms involved in one or more embodiments of this specification are explained.

[0027] Data aggregation: The process of combining multiple data items into one or more meaningful summary data. This can be achieved through various methods, including statistical calculations, grouping, aggregation, etc. Including but not limited to: SELECT statement: used to specify the data columns to be retrieved, which can include the results of aggregate functions; SUM function: used to calculate the sum of a column; COUNT function: used to calculate the number of rows in a column; AVG function: used to calculate the average value of a column; MAX function: used to return the maximum value in a column; MIN function: used to return the minimum value in a column; MAX function: used to return the maximum value in a column; GROUP BY function: used to group data, usually used with aggregate functions.

[0028] Hash-based aggregation algorithm: an efficient data processing technology, often used in database systems to perform aggregate calculations, such as calculating sums, averages, counts, etc. It mainly uses hash tables to quickly group and summarize data. The specific process is as follows: 1. Grouping phase: In this phase, the database first scans the input data and generates a hash value for each aggregation key (such as the value of a column). This hash value is used to distribute the data to different buckets in the hash table. Each bucket corresponds to an aggregation key. 2. Aggregation phase: For each bucket, the database calculates the aggregation results of the group, such as sum, count, or other aggregation functions. This means that for rows with the same aggregation key, the database will perform calculations in the corresponding bucket. 3. Result output: Finally, the aggregation results will be output as a new data set, usually including the aggregation key and its corresponding aggregation function results.

[0029] Aggregation degree: An indicator used to measure the aggregation effect. It is usually calculated by comparing the number of rows before and after aggregation. Given the number of rows R of the aggregated column and the number of rows C after aggregation, the aggregation degree G can be calculated using Formula 1:

[0030] G=R / C Formula 1

[0031] Among them, R is the number of rows before aggregation (the number of rows of original data), and C is the number of rows after aggregation (the number of rows of aggregated data). The larger the value of the aggregation degree G, the smaller the number of rows of the same amount of original data, which usually indicates a better aggregation effect.

[0032] Two-Phase Aggregation: A common database query optimization technique, mainly used to efficiently perform aggregation operations when processing large-scale data sets. The basic idea is to divide the aggregation process into two stages to reduce the size of the intermediate result set and reduce the computational cost. The process of two-phase aggregation is as follows: 1. Phase 1: In the first stage, the data is divided into multiple partitions. Each partition will calculate the aggregation results (such as sum, count, average, etc.) locally. This usually involves local aggregation of each partition, such as calculating the respective data on each node first. The result of this stage is the local aggregation result of each partition, rather than the global aggregation result. 2. Phase 2: In the second stage, the local aggregation results of all partitions are aggregated together for the final global aggregation. This step usually requires the results of the first stage to be transferred from each partition to the master node, or centralized processing through other mechanisms. Finally, the global aggregation result required by the user is obtained.

[0033] Two-stage aggregation has the following advantages: 1. Reduced data transmission: The local aggregation results of the first stage are usually smaller than the original data set, so only a smaller result set needs to be transmitted to the second stage, reducing network bandwidth usage. 2. High efficiency: By calculating the aggregation locally, it can reduce the consumption of computing resources and improve the overall aggregation efficiency, especially in distributed or parallel processing environments. Usage scenarios of two-stage aggregation: It is often used in big data environments, distributed databases, and data warehouses, such as performing GROUP BY operations when executing SQL queries to improve performance and reduce resource usage. In actual applications, many modern database systems and big data processing frameworks have implemented similar mechanisms to optimize aggregation queries.

[0034] For two-stage aggregation, Figure 1 , Figure 2 and Figure 3 An example is shown.

[0035] Figure 1 One of the schematic diagrams of a data aggregation method is shown, Figure 1 As shown in the data aggregation scenario with high cardinality, two data to be aggregated in a batch: "aaaabbbbbaaaccc...cccxxxxxx" and "cccaaaaaxxxxxbbb...bbaaaacc" are locally aggregated on two local nodes respectively to obtain two local aggregated data: Local aggregated data 1 is shown in Table 1:

[0036]

[0037] Table 1

[0038] The local aggregate data 2 is shown in Table 2:

[0039]

[0040] Table 2

[0041] The local aggregate data 1 and the local aggregate data 2 are sent to two central nodes for aggregation to obtain two global aggregate data: the global aggregate data 1 is shown in Table 3:

[0042]

[0043] Table 3

[0044] The global aggregate data 2 is shown in Table 4:

[0045]

[0046] Table 4

[0047] According to Tables 1 to 4, it can be found that most of the data aggregation operations have been basically completed during local aggregation, and only simple aggregation is required on this basis during global aggregation. Therefore, when the aggregation degree of local aggregation is high, the amount of data access can be greatly reduced, improving the overall performance.

[0048] Figure 2 A second schematic diagram of a data aggregation method is shown, such as Figure 2 As shown in the data aggregation scenario with low cardinality, two data to be aggregated in a batch, "abcdef...xrule" and "nskly...znvpoz", are locally aggregated on two local nodes respectively to obtain two local aggregated data: Local aggregated data 1 is shown in Table 5:

[0049]

[0050] Table 5

[0051] The local aggregate data 2 is shown in Table 6:

[0052]

[0053] Table 6

[0054] The local aggregate data 1 and the local aggregate data 2 are sent to two central nodes for aggregation to obtain two global aggregate data: the global aggregate data 1 is shown in Table 7:

[0055]

[0056] Table 7

[0057] The global aggregate data 2 is shown in Table 8:

[0058]

[0059] Table 8

[0060] According to Tables 5 to 8, it can be found that data aggregation operations are basically not completed during local aggregation, and most data aggregation operations need to be performed on this basis during global aggregation. Therefore, when the aggregation degree of local aggregation is low, not only can the amount of data access be reduced, but the low-cardinality aggregation keys of local aggregation continue to accumulate locally, causing a sharp increase in local cache, and further causing overall performance regression. At this time, a better approach should be to upload the low-cardinality data to be aggregated to the global aggregation directly or as soon as possible, and do not accumulate it locally.

[0061] Figure 3 A third schematic diagram of a data aggregation method is shown, Figure 3 As shown in the figure: Data aggregation scenario when high cardinality and low cardinality are mixed. Two data to be aggregated in a batch: "aaaabbbbbaaaccc...cccxxxxxx" and "cccaaaaaxxxxxbbb...bbaaaacc" are locally aggregated on two local nodes respectively to obtain two local aggregated data: Local aggregated data 1 is shown in Table 9:

[0062]

[0063] Table 9

[0064] The local aggregate data 2 is shown in Table 10:

[0065]

[0066] Table 10

[0067] The local aggregate data 1 and the local aggregate data 2 are sent to two central nodes for aggregation to obtain two global aggregate data: the global aggregate data 1 is shown in Table 11:

[0068]

[0069] Table 11

[0070] The global aggregate data 2 is shown in Table 12:

[0071]

[0072] Table 12

[0073] According to Tables 9 to 12, it can be found that the overall degradation to direct global aggregation or continuing two-stage aggregation cannot achieve the ideal performance effect. A mechanism is needed to distinguish between high-cardinality and low-cardinality aggregation keys.

[0074] In most real-time scenarios, it is impossible to determine the cardinality of the data items corresponding to each aggregation key. One way is to use statistical information in the database, use technical means such as histograms to summarize and count the aggregation keys, and estimate the cardinality of each aggregation key. However, when the amount of data is too large, the data is updated very frequently, or the data is updated irregularly, this statistical method cannot obtain the accurate cardinality of the aggregation key because the aggregation key is complex (multi-layer key) or the aggregation key is the function calculation result of other aggregation keys. Therefore, it is impossible to set an accurate aggregation key cardinality threshold as a criterion for determining whether to execute the "one-stage aggregation" strategy or the "two-stage aggregation" strategy.

[0075] In response to the above problems, a data aggregation method is provided in this specification. This specification also involves another data aggregation method, a distributed system, a data aggregation device, another data aggregation device, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0076] See also Figure 4 , Figure 4 A flow chart of a data aggregation method provided by an embodiment of the present specification is shown. The method is applied to a local node of a distributed system, and the distributed system also includes a central node, and includes the following specific steps:

[0077] Step 402: Acquire multiple data to be aggregated, wherein any data to be aggregated corresponds to the first key.

[0078] The embodiments of this specification can be applied to real-time data analysis (such as traffic management, advertising, smart cities), Internet of Things applications (such as smart homes, industrial monitoring), operation and maintenance management, big data processing (such as search engines, video sites, social networking sites), and other scenarios that use distributed systems to aggregate data. The examples in the embodiments of this specification do not limit the application scenarios of the embodiments of this specification.

[0079] A distributed system is a system composed of multiple server nodes, and multiple server nodes transmit data through the network. In a distributed system, each server node can run a program independently or work together through a message passing mechanism to realize the functions of the entire distributed system. Distributed systems are highly scalable and reliable. In big data processing scenarios, distributed systems are used to process large amounts of data and accelerate the data processing process through parallel computing and distributed storage. For example, the Hadoop distributed system. The local node is a server node responsible for performing local aggregation operations in a distributed system. The local node has certain computing resources and storage resources, which are used to perform local aggregation to reduce the amount of data transmitted in the distributed system, thereby improving the performance of the overall system. For example, a data node (DataNode) in a Hadoop distributed system is responsible for storing and processing part of the data. The central node is a server node responsible for performing global aggregation operations in a distributed system. The central node is used to coordinate and manage local nodes in the distributed system, receive local aggregation results (intermediate results) or data to be aggregated from each local node, and perform global aggregation. For example, a name node (NameNode) in a Hadoop distributed system is responsible for managing and coordinating the file system of the entire Hadoop cluster. It should be noted that the local node is a server node in a distributed system used to aggregate part of the data, and the central node is a server node in a distributed system used to aggregate local aggregated data that has been aggregated by multiple local nodes. There may be no essential difference between the two in terms of physical aspects such as software and hardware resources and architecture. The only difference is the roles and responsibilities performed in the distributed system. The two are a set of relative concepts that work together to complete the two-stage aggregation strategy.

[0080] The multiple data to be aggregated are data items that need to be aggregated on the local node. The multiple data to be aggregated have corresponding aggregation keys, and the multiple data to be aggregated can belong to the same batch. For example, the order data generated from 19:00 to 19:30 on a local node includes three aggregation keys: merchant number, order number, and order amount.

[0081] The first key is the aggregation key corresponding to any current data to be aggregated. The aggregation key generally represents the category or data of the data to be aggregated. Multiple data to be aggregated can be aggregated into key-value pairs with a higher degree of aggregation through the aggregation key. For example, any of the three aggregation keys: merchant number, order number, and order amount can be used as the first key.

[0082] To obtain multiple data to be aggregated, an optional way is to read multiple data to be aggregated from a data source. The data source may be a database, a file system, a message queue, etc. For example, a batch of order data is read from the Hadoop Distributed File System (HDFS), and fields such as merchant number, order number, and order amount can be used as aggregation keys. To obtain multiple data to be aggregated, another optional way is to receive multiple data to be aggregated through a message queue. The message queue can be used for real-time data stream processing, such as Kafka, RabbitMQ, etc. For example, a batch of order data is consumed from Kafka. To obtain multiple data to be aggregated, another optional way is to pull multiple data to be aggregated through a data interface. The data interface may be an application programming interface (API), which obtains data from an external system, such as RESTful API, GraphQL, etc.

[0083] For example, a Hadoop distributed system is used to process order data of an e-commerce platform in real time. The Hadoop distributed system includes 2 central nodes and 4 local nodes. On one of the local nodes, 1,000,000 order data are read from the Hadoop file distribution system, and any order data includes a merchant number, an order number, and an order amount.

[0084] Obtain multiple data to be aggregated, providing data support for subsequent local aggregation. Any data to be aggregated corresponds to the first key, providing data support for subsequent updating of heat weight.

[0085] Step 404: Aggregate multiple data to be aggregated, obtain local aggregated data, and update the heat weight of the first key.

[0086] Local aggregated data is the data item generated after aggregating multiple data to be aggregated on the local node. It is the intermediate result of two-stage aggregation. The local aggregated result has a higher degree of aggregation than the multiple data to be aggregated. The local aggregated result contains the aggregation key and the aggregation function value corresponding to the aggregation key, which is expressed in the form of key-value pair data. Compared with directly transmitting multiple data to be aggregated to the central node for global aggregation, the local aggregated result is initially aggregated and then transmitted to the central node for global aggregation, which reduces the amount of data transmission in the network and improves the overall performance of the distributed system.

[0087] The heat weight of the first key is a quantitative indicator that measures the frequency of aggregation of the data items corresponding to the first key over a period of time. The heat weight reflects the activity of the first key. Some keys may be more important due to large data volume or high generation frequency, while other keys may be relatively inactive. By introducing the heat weight mechanism, compared with a fixed cardinality threshold, the data aggregation strategy of the key can be dynamically and flexibly evaluated and adjusted, thereby optimizing the data processing process and improving system performance. Heat weights include but are not limited to: total number of aggregation steps, number of interval steps, data volume, and other indicators (such as order amount, user access frequency, etc., depending on specific needs).

[0088] It should be noted that data aggregation corresponds to aggregation functions, taking SQL statements as an example, including but not limited to: SELECT function, SUM function, COUNT function, AVG function, MAX function, MIN function, and GROUP BY function.

[0089] Aggregate multiple data to be aggregated to obtain local aggregated data. One optional way is to calculate the aggregation function on multiple data to be aggregated to obtain local aggregated data. Another optional way is to use a hash table to aggregate multiple data to be aggregated to obtain local aggregated data. For example, create a hash table to store the aggregation results, traverse multiple data to be aggregated, determine the first key of each data item as a hash value, and assign it to the corresponding bucket of the hash table, calculate the aggregation function value in each bucket, and generate a local aggregation result. Another optional way is a combination of the above two ways, which is not limited here.

[0090] Update the heat weight of the first key. One optional way is to add the number of steps (fixed value) to the heat weight of the first key. Generally, the value is 1. Another optional way is to add the number of aggregation interval steps (non-fixed value) to the heat weight of the first key. Another optional way is to add the amount of data to be aggregated corresponding to the first key to the heat weight of the first key. There is no limitation here.

[0091] Exemplarily, the aggregate query statement (aggregate function) includes:

[0092] SELECT seller_id, COUNT(order_id) as number of orders

[0093] FROM Orders table

[0094] GROUP BY seller_id

[0095] Create a hash table on the local node to store local aggregation results. Traverse 1,000,000 order data, determine the first key "seller number" of each data item as the hash value, and assign it to the corresponding bucket of the hash table. Perform aggregation function calculation in each bucket to obtain local aggregate data: seller number 'user1', number of orders 20,000; seller number 'user4', number of orders 300; seller number 'user9', number of orders 20..., and add 1 to the total number of aggregation steps for the first key "seller number" ('user1', 'user4', 'user9', etc.).

[0096] Aggregate multiple data to be aggregated to obtain local aggregate data, provide intermediate result data for subsequent global aggregation, realize two-stage aggregation, reduce data transmission volume, reduce network bandwidth occupancy, reduce computing resource consumption, and improve overall aggregation efficiency. Update the heat weight of the first key, and provide weight support for subsequent determination of the target key from at least one key.

[0097] Step 406: Determine a target key from at least one key according to the heat weight of at least one key corresponding to each data in the local node, wherein the at least one key includes the first key.

[0098] The data in the local node are data items stored and processed on the local node, including data to be aggregated and / or local aggregated data. In the case where the local node has data cache, these data include the previous data to be aggregated and / or the local aggregated data after aggregation in addition to the data to be aggregated and the local aggregated data after aggregation that currently need to be aggregated.

[0099] Each data item has at least one key corresponding to it, which is the aggregation key corresponding to each data item (data to be aggregated or local aggregated data). The aggregation key generally represents the category or data of the data to be aggregated. Multiple data to be aggregated can be aggregated into key-value pair data with a higher degree of aggregation through the aggregation key.

[0100] The heat weight of at least one key is a quantitative indicator that measures the frequency of aggregation of the data items corresponding to at least one key within a period of time. Heat weights include but are not limited to: total number of aggregation steps, number of interval steps, data volume, and other indicators (such as order amount, user access frequency, etc., depending on specific needs). The target key is the aggregation key corresponding to the data item that needs to be globally aggregated according to the heat weight of at least one key corresponding to each data. Since the target key is selected based on the heat weight of at least one key, and the heat weight is updated after local aggregation, the target key is dynamic and flexible. For example, for the data items of the current high heat weight aggregation items, they are not transmitted to the central node for global aggregation, but are retained in the local node to avoid the subsequent acquisition of the data items of the aggregation items and repeated transmission. For the data items of the current low heat weight, they are directly transmitted to the central node for global aggregation. By retaining the aggregation key with high heat weight, it can be ensured that important data with high aggregation needs within a period of time can be processed preferentially in the local node.

[0101] According to the heat weight of at least one key corresponding to each data in the local node, the target key is determined from at least one key. An optional method is: according to the heat weight of at least one key corresponding to each data in the local node, the target key is determined from at least one key from low to high.

[0102] It should be noted that, according to the heat weight of at least one key corresponding to each data in the local node, the target key is determined from at least one key, and a window can be set, including but not limited to: a time window and a space window. When the window is triggered, the target key is determined from at least one key according to the heat weight of at least one key corresponding to each data in the local node. Specifically, when the preset time window is triggered, the target key is determined from at least one key according to the heat weight of at least one key corresponding to each data in the local node. Alternatively, when the space window of the cache space on the local node is reached, the target key is determined from at least one key according to the heat weight of at least one key corresponding to each data in the local node.

[0103] For example, when each hash bucket cannot be expanded and the hash table overflows, the total number of aggregation steps corresponding to 20 keys "seller ID" ('user1', 'user2', ... 'user20') of each data item in the local node (data to be aggregated and local aggregated data) is: seller ID 'user1', total number of aggregation steps 30000; seller ID 'user2', total number of aggregation steps 50000; ... seller ID 'user20', total number of aggregation steps 45000. Determine the target key from the 20 keys from low to high: seller ID 'user1'.

[0104] According to the heat weight of at least one key corresponding to each data in the local node, the target key is determined from at least one key, wherein at least one key includes a first key, and target keys with high and low cardinality are dynamically and flexibly distinguished, providing support for the subsequent transmission of target data to the central node.

[0105] Step 408: The target data corresponding to the target key is transmitted to the central node for aggregation to obtain global aggregated data.

[0106] The target data corresponding to the target key is the data item corresponding to the target key on the local node, including the data to be aggregated and / or the local aggregated data. In the case where the local node has data cache, these data include the previous data to be aggregated and / or the local aggregated data after aggregation in addition to the current aggregated local aggregated data.

[0107] The global aggregate data is the data item generated after the target data is aggregated on the central node. It is the final result of the two-stage aggregation. The global aggregation result is obtained by further data aggregation based on the target data transmitted by at least one local node. The global aggregation result has a higher degree of aggregation than the multiple data to be aggregated and the local aggregation results on the local node. The global aggregation result includes the aggregation key and the aggregation function value corresponding to the aggregation key, and is expressed in the form of key-value pair data.

[0108] Transmit to the central node for aggregation to obtain global aggregate data. One optional way of global aggregation on the central node is: perform aggregation function calculation on the target data to obtain global aggregate data. Another optional way is: use a hash table to aggregate the target data to obtain global aggregate data. Another optional way is: a combination of the above two ways, which is not limited here.

[0109] Exemplarily, all data items of the target key (merchant number 'user1') are taken out from the hash bucket and transmitted to two central nodes. All data items of the target key (merchant number 'user1') from four local nodes are integrated on the two central nodes, and the aggregation function is calculated to obtain global aggregate data: seller number 'user1', number of orders 100,000.

[0110] In the embodiments of the present specification, by introducing a key heat weight mechanism on the local node, the frequency of key aggregation in a period of time is reflected, and the target key is determined from at least one key according to the heat weight of at least one key corresponding to each data in the local node, and the target keys with high and low cardinality are dynamically and flexibly distinguished. The target data corresponding to the target key is transmitted to the central node for global aggregation, global aggregated data is obtained, and a suitable aggregation strategy is executed. This not only improves the efficiency of local aggregation and reduces unnecessary data transmission overhead, but also significantly improves the speed and efficiency of data aggregation in distributed systems.

[0111] Another problem that is easily caused in two-stage aggregation is that a large number of data items are accumulated on the local node, which seriously occupies the cache space of the local node, causing a sharp drop in data access efficiency and affecting the effect of local aggregation.

[0112] In view of the above problem, in an optional embodiment of the present specification, the local node includes a cache space;

[0113] After step 402, the following specific steps are also included: writing a plurality of data to be aggregated into the cache space;

[0114] Correspondingly, step 406 includes the following specific steps: when the size of the cache space reaches a preset threshold, determining a target key from at least one key according to a heat weight of at least one key corresponding to each data in the cache space;

[0115] Correspondingly, step 408 includes the following specific steps: taking the target data corresponding to the target key from the cache space, and transmitting it to the central node for aggregation to obtain global aggregated data.

[0116] The cache space is a memory area on the local node for temporary storage of data. The cache space can be a hash table, hash bucket or other data structure used to store data to be aggregated and local aggregation results. The main purpose of the cache space is to reduce the frequent read and write operations of data and improve the efficiency of data processing. The cache space stores the data items that currently need to be aggregated on the local node, which can be the data to be aggregated directly read from the data source or the local aggregated data after local aggregation. The cache space is designed to reduce the amount of data transmitted in the network and improve the overall performance of the system. The cache space can correspond to a memory address range on the local node. The size of the cache space is the amount of data currently stored in the cache space, which changes dynamically. When the size of the cache space reaches the preset threshold, the cache elimination mechanism will be triggered to determine the target key, and then the target data corresponding to the target key will be transmitted to the central node for global aggregation.

[0117] The preset threshold is a critical value set for the cache space. When the amount of data in the cache space reaches or exceeds this critical value, the cache elimination mechanism will be triggered. Reasonable setting of the preset threshold can balance memory usage and data processing efficiency. Generally, the threshold size is set according to the size of the processor cache. For example, the preset threshold size is M, which generally does not exceed the size of the L3 cache (L3 cache), that is, less than 16MB.

[0118] An optional way of writing the multiple data to be aggregated into the cache space is: writing the multiple data to be aggregated into a cache subspace of the cache space.

[0119] The space size of the cache space reaches a preset threshold. One optional way is: the space size of each cache subspace in the cache space reaches the preset threshold. Another optional way is: the space size of any cache subspace in the cache space reaches the preset threshold. Another optional way is: the total space size of any cache subspace in the cache space reaches the preset threshold. There is no limitation here.

[0120] An optional way of taking the target data corresponding to the target key out of the cache space is to take the target data corresponding to the target key out of the cache subspace of the cache space.

[0121] For example, a hash table is created on the local node to store the data to be aggregated. 1,000,000 order data are written into the hash bucket of the hash table. The merchant number of each order data is used as the hash key, and the order number, order amount and number of orders are used as the value. The preset threshold of the hash table size is 16MB. When the amount of data in the hash table reaches 16MB and is about to overflow, the cache elimination mechanism is triggered: according to the total number of aggregation steps corresponding to 20 keys "seller number" ('user1', 'user2', ... 'user20') of each data item in the local node (data to be aggregated and local aggregated data): seller number 'user1', total number of aggregation steps 30,000; seller number 'user2', total number of aggregation steps 50,000; ... seller number 'user20', total number of aggregation steps 45,000. Determine the target key from the 20 keys from low to high: seller number 'user1'. All data items of the target key (merchant number 'user1') are taken out from the hash bucket and transmitted to the two central nodes. All data items of the target key (merchant number 'user1') from the four local nodes are integrated on the two central nodes, and the aggregation function is calculated to obtain the global aggregate data: seller number 'user1', number of orders 100000.

[0122] In the embodiments of the present specification, the cache mechanism is optimized in a fine-grained manner. Based on the introduction of a heat weight mechanism, when the size of the cache space reaches a preset threshold, the cache space is cleaned according to the heat weight, and data items corresponding to aggregation keys with high heat weights are intelligently retained, thereby improving cache utilization and reducing unnecessary memory accesses, further improving the benefits of local aggregation.

[0123] In an optional embodiment of the present specification, the cache space is pre-divided into a plurality of cache subspaces, and any cache subspace has a corresponding space capacity;

[0124] Writing multiple data to be aggregated into the cache space on the local node includes the following specific steps: writing the multiple data to be aggregated into the target cache subspace where the first key is located; and expanding the target cache subspace when the space size of the target cache subspace reaches the corresponding space capacity.

[0125] The cache subspace is a plurality of sub-areas divided from the cache space. The cache subspace can be a hash bucket or other data structure, which is used to store the data to be aggregated or the local aggregation result corresponding to a specific aggregation key. The design of the cache subspace is to better manage and optimize the use of the cache space and improve the efficiency of data processing. In a distributed system, a cache subspace can be used to store data items corresponding to at least one aggregation key, and the data items corresponding to the same aggregation key are generally stored in one cache subspace. Each cache subspace has an independent space capacity, which can be dynamically adjusted according to the characteristics and requirements of the data. By allocating data items to different cache subspaces, data can be managed more effectively to avoid excessive occupation of a single cache space. The cache subspace can correspond to a memory address range on the local node. Among them, each cache subspace records the aggregation key, aggregation function value and heat weight of the data item. The space size of the cache subspace is the amount of data currently stored in the cache subspace, which changes dynamically. When the space size of the cache space reaches the space capacity, the cache expansion mechanism will be triggered to increase the space capacity. The target cache subspace is a cache subspace used to store the first key, and the data items corresponding to the first key are generally stored in the target cache subspace. The space capacity of the cache subspace is the upper limit of the amount of data stored in the cache subspace. This capacity is determined during pre-division and can be adjusted according to storage conditions.

[0126] For example, write 1,000,000 order data into 8 hash buckets where each aggregation key (merchant number) of the hash table is located. The merchant number of each order data is used as the hash key, and the number of orders and the heat weight are used as the value. The preset threshold of the hash table size is 16MB, and the capacity of any hash bucket is 1MB. When the amount of data in the hash table reaches 8MB and is about to overflow, the expansion mechanism is triggered: the capacity of the hash bucket is multiplied by 2.

[0127] In the embodiments of the present specification, by dividing the cache space into multiple cache subspaces and setting an independent space capacity for each cache subspace, data storage can be managed more finely. When the space size of the target cache subspace reaches its space capacity, the capacity is dynamically expanded to ensure efficient storage and processing of data. This not only avoids excessive occupancy of a single cache space and improves cache utilization, but also reduces frequent data memory access operations, significantly improving the efficiency of local aggregation and the overall performance of the system.

[0128] In an optional embodiment of the present specification, the heat weight of the first key includes the total number of aggregation steps and the number of interval steps from the last aggregation;

[0129] In step 404, the heat weight of the first key is updated, including the following specific steps: if the first key appears for the first time, the total aggregation steps of the first key and the interval steps of the first key are initialized; if the first key does not appear for the first time, the total aggregation steps of the first key are increased by one, and the interval steps of the first key are updated to the current aggregation steps, wherein the current aggregation step is increased by one after the data to be aggregated corresponding to any one of the at least one key is aggregated.

[0130] The total number of aggregation steps is the cumulative number of times the aggregation operation is executed for the first key within a period of time. It is an important indicator to measure the activity of the aggregation key and reflects the aggregation frequency of the aggregation key data item. The total number of aggregation steps can be used to identify which aggregation keys have high cardinality and which have low cardinality. The interval step is the number of intervals between the last execution of the aggregation operation and the current execution of the aggregation operation for the first key. It is an important indicator to measure the active interval between two aggregations of the aggregation key. The interval step reflects which aggregation keys are continuously active and which are intermittently active. The current aggregation step is the total number of steps executed by the global aggregation operation. The current aggregation step is a global counter used to track the progress of aggregation operations in the entire system.

[0131] It should be noted that in order to efficiently manage and compare heat weights, the total number of aggregation steps and the number of interval steps can be compressed into a long integer value heat weight. The heat weight is calculated as shown in Formula 2:

[0132] Heat weight = ((long) total number of aggregation steps << 32) | interval step number formula 2

[0133] Among them, formula 2 is used to merge two 32-bit integer values ​​(total number of aggregation steps and interval number of steps) into a 64-bit long integer value. This ensures that the greater the total number of aggregation steps (i.e., more repetitions) and the greater the interval number of steps (i.e., most recently updated), the greater the heat weight.

[0134] Optionally, the maximum value max_count and the minimum value min_count of the total number of steps of the aggregation are maintained in real time. These values ​​will be reset each time a cache cleanup operation is performed.

[0135] It should be noted that in the embodiments of this specification, by taking into account both the total number of aggregation steps and the number of update steps, the aggregation column of the hotspot value can be prevented from occupying the cache for a long time. Even if a value is repeated many times, if the same key value has been fully aggregated, or is distributed far apart and dispersed in the table, the heat weight determined by comprehensively considering the total number of aggregation steps and the number of update steps can ensure that these values ​​will not occupy the cache space for a long time, thereby achieving a better local aggregation effect.

[0136] For example, if the first key appears for the first time, the total aggregation steps of the first key are initialized: count=1, and the interval steps of the first key are initialized; last_update_step=0. If the first key does not appear for the first time, the total aggregation steps of the first key are increased by one: count=count+1, and the interval steps of the first key are updated to the current aggregation steps: last_update_step=current_step, where the current aggregation step current_step is the full step counter, and any aggregation operation of the aggregation key will trigger current_step+1.

[0137] In the embodiments of the present specification, the total number of aggregation steps of the first key can reflect the activity level of the aggregation key, while the interval number of steps of the first key can measure the activity interval of the aggregation key. The combination of the two not only helps to identify high-cardinality and low-cardinality aggregation keys, but also ensures that even for data items with many repetitions but scattered distribution, resources will not be wasted due to long-term cache occupation.

[0138] For a certain aggregation key, it is generally believed that if the degree of local aggregation exceeds G (see formula 1), two-stage aggregation will have good performance benefits. Otherwise, it is better to directly transmit to the central node for global aggregation. The cache cleanup strategy can be used:

[0139] In an optional embodiment of the present specification, a cache cleaning strategy is provided. The heat weight includes the total number of aggregation steps;

[0140] When the size of the cache space reaches a preset threshold, a target key is determined from at least one key according to the heat weight of at least one key corresponding to each data in the cache space, including the following specific steps: when the size of the cache space reaches a preset threshold, a quantile of the total number of aggregated steps is determined according to a preset aggregation degree; based on the quantile of the total number of aggregated steps, the total number of aggregated steps of at least one key in the cache space is traversed to determine the target key from at least one key.

[0141] The preset aggregation degree is a threshold set to measure the aggregation effect of the two-stage aggregation. It is determined based on the amount of data before and after aggregation, as shown in Formula 1. After the data is aggregated on the local node, when the aggregation degree exceeds this value, the two-stage aggregation is more advantageous than direct global aggregation. The preset aggregation degree is used to evaluate the effectiveness of local aggregation. If the local aggregation degree of an aggregation key exceeds the preset aggregation degree, it means that the aggregation key has been sufficiently aggregated on the local node, which can reduce the amount of data transmission and improve the performance of the overall system. On the contrary, if the aggregation degree does not reach the preset aggregation degree, it is more efficient to directly transfer the data to the central node for global aggregation. For example, the preset aggregation degree G is set to 6, which means that if the aggregation degree (i.e., the total number of aggregation steps) of an aggregation key on the local node exceeds 6, local aggregation is considered to be beneficial. If the total number of aggregation steps of a certain aggregation key is 7, the data items of the aggregation key can be aggregated on the local node and then transferred to the central node for global aggregation. If the total number of aggregation steps is 5, it may be more appropriate to directly transfer the data to the central node for global aggregation.

[0142] The total number of aggregation steps quantile is the value at a certain percentile in the total number of aggregation steps of at least one key. The total number of aggregation steps quantile is used to determine which aggregation keys have a low total number of aggregation steps, so as to decide whether the data items corresponding to these keys need to be cleared. The total number of aggregation steps quantile determines which aggregation key data items should be cleared and directly transmitted to the central node for global aggregation in the cache cleanup strategy. By calculating the quantile of the total number of aggregation steps, those aggregation keys with low aggregation degree can be dynamically identified to avoid these keys occupying cache space for a long time, thereby improving the cache utilization and the overall performance of the system. For example, the total number of aggregation steps of the aggregation keys are: 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, and the preset aggregation degree G is set to 6. It is necessary to calculate (G-1) / G as the total number of aggregation steps quantile, that is, the quantile of 83%, 0.83×10=8.3, and take the value of the 9th position, that is, the total number of aggregation steps quantile is 90. The data items corresponding to the aggregation keys (10, 20, 30, 40, 50, 60, 70, 80) with a total aggregation step number lower than 90 will be cleared and directly transferred to the central node for global aggregation.

[0143] In the embodiments of this specification, by introducing preset aggregation degree and total aggregation step quantiles, data items in the cache space are dynamically managed to ensure that data corresponding to keys with high aggregation degree are processed preferentially on the local node, while data corresponding to keys with low aggregation degree are directly transmitted to the central node, thereby optimizing cache utilization, reducing unnecessary data transmission, and significantly improving the overall performance of the system and data processing efficiency.

[0144] According to the preset aggregation degree, the total number of aggregation steps is determined. If you need to find the exact total number of aggregation steps, you often have to use sorting methods, which has a complexity of O(N^2) and is not conducive to local aggregation performance. Therefore, you can use estimation to determine the total number of aggregation steps.

[0145] In an optional embodiment of the present specification, determining the quantile of the total number of aggregated steps according to a preset aggregation degree includes the following specific steps: dividing the numerical range of the total number of aggregated steps of at least one key in the cache space to obtain a plurality of numerical subranges;

[0146] Count the number of total aggregation steps in multiple numerical sub-ranges, and estimate the target numerical sub-range where the quantile of the total aggregation steps is located based on the number and the preset aggregation degree; determine the median of the target numerical sub-range as the quantile of the total aggregation steps.

[0147] The numerical range of the total number of steps in the aggregation is the range between the maximum and minimum values ​​of the total number of steps in the aggregation of at least one key. The numerical range of the total number of steps in the aggregation is used to determine the distribution of the data and help determine how to divide the numerical sub-ranges so as to more efficiently estimate the quantiles of the total number of steps in the aggregation. By understanding the numerical range, interval division can be performed more accurately, thereby reducing the computational complexity. For example, the total number of steps in the aggregation of all aggregation keys in the cache space are: 10, 20, 30, 40, 50, 60, 70, 80, 90, 100. Then the numerical range of the total number of steps in the aggregation is from 10 to 100.

[0148] The numerical subrange is to divide the numerical range of the total number of aggregation steps into multiple smaller intervals, each of which is called a numerical subrange. The division of numerical subranges is to simplify the calculation process of quantiles. By dividing the data into multiple intervals, the number of data points in each interval can be quickly counted, thereby estimating the interval in which the quantile is located. This method can reduce the computational complexity from O(N^2) to O(N). For example, the numerical range of the total number of aggregation steps is from 10 to 100, which is divided into 10 numerical subranges: numerical subrange 1: [10, 19]; numerical subrange 2: [20, 29]; numerical subrange 3: [30, 39]; numerical subrange 4: [40, 49]; numerical subrange 5: [50, 59]; numerical subrange 6: [60, 69]; numerical subrange 7: [70, 79]; numerical subrange 8: [80, 89]; numerical subrange 9: [90, 99]; numerical subrange 10: [100, 109].

[0149] By counting the number of total aggregation steps in each numerical sub-range, the interval where the quantile of the total aggregation steps is located can be quickly determined. For example, the total aggregation steps are: 10, 20, 30, 40, 50, 60, 70, 80, 90, 100. Put these values ​​into the above 10 sub-ranges and count the number of each sub-range: numerical sub-range 1: [10, 19] - 1; numerical sub-range 2: [20, 29] - 1; numerical sub-range 3: [30, 39] - 1; numerical sub-range 4: [40, 49] - 1; numerical sub-range 5: [50, 59] - 1; numerical sub-range 6: [60, 69] - 1; numerical sub-range 7: [70, 79] - 1; numerical sub-range 8: [80, 89] - 1; numerical sub-range 9: [90, 99] - 1; numerical sub-range 10: [100, 109] - 1.

[0150] The target numerical subrange is a numerical subrange containing quantiles determined based on the preset degree of aggregation and the number of total aggregation steps. By accumulating the number of total aggregation steps in each numerical subrange, the interval in which the cumulative number reaches the quantile requirement can be determined, and this interval is the target numerical subrange. For example, if the preset degree of aggregation G is 6, it is necessary to calculate (G-1) / G as the quantile of the total number of aggregation steps, that is, the 83% quantile, and the number in each subrange is 1. Accumulate the number until the 83% quantile is reached: the cumulative number of the first 8 subranges is 8, accounting for 80% of the total, and the cumulative number of the 9th subrange is 9, accounting for 90% of the total. Therefore, the 9th subrange [90, 99] is the target numerical subrange.

[0151] The median of the target value subrange refers to the total number of aggregation steps in the middle of the target value subrange, which is used to estimate the specific value of the quantile. By taking the median of the target value subrange, the value of the quantile can be approximately determined, thus avoiding complex sorting operations. For example, the target value subrange is [90, 99], and there is only one value 90 in this subrange. Therefore, the median of the target value subrange is 90. If there are multiple values ​​in the subrange, such as [90, 91, 92, 93, 94, 95, 96, 97, 98, 99], the median is (94+95) / 2=94.5.

[0152] In the embodiments of the present specification, the value of the total number of aggregated steps quantile is estimated in a manner with a complexity of O(N), thereby reducing the performance consumption of determining the total number of aggregated steps quantile. This allows for effective management of cache space, ensuring that high-aggregation keys are processed preferentially on local nodes, while low-aggregation keys are directly transmitted to central nodes, thereby improving the overall performance of the system and data processing efficiency.

[0153] In an optional embodiment of the present specification, a cache cleaning strategy is provided. The heat weight includes the total number of aggregation steps;

[0154] When the size of the cache space reaches a preset threshold, a target key is determined from at least one key according to the heat weight of at least one key corresponding to each data in the cache space, including the following specific steps: when the size of the cache space reaches a preset threshold, according to a preset aggregation degree, the total number of aggregation steps of at least one key in the cache space is traversed, and a target key is determined from at least one key.

[0155] It should be noted that for keys whose total aggregation steps are lower than the preset aggregation degree G, they are determined as target keys and cleared. This is to prevent the (G-1) / G quantile from having a low cardinality when the distribution of the aggregated columns is extremely uneven, making it difficult to clear the low cardinality buckets.

[0156] For example, the total aggregation steps of all aggregation keys in the cache space are 10, 20, 30, 40, 50, 60, 70, 80, 90, and 100. The preset aggregation degree G is set to 6. After traversing these total aggregation steps, it is found that the keys with total aggregation steps of 10, 20, 30, 40, and 50 are lower than the preset aggregation degree G, so the data items corresponding to these keys will be cleared and directly transmitted to the central node for global aggregation.

[0157] In the embodiments of this specification, by introducing a preset aggregation degree, the data items in the cache space are dynamically managed to ensure that the data items corresponding to the keys with low aggregation degrees are cleared in time and transmitted to the central node for global aggregation, thereby avoiding the cache space being occupied by low-cardinality keys for a long time, improving the cache utilization and the overall performance of the system, and at the same time, reducing unnecessary data transmission, optimizing the efficiency of local aggregation, and significantly improving the data processing capabilities of the distributed system.

[0158] In an optional embodiment of the present specification, a cache cleaning strategy is provided. The heat weight includes the interval steps from the last aggregation;

[0159] When the size of the cache space reaches a preset threshold, a target key is determined from at least one key according to the heat weight of at least one key corresponding to each data in the cache space, including the following specific steps: when the size of the cache space reaches a preset threshold, the interval steps of at least one key are traversed according to a preset interval step threshold, and a target key is determined from at least one key.

[0160] The preset interval step threshold is a threshold for measuring the active interval of the aggregation key. If the interval step of an aggregation key exceeds the preset interval step threshold, it means that the aggregation key has not been aggregated for a long time, which may be because the same key value has been locally aggregated or the key value is distributed far away in the table. In this case, the data items corresponding to these keys can be cleared to free up cache space.

[0161] It should be noted that for keys whose interval steps exceed the preset interval step threshold, they are determined as target keys and cleared. This can prevent keys with large cardinality but the same key value that have all completed local aggregation, or keys with key values ​​distributed far apart in the table, from occupying local aggregation memory space for a long time.

[0162] For example, the total aggregation steps of all aggregation keys in the cache space are: 10, 20, 30, 40, 50, 60, 70, 80, 90, 100. The preset interval step threshold is set to the total number of updates between 3 rounds of cache cleaning, and the total number of updates between each round of cache cleaning is 30. Traversing these interval steps, it is determined that the keys with interval steps of 90 and 100 exceed the preset interval step threshold (30×3=90), so the data items corresponding to these keys will be cleared and directly transmitted to the central node for global aggregation.

[0163] In the embodiments of the present specification, a preset interval step threshold is introduced to dynamically manage data items in the cache space, ensuring that data items corresponding to keys that have been inactive for a long time are promptly cleared and transmitted to the central node for global aggregation, thereby avoiding the cache space being occupied by inactive keys for a long time, improving cache utilization and the overall performance of the system, and at the same time, optimizing the efficiency of local aggregation, reducing unnecessary data transmission, and significantly improving the data processing capability and response speed of the distributed system.

[0164] In an optional embodiment of the present specification, after step 402, the following specific steps are further included: determining whether the frequency of the first key exceeds a preset threshold; if the frequency of the first key does not exceed the preset threshold, transmitting the plurality of data to be aggregated to the central node for aggregation to obtain global aggregated data;

[0165] Correspondingly, step 404 includes the following specific steps: when the frequency of the first key exceeds a preset threshold, aggregating multiple data to be aggregated, obtaining local aggregated data, and updating the heat weight of the first key.

[0166] The frequency of the first key is the number of times the data item corresponding to the first key appears within a preset time period. This frequency reflects the activity of the first key in the data stream. The frequency of the first key is used to evaluate the data volume and generation speed of the first key, the aggregation key. By monitoring this frequency, you can dynamically decide whether to use a one-stage aggregation or a two-stage aggregation strategy to process the data items corresponding to the first key to reduce network transmission and resource overhead of central nodes. Keys with high frequencies usually mean large data volumes and fast generation speeds, which are suitable for local aggregation first and then global aggregation; while keys with low frequencies may be more suitable for direct global aggregation.

[0167] The preset threshold is used to determine whether the frequency of the first key reaches the local aggregation standard. It can be set according to specific application scenarios and requirements. The preset threshold is a parameter that distinguishes high cardinality (frequency) from low cardinality (frequency) data.

[0168] Exemplarily, the preset time period is 1 hour, and the preset threshold is 10 times. Within 1 hour, the seller number "user1" appears 20,000 times, the seller number "user3" appears 7 times, the seller number "user4" appears 300 times, and the seller number "user9" appears 20 times. For the seller number "user1", its frequency is 20,000 times / hour, which exceeds the preset threshold of 10 times / hour. For the seller number "user3", its frequency is 7 times / hour, which does not exceed the preset threshold of 10 times / hour. For the seller number "user4", its frequency is 300 times / hour, which exceeds the preset threshold of 10 times / hour. For the seller number "user9", its frequency is 20 times / hour, which exceeds the preset threshold of 10 times / hour. For the seller number "user3", all data items of the seller number "user3" are transmitted to two central nodes, and the two central nodes perform aggregation function calculations on all data items of the seller number "user3" from the four local nodes to obtain global aggregate data: seller number 'user1', number of orders 300. For seller IDs "user1", "user4" and "user9", a hash table is created on the local node to store local aggregation results. The 1,000,000 order data are traversed, and the first key "seller ID" of each data item is determined as a hash value, and it is assigned to the corresponding bucket of the hash table. Aggregation function calculation is performed in each bucket to obtain local aggregate data: seller ID 'user1', number of orders 20,000; seller ID 'user4', number of orders 300; seller ID 'user9', number of orders 20.

[0169] In the embodiments of this specification, by determining whether the frequency of the first key exceeds a preset threshold, a suitable aggregation strategy is flexibly selected to effectively reduce the amount of data transmitted over the network, reduce the load on the central node, and improve the performance and efficiency of the overall system.

[0170] In an optional embodiment of the present specification, the (1 / G) keys remaining after the above three cache cleaning strategies are retained and participate in the next round of aggregation process until the size of the cache space reaches the preset threshold again, triggering a new round of cache cleaning.

[0171] against Figure 4 Instructions for use of the embodiment of the invention, Figure 5 A schematic diagram of a data aggregation method provided by an embodiment of this specification is shown, such as Figure 5 As shown: multiple data to be aggregated in a batch are locally aggregated on two local nodes respectively to obtain two local aggregated data: local aggregated data 1 is shown in Table 13:

[0172]

[0173] Table 13

[0174] The local aggregate data 2 is shown in Table 14:

[0175]

[0176] Table 14

[0177] In the process of continuous local aggregation, local aggregation 1 and local aggregation 2 start to use heat weight to clean up the cache when the hash table space is full. Assume that the aggregation degree G=6, the preset interval step threshold is 20000, and the current aggregation step is 50000, which means that the keys with an interval step of less than 30000 from the last aggregation will be cleared.

[0178] For local aggregation 1: The total number of aggregation steps of 'user3' ~ 'user7' is lower than the 83% quantile of the total number of aggregation steps, which is a low cardinality key. The interval number of steps of 'user9' exceeds the limit, which is a high cardinality hotspot value, but the update time is too long.

[0179] For local aggregation 2: The total number of aggregation steps of 'user9' ~ 'user7' is lower than the 83% total number of aggregation steps quantile, which belongs to the low cardinality key.

[0180] The interval step number of 'user3' exceeds the limit, which is a high cardinality hotspot value, but the update time is too long.

[0181] The hash buckets corresponding to these keys are cleared and uploaded to the central node for global aggregation.

[0182] Two global aggregate data are obtained: Global aggregate data 1 is shown in Table 15:

[0183]

[0184] Table 15

[0185] The global aggregate data 2 is shown in Table 16:

[0186]

[0187] Table 16

[0188] The next round of aggregation process is carried out until all local aggregation processes are completed.

[0189] like Figure 5 As shown, a two-stage aggregation method combining dynamic aggregation strategy and hash table management optimization in the embodiment of this specification. Different from the existing direct global aggregation and two-stage aggregation methods, the embodiment of this specification dynamically adjusts the aggregation strategy through cardinality judgment: keys with high cardinality (frequency) are preferentially aggregated locally, while keys with low cardinality (frequency) are directly uploaded to the global aggregation. In this way, local aggregation is more efficient and reduces invalid calculations and transmissions. At the same time, the embodiment of this specification manages the local aggregation results through the cache elimination mechanism of the hash table. Whenever the hash table space is full, the hash buckets corresponding to some keys are cleaned up according to the heat weights such as the number of occurrences and the last update time, further improving the benefits of local aggregation. Finally, based on the frequency and time factors of the aggregation key, a heat weight mechanism is introduced to improve the utilization efficiency of the memory space of the local aggregation hash table, dynamically manage hotspot aggregation keys, prevent the cache from being occupied for a long time, and significantly improve the coping ability of the distributed system. Moreover, it is more adaptable and can be dynamically adjusted in different aggregation scenarios, including high aggregation, low aggregation, and high and low mixed situations.

[0190] See also Figure 6 , Figure 6 A flowchart of another data aggregation method provided by an embodiment of the present specification is shown. The method is applied to a central node of a distributed system, and the distributed system also includes a local node, and includes the following specific steps:

[0191] Step 602: Receive target data corresponding to a target key transmitted by a local node, wherein the step of determining the target key at the local node includes: obtaining multiple data to be aggregated, aggregating the multiple data to be aggregated, obtaining local aggregated data, and updating the heat weight of the first key, and determining the target key from at least one key according to the heat weight of at least one key corresponding to each data in the local node, wherein any data to be aggregated corresponds to the first key, and at least one key includes the first key.

[0192] Step 604: Aggregate the target data to obtain global aggregated data.

[0193] Aggregate the target data to obtain global aggregate data. One optional way is to calculate the aggregation function on the target data to obtain global aggregate data. Another optional way is to use a hash table to aggregate the target data to obtain global aggregate data. Another optional way is to combine the above two ways, which are not limited here.

[0194] In the embodiments of the present specification, by introducing a key heat weight mechanism on the local node, the frequency of key aggregation in a period of time is reflected, and the target key is determined from at least one key according to the heat weight of at least one key corresponding to each data in the local node, and the target keys with high and low cardinality are dynamically and flexibly distinguished. The target data corresponding to the target key is transmitted to the central node, and global aggregation is performed at the central node to obtain global aggregated data, and execute a suitable aggregation strategy, which not only improves the efficiency of local aggregation and reduces unnecessary data transmission overhead, but also significantly improves the speed and efficiency of data aggregation in distributed systems.

[0195] It should be noted that Figure 6 The technical solution for data aggregation in Figure 4 The technical scheme of the data aggregation method belongs to the same concept. Figure 6 For details not described in detail in the technical solution of data aggregation, please refer to the above Figure 4 Description of the technical solution of the data aggregation method in .

[0196] The following combination Figure 7 , taking the application of the data aggregation method provided in this specification to IoT data as an example, the data aggregation method is further explained. Figure 7 A processing flow chart of a data aggregation method applied to IoT data provided by an embodiment of the present specification is shown, which is applied to a local node of a distributed system of the IoT, and the distributed system of the IoT also includes a central node, including the following specific steps:

[0197] Step 702: Acquire multiple IoT data collected by the sensor device, wherein any IoT data corresponds to the first key.

[0198] Step 704: When the frequency of the first key does not exceed a preset threshold, multiple IoT data are transmitted to a central node for aggregation to obtain global aggregated data.

[0199] Step 706: When the frequency of the first key exceeds a preset threshold, write the plurality of IoT data into the target hash bucket where the first key is located, and when the space size of the target hash bucket reaches the corresponding space capacity, expand the target hash bucket.

[0200] Step 708: Aggregate multiple IoT data to obtain local aggregated data, and update the total aggregation steps of the first key and the interval steps from the last aggregation, wherein updating the total aggregation steps of the first key and the interval steps from the last aggregation includes: if the first key appears for the first time, initializing the total aggregation steps of the first key and the interval steps of the first key; if the first key does not appear for the first time, adding one to the total aggregation steps of the first key, and updating the interval steps of the first key to the current aggregation steps, wherein the current aggregation step is added by one after aggregating the IoT data corresponding to any one of the at least one key.

[0201] Step 710: When the space size of the hash table space reaches a preset threshold, determine the total aggregation step quantile according to the preset aggregation degree, traverse the total aggregation steps of at least one key in the hash table space based on the total aggregation step quantile, and determine a first target key from at least one key; according to the preset aggregation degree, traverse the total aggregation steps of at least one key in the hash table space, and determine a second target key from at least one key; according to the preset interval step threshold, traverse the interval steps of at least one key, and determine a third target key from at least one key.

[0202] Step 712: The target data corresponding to the first target key, the second target key and the third target key are taken out from the hash bucket corresponding to the hash table space, and transmitted to the central node for aggregation to obtain global aggregated data.

[0203] Step 714: Receive the IoT device operation instruction fed back by the central node, wherein the IoT device operation instruction is generated based on the global aggregated data.

[0204] In the embodiments of this specification, the following effects are achieved: 1. Dynamic aggregation strategy: A dynamic mixed aggregation mode is adopted, and the aggregation strategy can be automatically adjusted according to the aggregation degree of IoT data. This flexibility makes the system more efficient in processing low-aggregation data and hotspot data, especially in the IoT environment, where there are many data sources and various types, and the ability to dynamically adjust the aggregation strategy is particularly important. 2. Cache management: By introducing the total number of aggregation steps and the interval number of steps from the last aggregation as the update mechanism of the heat weight, the cache cleanup strategy can be determined according to the frequency of IoT data and its update time. This mechanism ensures that frequently accessed data can be retained in the cache first, which not only improves the efficiency of cache usage, but also significantly reduces unnecessary memory access operations, which is particularly important for resource-constrained IoT devices. 3. Hotspot data processing: For the common high-cardinality hotspot aggregation keys in the IoT, dynamic management can be achieved to avoid the problem of hotspot data occupying cache resources for a long time. This greatly enhances the system's responsiveness and processing efficiency in the face of a large number of concurrent requests or data surges. 4. Adaptability: The aggregation strategy can be dynamically adjusted according to the different aggregation characteristics of IoT data (such as high aggregation, low aggregation, or a mixture of the two). This flexibility not only helps improve the efficiency of data processing, but also better supports the diverse needs of IoT applications, such as real-time monitoring and data analysis.

[0205] Corresponding to the above method embodiment, this specification also provides a distributed system embodiment. Figure 8 FIG. 1 shows a schematic diagram of a distributed system provided by an embodiment of this specification. Figure 8 As shown, the distributed system 800 includes a local node 810 and a central node 820;

[0206] The local node 810 is used to obtain multiple data to be aggregated, aggregate the multiple data to be aggregated, obtain local aggregated data, and update the heat weight of the first key, determine the target key from the at least one key according to the heat weight of at least one key corresponding to each data in the local node 810, and transmit the target data corresponding to the target key to the central node 820, wherein any data to be aggregated corresponds to the first key, and at least one key includes the first key;

[0207] The central node 820 is used to receive the target data corresponding to the target key transmitted by the local node 810, aggregate the target data, and obtain global aggregate data.

[0208] In the embodiments of this specification, for the "one-stage aggregation" strategy and the "two-stage aggregation" strategy on the distributed system, a key heat weight mechanism is introduced on the local node to reflect the frequency of key aggregation in a period of time. According to the heat weight of at least one key corresponding to each data in the local node, the target key is determined from at least one key, and the target keys with high and low cardinality are dynamically and flexibly distinguished. The target data corresponding to the target key is transmitted to the central node, and global aggregation is performed at the central node to obtain global aggregated data, and an appropriate aggregation strategy is executed. This not only improves the efficiency of local aggregation and reduces unnecessary data transmission overhead, but also significantly improves the speed and efficiency of data aggregation in distributed systems.

[0209] The above is a schematic scheme of a distributed system of this embodiment. It should be noted that the technical scheme of the distributed system and the technical scheme of the above data aggregation method belong to the same concept, and the details not described in detail in the technical scheme of the distributed system can be referred to the description of the technical scheme of the above data aggregation method.

[0210] Corresponding to the above method embodiment, this specification also provides a data aggregation device embodiment, Fig. 9 FIG. 2 shows a schematic diagram of the structure of a data aggregation device provided by an embodiment of the present specification. Fig. 9 As shown, the device is applied to a local node of a distributed system, and the distributed system also includes a central node, including:

[0211] The acquisition module 902 is configured to acquire a plurality of data to be aggregated, wherein any data to be aggregated corresponds to a first key;

[0212] The local aggregation module 904 is configured to aggregate the multiple data to be aggregated, obtain local aggregated data, and update the heat weight of the first key;

[0213] A determination module 906 is configured to determine a target key from at least one key according to a heat weight of at least one key corresponding to each data in the local node, wherein the at least one key includes a first key;

[0214] The first global aggregation module 908 is configured to transmit the target data corresponding to the target key to the central node for aggregation to obtain global aggregated data.

[0215] Optionally, the local node includes a cache space; the device further includes: a writing module configured to write the plurality of to-be-aggregated data into the cache space;

[0216] Correspondingly, the determination module 906 is further configured to: determine the target key from the at least one key according to the heat weight of the at least one key corresponding to each data in the cache space when the space size of the cache space reaches a preset threshold;

[0217] Correspondingly, the first global aggregation module 908 is further configured to: take out the target data corresponding to the target key from the cache space, and transmit it to the central node for aggregation to obtain global aggregated data.

[0218] Optionally, the cache space is pre-divided into a plurality of cache subspaces, and any cache subspace has a corresponding space capacity;

[0219] Correspondingly, the writing module is further configured to: write the multiple data to be aggregated into the target cache subspace where the first key is located; and expand the target cache subspace when the space size of the target cache subspace reaches the corresponding space capacity.

[0220] Optionally, the heat weight of the first key includes the total number of aggregation steps and the number of interval steps from the last aggregation;

[0221] Correspondingly, the local aggregation module 904 is further configured as follows: if the first key appears for the first time, the total aggregation steps of the first key and the interval steps of the first key are initialized; if the first key does not appear for the first time, the total aggregation steps of the first key are increased by one, and the interval steps of the first key are updated to the current aggregation steps, wherein the current aggregation step is increased by one after the data to be aggregated corresponding to any one of the at least one keys is aggregated.

[0222] Optionally, the heat weight includes the total number of aggregation steps;

[0223] Correspondingly, the determination module 906 is further configured to: when the space size of the cache space reaches a preset threshold, determine the total number of aggregated steps quantile according to the preset aggregation degree; based on the total number of aggregated steps quantile, traverse the total number of aggregated steps of at least one key in the cache space, and determine the target key from at least one key.

[0224] Optionally, the determination module 906 is further configured to: divide the numerical range of the total number of aggregated steps of at least one key in the cache space to obtain multiple numerical sub-ranges; count the number of total aggregated steps in the multiple numerical sub-ranges, and estimate the target numerical sub-range where the quantile of the total aggregated steps is located based on the number and a preset degree of aggregation; determine the median of the target numerical sub-range as the quantile of the total aggregated steps.

[0225] Optionally, the heat weight includes the total number of aggregation steps;

[0226] Correspondingly, the determination module 906 is further configured to: when the space size of the cache space reaches a preset threshold, traverse the total number of aggregation steps of at least one key in the cache space according to a preset aggregation degree, and determine the target key from the at least one key.

[0227] Optionally, the heat weight includes the number of steps from the last aggregation;

[0228] Correspondingly, the determination module 906 is further configured to: when the size of the cache space reaches a preset threshold, traverse the interval steps of at least one key according to a preset interval step threshold, and determine the target key from the at least one key.

[0229] Optionally, the device further includes: a second global aggregation module, configured to determine whether the frequency of the first key exceeds a preset threshold; if the frequency of the first key does not exceed the preset threshold, transmit the plurality of data to be aggregated to the central node for aggregation to obtain global aggregated data;

[0230] Correspondingly, the first global aggregation module 908 is further configured to: when the frequency of the first key exceeds a preset threshold, aggregate the multiple data to be aggregated, obtain local aggregated data, and update the heat weight of the first key.

[0231] In the embodiments of the present specification, by introducing a key heat weight mechanism on the local node, the frequency of key aggregation in a period of time is reflected, and the target key is determined from at least one key according to the heat weight of at least one key corresponding to each data in the local node, and the target keys with high and low cardinality are dynamically and flexibly distinguished. The target data corresponding to the target key is transmitted to the central node for global aggregation, global aggregated data is obtained, and a suitable aggregation strategy is executed. This not only improves the efficiency of local aggregation and reduces unnecessary data transmission overhead, but also significantly improves the speed and efficiency of data aggregation in distributed systems.

[0232] The above is a schematic scheme of a data aggregation device of this embodiment. It should be noted that the technical scheme of the data aggregation device and the technical scheme of the above data aggregation method belong to the same concept, and the details not described in detail in the technical scheme of the data aggregation device can be referred to the description of the technical scheme of the above data aggregation method.

[0233] Corresponding to the above method embodiment, this specification also provides a data aggregation device embodiment, Fig.10 FIG. 2 shows a schematic diagram of the structure of another data aggregation device provided by an embodiment of the present specification. Fig.10 As shown, the device is applied to the central node of a distributed system, and the distributed system also includes a local node, including:

[0234] The receiving module 1002 is configured to receive target data corresponding to a target key transmitted by a local node, wherein the step of determining the target key at the local node includes: acquiring multiple data to be aggregated, aggregating the multiple data to be aggregated, obtaining local aggregated data, and updating the heat weight of the first key, and determining the target key from at least one key according to the heat weight of at least one key corresponding to each data in the local node, wherein any data to be aggregated corresponds to the first key, and the at least one key includes the first key;

[0235] The global aggregation module 1004 is configured to aggregate the target data to obtain global aggregated data.

[0236] In the embodiments of the present specification, by introducing a key heat weight mechanism on the local node, the frequency of key aggregation in a period of time is reflected, and the target key is determined from at least one key according to the heat weight of at least one key corresponding to each data in the local node, and the target keys with high and low cardinality are dynamically and flexibly distinguished. The target data corresponding to the target key is transmitted to the central node, and global aggregation is performed at the central node to obtain global aggregated data, and execute a suitable aggregation strategy, which not only improves the efficiency of local aggregation and reduces unnecessary data transmission overhead, but also significantly improves the speed and efficiency of data aggregation in distributed systems.

[0237] The above is a schematic scheme of another data aggregation device of this embodiment. It should be noted that the technical scheme of the data aggregation device and the technical scheme of the above another data aggregation method belong to the same concept, and the details of the technical scheme of the data aggregation device that are not described in detail can all be referred to the description of the technical scheme of the above data aggregation method.

[0238] Fig.11 The structure block diagram of a computing device provided by an embodiment of the present specification is shown. The components of the computing device 1100 include but are not limited to a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 via a bus 1130, and the database 1150 is used to store data.

[0239] The computing device 1100 also includes an access device 1140, which enables the computing device 1100 to communicate via one or more networks 1160. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 1140 may include one or more of any type of network interface (e.g., a Network Interface Controller (NIC)) of wired or wireless, such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, and a Near Field Communication (NFC).

[0240] In one embodiment of the present specification, the above components of the computing device 1100 and Fig.11 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.11 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0241] The computing device 1100 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 1100 may also be a mobile or stationary server.

[0242] The processor 1120 is used to execute the following computer program / instructions, which implement the steps of the above-mentioned data aggregation method when executed by the processor.

[0243] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data aggregation method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data aggregation method.

[0244] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned data aggregation method when executed by a processor.

[0245] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned data aggregation method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned data aggregation method.

[0246] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned data aggregation method when executed by a processor.

[0247] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above data aggregation method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above data aggregation method.

[0248] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0249] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the contents contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0250] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0251] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0252] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data aggregation method, applied to a local node of a distributed system, wherein the distributed system also includes a central node, the method comprising: Acquire multiple data to be aggregated, wherein any data to be aggregated corresponds to the first key; Aggregate the multiple data to be aggregated to obtain local aggregated data, and update the heat weight of the first key. The multiple data to be aggregated are data items that currently need to be aggregated on the local node, and the heat weight of the first key is a quantitative indicator that measures the aggregation frequency of the data items corresponding to the first key within a period of time; Determine a target key from the at least one key according to the heat weight of at least one key corresponding to each data in the local node, wherein the at least one key includes the first key, each data in the local node is a data item stored and processed in the local node, including data to be aggregated and / or the aggregated local aggregated data, and the heat weight of the at least one key is a quantitative indicator measuring the aggregation frequency of the data item corresponding to the at least one key within a period of time; The target data corresponding to the target key is transmitted to the central node for aggregation to obtain global aggregated data.

2. The method according to claim 1, wherein the local node includes a cache space; After obtaining the plurality of data to be aggregated, the method further includes: Writing the plurality of to-be-aggregated data into the cache space; The determining a target key from the at least one key according to the heat weight of the at least one key corresponding to each data in the local node includes: When the size of the cache space reaches a preset threshold, determining a target key from the at least one key according to a heat weight of at least one key corresponding to each data in the cache space; The transmitting the target data corresponding to the target key to the central node for aggregation to obtain global aggregated data includes: The target data corresponding to the target key is taken out from the cache space and transmitted to the central node for aggregation to obtain global aggregated data.

3. According to the method of claim 2, the cache space is pre-divided into a plurality of cache subspaces, and any cache subspace has a corresponding space capacity; Writing the plurality of to-be-aggregated data into the cache space on the local node includes: Writing the plurality of to-be-aggregated data into the target cache subspace where the first key is located; When the space size of the target cache subspace reaches the corresponding space capacity, the target cache subspace is expanded.

4. The method according to claim 1, wherein the heat weight of the first key includes the total number of aggregation steps and the number of interval steps from the last aggregation; The updating of the heat weight of the first key includes: If the first key appears for the first time, initialize the total number of aggregation steps of the first key and the number of interval steps of the first key; If the first key does not appear for the first time, the total aggregation step number of the first key is increased by one, and the interval step number of the first key is updated to the current aggregation step number, wherein the current aggregation step number is increased by one after aggregating the data to be aggregated corresponding to any one of the at least one key.

5. The method according to any one of claims 2-3, wherein the heat weight comprises the total number of aggregation steps; When the size of the cache space reaches a preset threshold, determining a target key from at least one key according to a heat weight of at least one key corresponding to each data in the cache space includes: When the size of the cache space reaches a preset threshold, determining a quantile of the total number of aggregated steps according to a preset aggregation degree; Based on the aggregate total step quantile, the aggregate total step count of the at least one key in the cache space is traversed to determine a target key from the at least one key.

6. The method according to claim 5, wherein determining the quantile of the total number of aggregation steps according to the preset aggregation degree comprises: Dividing the numerical range of the aggregated total number of steps of the at least one key in the cache space to obtain a plurality of numerical sub-ranges; Counting the number of total aggregated steps in the multiple value sub-ranges, and estimating the target value sub-range in which the quantile of the total aggregated steps is located according to the number and a preset aggregation degree; The median of the subrange of the target value is determined as the aggregate total step number quantile.

7. The method according to any one of claims 2-3, wherein the heat weight comprises a total number of aggregation steps; When the size of the cache space reaches a preset threshold, determining a target key from at least one key according to a heat weight of at least one key corresponding to each data in the cache space includes: When the size of the cache space reaches a preset threshold, the total number of aggregation steps of the at least one key in the cache space is traversed according to a preset aggregation degree, and a target key is determined from the at least one key.

8. The method according to any one of claims 2-3, wherein the heat weight comprises the number of steps from the last aggregation; When the size of the cache space reaches a preset threshold, determining a target key from at least one key according to a heat weight of at least one key corresponding to each data in the cache space includes: When the size of the cache space reaches a preset threshold, the interval steps of the at least one key are traversed according to a preset interval step threshold, and a target key is determined from the at least one key.

9. The method according to claim 1, after obtaining the plurality of data to be aggregated, further comprises: Determining whether the frequency of the first key exceeds a preset threshold; When the frequency of the first key does not exceed the preset threshold, the plurality of data to be aggregated are transmitted to the central node for aggregation to obtain global aggregated data; The aggregating the plurality of data to be aggregated to obtain local aggregated data, and updating the heat weight of the first key includes: When the frequency of the first key exceeds the preset threshold, the plurality of data to be aggregated are aggregated to obtain local aggregated data, and the heat weight of the first key is updated.

10. A data aggregation method, applied to a central node of a distributed system, wherein the distributed system also includes a local node, the method comprising: Receive target data corresponding to a target key transmitted by the local node, wherein the step of determining the target key at the local node includes: acquiring a plurality of data to be aggregated, aggregating the plurality of data to be aggregated, obtaining local aggregated data, and updating a heat weight of a first key, and determining a target key from the at least one key according to a heat weight of at least one key corresponding to each data in the local node, wherein any data to be aggregated corresponds to the first key, the at least one key includes the first key, the plurality of data to be aggregated are data items that currently need to be aggregated on the local node, the heat weight of the first key is a quantitative indicator measuring the aggregation frequency of the data items corresponding to the first key within a period of time, the data in the local node are data items stored and processed at the local node, including the data to be aggregated and / or the aggregated local aggregated data, and the heat weight of the at least one key is a quantitative indicator measuring the aggregation frequency of the data items corresponding to the at least one key within a period of time; The target data is aggregated to obtain global aggregated data.

11. A distributed system comprising a local node and a central node; The local node is used to obtain multiple data to be aggregated, aggregate the multiple data to be aggregated, obtain local aggregated data, and update the heat weight of the first key, determine the target key from the at least one key according to the heat weight of at least one key corresponding to each data in the local node, and transmit the target data corresponding to the target key to the central node, wherein, Any data to be aggregated corresponds to the first key, the at least one key includes the first key, the multiple data to be aggregated are data items that currently need to be aggregated on the local node, the heat weight of the first key is a quantitative indicator measuring the aggregation frequency of the data item corresponding to the first key within a period of time, each data in the local node is a data item stored and processed in the local node, including the data to be aggregated and / or the aggregated local aggregated data, and the heat weight of the at least one key is a quantitative indicator measuring the aggregation frequency of the data item corresponding to the at least one key within a period of time; The central node is used to receive the target data corresponding to the target key transmitted by the local node, aggregate the target data, and obtain global aggregate data.

12. A computing device comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer-readable storage medium storing a computer program / instruction, wherein the computer program / instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

14. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images