Data aggregation processing method, system and device

Abstract

The present disclosure relates to the technical field of databases. Provided in the embodiments are a data aggregation processing method, a system and a device. In the embodiments of the present disclosure, a distributed database system performs data aggregation processing, such that both data storage and the data aggregation processing are executed by the distributed database. Therefore, there is no need to build an additional processing system for data aggregation processing, and further there is no need to copy data and respectively send the data to the distributed database system and the additionally built processing system. In this way, occupancy of transmission resources can be reduced.

Description

Data aggregation processing method, system and device

[0001] This disclosure claims priority to Chinese patent applications with application number 202311125364.8 filed on September 1, 2023, entitled “A data processing method, apparatus and computing device cluster” and with application number 202311863788.4 filed on December 19, 2023, entitled “Method, system and device for data aggregation processing”, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The present disclosure relates to the field of database technology, and in particular to a method, system, and device for data aggregation processing. Background Art

[0003] With the rapid development of technologies such as the Internet and the Internet of Things, many businesses need to store massive amounts of "time series data." For example, this data could be temperature data periodically collected by a temperature sensor, or user browsing history. Distributed database systems can store massive amounts of time series data. In practical applications, aggregation processing is often required. Aggregation processing involves performing specified calculations (such as summation) on time series data that meets certain conditions (for example, time series data belonging to the same time period).

[0004] In related technologies, an additional processing system is built outside the distributed database system to perform aggregation processing. The management platform can send time series data to the distributed database system and the processing system respectively. The distributed database system performs conventional storage on the time series data, and the processing system performs aggregation processing on the time series data.

[0005] In the above technologies, the same data needs to be copied and sent to multiple entities for processing, which wastes a lot of transmission resources.

[0006] Summary of the Invention

[0007] The present disclosure provides a method, system, and device for data aggregation processing, which can reduce the waste of transmission resources.

[0008] In a first aspect, a method for data aggregation processing is provided, which is applied to a distributed database system, wherein the distributed database system includes a coordination node and multiple storage nodes. The processing process of the method may include: the coordination node receives an aggregation instruction for an original data table; then, the coordination node may obtain a first sharding key of an aggregated data table corresponding to the original data table, wherein the aggregated data table is used to record data after aggregation processing of the data in the original data table, and the first sharding key includes at least one first attribute tag of the aggregated data table; further, the coordination node may determine first allocation information, wherein the first allocation information is used to indicate storage nodes respectively allocated to multiple first attribute value sets corresponding to the first sharding key, and the first attribute value set consists of one attribute value for each first attribute tag; the coordination node compares the first attribute value set in the first allocation information with the attribute value in the data unit corresponding to the original data table to distribute the data unit corresponding to the original data table to the corresponding storage node; the storage node performs aggregation processing based on the received data unit.

[0009] The data unit corresponding to the original data table can be the data unit of the original data table directly received by the coordination node, or it can be the data unit obtained after the coordination node performs certain processing on the data unit of the received original data table (for example, deleting some fields, aggregating, etc.).

[0010] In this way, data aggregation processing is performed through a distributed database system. Both data storage and data aggregation processing are completed by the distributed database, eliminating the need to build an additional processing system for data aggregation processing. Furthermore, there is no need to copy data and send it separately to the distributed database system and the additional processing system. This approach can reduce the use of transmission resources.

[0011] In one possible implementation, the first sharding key is the same as the second sharding key of the original data table, and the second sharding key includes at least one second attribute tag of the original data table. In this case, the coordination node can determine the first allocation information based on the second allocation information of the original data table. The second allocation information is used to indicate the storage nodes respectively allocated to the multiple second attribute value sets corresponding to the second sharding key, and the storage nodes indicated by the second allocation information are used to perform disk flushing. The second attribute value set consists of one attribute value for each second attribute tag. In the first allocation information and the second allocation information, the storage nodes corresponding to the same attribute value set are the same.

[0012] In this way, the first allocation information for aggregation directly reuses the second allocation information for disk flushing, which can simplify processing, improve efficiency, and reduce the occupation of processing resources.

[0013] In one possible implementation, the first shard key and the second shard key are different and are subsets of the second shard key, and the second shard key includes at least one second attribute tag of the original data table. In this case, the coordinating node can determine multiple first attribute value sets corresponding to the first shard key and multiple second attribute value sets corresponding to the second shard key, and then determine the first allocation information based on the multiple first attribute value sets and the multiple second attribute value sets. The second attribute value set consists of one attribute value for each second attribute tag.

[0014] In one possible implementation, for each first attribute value set, the coordination node can determine at least one second attribute value set that includes the first attribute value set, and select a storage node from the storage nodes corresponding to the at least one second attribute value set as the storage node corresponding to the first attribute value set; further, the coordination node can determine the first allocation information based on the storage node corresponding to each first attribute value set.

[0015] In this way, when the first shard key is a subset of the second shard key, the data that needs to be flushed to disk in the storage node can be aggregated as much as possible, reducing additional data transmission and improving network resource utilization.

[0016] In a possible implementation, the coordination node may determine the first allocation information based on a storage node selection rule, wherein the storage node selection rule may include random selection, selection in a specified order, or selection based on resource occupancy.

[0017] This method of determining the first allocation information can be performed in any case, and the first sharding key and the second sharding key may not satisfy any relationship.

[0018] This provides greater flexibility when allocating storage nodes for aggregation processing.

[0019] In one possible implementation, the first shard key and the second shard key are different and are not a subset of the second shard key. In this case, the coordination node may determine the first allocation information based on the above storage node selection rule.

[0020] In this way, for the case where the first shard key and the second shard key are the same, and the first shard key is a subset of the second shard key, you can choose the above more resource-saving processing method. For the case where the first shard key and the second shard key are different and are not a subset of the second shard key, you can use this more universal and flexible processing method.

[0021] In one possible implementation, the first shard key is determined based on the aggregation instruction. Alternatively, the first shard key may be obtained through other methods, such as manually storing configuration information carrying the first shard key in the coordinating node, and obtaining the first shard key from the configuration information. In another example, the first shard key may be set using a create shard key instruction, i.e., the first shard key is obtained from the create shard key instruction.

[0022] The above-mentioned setting of the first shard key through the aggregation instruction can complete the aggregation task and the setting of the shard key through only one aggregation instruction, which helps to improve operation efficiency.

[0023] In one possible implementation, the processing of distributing data units can be as follows: the coordination node can compare the first attribute value set in the first allocation information and the second attribute value set in the second allocation information with the attribute values ​​in the data units corresponding to the original data table, respectively, to add aggregation tags and / or write tags to the data units corresponding to the original data table, and distribute them to the corresponding storage nodes.

[0024] The aggregation mark is used to instruct the storage node to perform aggregation processing on the data unit, and the write mark is used to instruct the storage node to perform disk flushing processing on the data unit.

[0025] In this way, through aggregation marking and write marking, the storage node can clearly distinguish which data units need to be flushed to disk, which data units need aggregation processing, and which data units need both aggregation processing and flushing to disk processing, so as to improve the processing efficiency of the storage node.

[0026] In one possible implementation, the processing of distributing data units can be as follows: the coordination node can determine the first attribute value set that matches the attribute value of the first data unit of the received original data table and the corresponding first storage node in the first allocation information, and can determine the second attribute value set that matches the attribute value of the first data unit of the received original data table and the corresponding second storage node in the second allocation information; when the first storage node and the second storage node are the same storage node, the coordination node can add an aggregation mark and a write mark to the first data unit, and send it to the first storage node; when the first storage node and the second storage node are not the same storage node, the coordination node can copy the first data unit to obtain the second data unit, aggregate the first data unit with other data units that match the first attribute value set, add an aggregation mark to the data unit obtained by the aggregation process and send it to the first storage node, add a write mark to the second data unit and send it to the second storage node.

[0027] In this way, the first-level aggregation can be performed on the coordination node, and then the second-level aggregation can be performed on the storage node. In some application scenarios where the pressure on the coordination node is relatively small, the processing resources of the coordination node can be well utilized to reduce the processing burden of the storage node.

[0028] In one possible implementation, the processing of distributing data units can be as follows: the coordination node can determine a first attribute value set that matches the attribute value of the first data unit of the received original data table and the corresponding first storage node in the first allocation information, and can determine a second attribute value set that matches the attribute value of the first data unit of the received original data table and the corresponding second storage node in the second allocation information; when the first storage node and the second storage node are the same storage node, the coordination node can add an aggregation mark and a write mark to the first data unit, and send it to the first storage node; when the first storage node and the second storage node are not the same storage node, the coordination node can copy the first data unit to obtain the second data unit, add an aggregation mark to the first data unit and send it to the first storage node, and add a write mark to the second data unit and send it to the second storage node.

[0029] In this way, the coordination node only performs distribution work and does not participate in aggregation processing. In some application scenarios where the coordination node is under greater pressure, it can prevent the coordination node from being stuck or even crashing.

[0030] In one possible implementation, the storage node may apply for a first temporary cache from the first cache pool and write the received data units into the first temporary cache; then, the storage node may write the data units in the first temporary cache into the storage disk; further, the storage node may apply for a second temporary cache having the same space size as the first temporary cache from the second cache pool and release the second temporary cache to the first cache pool; thereafter, the storage node performs aggregation processing based on the data units in the first temporary cache; and the storage node releases the first temporary cache to the second cache pool.

[0031] In this way, the first buffer pool is supplemented by the second temporary buffer, thereby preventing memory leaks in the first buffer pool.

[0032] In a second aspect, a distributed database system is provided, which includes a coordination node and multiple storage nodes. The coordination node and the multiple storage nodes are used to execute the method provided by the first aspect and its possible implementation methods.

[0033] In a third aspect, a device for data aggregation processing is provided, which includes at least one module, and the at least one module is used to implement the method provided by the first aspect and its possible implementation methods.

[0034] In a fourth aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the method provided by the above-mentioned first aspect and its possible implementation methods.

[0035] In a fifth aspect, a computer device is provided, comprising a memory and a processor, wherein the memory is configured to store computer instructions and the processor executes the computer instructions stored in the memory to cause the computer device to perform the method provided in the first aspect and possible implementations thereof.

[0036] In a sixth aspect, a computer-readable storage medium is provided, which stores computer program code. When the computer program code is executed by a computer device, the computer device executes the method provided by the first aspect and its possible implementation methods.

[0037] In a seventh aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the method provided by the first aspect and possible implementations thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG1 is a schematic structural diagram of a distributed database system provided by an embodiment of the present disclosure;

[0039] FIG2 is a flow chart of a method for data aggregation processing provided by an embodiment of the present disclosure;

[0040] FIG3 is a schematic diagram of allocation information provided by an embodiment of the present disclosure;

[0041] FIG4 is a schematic diagram of a dual-pool exchange process provided by an embodiment of the present disclosure;

[0042] FIG5 is a flow chart of a method for data aggregation processing provided by an embodiment of the present disclosure;

[0043] FIG6 is a schematic diagram of allocation information provided by an embodiment of the present disclosure;

[0044] FIG7 is a flow chart of a method for data aggregation processing provided by an embodiment of the present disclosure;

[0045] FIG8 is a schematic diagram of allocation information provided by an embodiment of the present disclosure;

[0046] FIG9 is a schematic structural diagram of a data aggregation processing apparatus provided by an embodiment of the present disclosure;

[0047] FIG10 is a schematic structural diagram of a data aggregation processing apparatus provided by an embodiment of the present disclosure;

[0048] FIG11 is a schematic diagram of a computing device provided by an embodiment of the present disclosure;

[0049] FIG12 is a schematic diagram of a computing device cluster provided by an embodiment of the present disclosure;

[0050] FIG13 is a schematic diagram of a computing device cluster provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0052] The concepts involved in this disclosure are explained below:

[0053] Time series data

[0054] Time series data is a series of data generated continuously over time. Time series data carries a timestamp (used to identify the time when the time series data was sent). For example, time series data can be data periodically collected by IoT sensors, transaction data in the financial market, and so on.

[0055] polymerization

[0056] Aggregation, also known as aggregate calculation or aggregate processing, involves dividing database data into groups according to specific rules, performing specified calculations (e.g., summation) on multiple rows of data within each group, and obtaining a single calculation result for each group as the aggregated result. Aggregation of time series data involves performing specified calculations on time series data that meets certain time conditions (e.g., time series data within the same time period).

[0057] Original data table

[0058] A raw data table is used to record raw data before aggregation. Raw data can be directly obtained through data collection or generated in real time during business execution. A row of data in a raw data table is called a data unit.

[0059] The embodiments of the present disclosure use an original data table recording time series data as an example to illustrate the solution.

[0060] Aggregate Data Table

[0061] An aggregate data table is a data table that aggregates the data in the original data table. A row of data in an aggregate data table is called an aggregate data unit. If the data stored in the original data table is time series data, the data in the aggregate data table is also time series data.

[0062] Shard key (shardkey)

[0063] A shard key consists of at least one attribute tag in a data table. Data units can be grouped based on the attribute value corresponding to this at least one attribute tag. The purpose of grouping can be to partition multiple data units for storage or group aggregation. The shard key of an aggregate data table is used to group and aggregate data units. The shard key of the original data table (assuming the original data table is not aggregated) is used to partition data units for storage.

[0064] Take an example to illustrate the usage of shard keys: for example, the two data units of a certain original data table are M and N, and the shard keys of the original data table are tagA and tagB. tagA and tagB are two attribute labels in the original data table. The attribute values ​​corresponding to tagA and tagB in data unit M are am and bm, and the attribute values ​​corresponding to tagA and tagB in data unit N are an and bn. If am is equal to an and bm is equal to bn, then data units M and N should be grouped together and stored in the same path. If am is not equal to an or bm is not equal to bn, then data units M and N should not be grouped together and should be stored in different paths.

[0065] The attribute tags or tags used for the shard key of a data table can be set manually or by using the Create Shard Key command. The Create Shard Key command format is as follows:

[0066] CREAT MEASUREMENT table_name WITH SHARDKEY tagA,tagB……

[0067] In this instruction, "CREATE MEASUREMENT" is the instruction name, and "WITH SHARDKEY" is used to introduce the attribute labels contained in the shard key. The characters of these two fields are fixed and can be called fixed fields.

[0068] "table_name" and "tagA, tagB..." are parameter fields. The content in the parameter fields can be set according to actual needs. "table_name" is the name of the data table for creating the shard key, and "tagA, tagB..." are the attribute tags that make up the shardkey.

[0069] Aggregation directives

[0070] The aggregation instruction is used to instruct the generation of an aggregated data table based on the original data table, that is, the data units of the original data table received after the aggregation instruction are aggregated to obtain aggregated data units, and the aggregated data units are used to combine into an aggregated data table. The data in the aggregated data table also belongs to time series data.

[0071] Aggregate directives can take the following forms:

[0072] CREATE STREAM stream_name INTO agg_table_name AS SELECT sum("value_tag")AS"agg_value_tag"FROM table_name GROUP BY TIME(time_1),"group_tagA","group_tagB"DELAY time_2

[0073] In this instruction, "CREATE STREAM" is the instruction name, "INTO" is used to extract the table name of the aggregate data table, "AS SELECT" is used to extract the attribute label of the numeric field of the original data table, which is the input data for the aggregate calculation. The subsequent "AS" is used to extract the attribute label of the numeric field of the aggregate data table, which is the aggregation result. "FROM" is used to extract the table name of the original data table, "GROUP BY" is used to extract the aggregation condition, "TIME" is used to extract the duration corresponding to the time window, and "DELAY" is used to extract the delay duration. The characters of these fields are fixed and can be called fixed fields.

[0074] "stream_name", "agg_table_name", "sum("value_tag")", "agg_value_tag", "table_name", "time_1", "group_tagA", "group_tagB", and "time_2" are parameter fields. The contents in the parameter fields can be set according to actual needs. The following describes these fields one by one:

[0075] "stream_name" is the name of the streaming aggregation task, "agg_table_name" is the name of the aggregation data table, which is the data table that records the result data after the aggregation calculation of time series data, and "table_name" is the name of the original data table, which is the data table that records time series data.

[0076] "TIME(time_1),"group_tagA","group_tagB"" can be generally referred to as the aggregation condition. Among them, "time_1" is the duration of the time window, and the time window is the time unit of the aggregation processing. For example, when time_1 is 1m, it means that a time window is created with 1 minute as the time unit, which means that the data units within 1 minute are aggregated to obtain aggregated data units. "group_tagA" and "group_tagB" are the attribute labels of the original data table, and are also the attribute labels of the columns to be added to the aggregated data table (here we take 2 attribute labels as an example, and their number can be any positive integer). When generating the aggregated data table, these two columns of data in the original data table are added to the aggregated data table. Based on the setting of the aggregation instruction parsing logic, "group_tagA" and "group_tagB" can be set as the sharding key of the aggregated data table by default. The above aggregation condition means: based on the duration of the time window "time_1", for multiple data units within a time window, data units with exactly the same attribute values ​​corresponding to "group_tagA" and "group_tagB" are grouped together, and data units in the same group are aggregated to obtain an aggregated data unit.

[0077] "time_2" is the delay duration, which is the allowed delay duration for data unit transmission. For example, if the time window duration is 5 minutes and the delay duration is 1m (1 minute), 9:00-9:05 is a time window. When a data unit with a timestamp of 9:04 is received at 9:05:30, and the data unit is received before 9:06 (9:05 plus 1 minute), the data unit can be aggregated with other data units in the 9:00-9:05 time window.

[0078] "value_tag" is the attribute tag corresponding to the numeric field in the original data table (the input data for the aggregate calculation). "sum()" represents the algorithm for the aggregate calculation. Here, addition is used as an example, but other algorithms can also be used. Besides "sum", calculation tasks can also include averaging, maximum, minimum, or variance, etc. Technical personnel can configure these tasks based on actual business needs. "agg_value_tag" is the attribute tag corresponding to the numeric field in the aggregated data table (the aggregation result).

[0079] To illustrate the above aggregation instructions in more detail, an example is given below:

[0080] Assume the aggregation instruction is:

[0081] CREATE STREAM test INTO table_2AS SELECT sum("value")AS"agg_value"FROM table_1GROUP BY TIME(1m),"tagA","tagB"DELAY 20s

[0082] The table name corresponding to the original data table is table_1. Assume that the set of time series data in a certain time window in table_1 is as shown in Table 1:

[0083] Table 1

[0084] Based on the values ​​of tagA and tagB in the aggregation condition, the time series data can be divided into four groups: the first group consists of data units where the attribute value corresponding to tagA is a1 and the attribute value corresponding to tagB is b1; the second group consists of data units where the attribute value corresponding to tagA is a1 and the attribute value corresponding to tagB is b2; the third group consists of data units where the attribute value corresponding to tagA is a2 and the attribute value corresponding to tagB is b1; and the fourth group consists of data units where the attribute value corresponding to tagA is a2 and the attribute value corresponding to tagB is b2. {a1, b1}, {a1, b2}, {a2, b1}, and {a2, b2} are the four attribute value sets of table_1, and each attribute value set corresponds to a group.

[0085] The value attribute values ​​of the four groups of data units are summed up to obtain four aggregated data units. The aggregated data units are recorded in the aggregated data table, table_2, as shown in Table 2:

[0086] Table 2

[0087] The embodiment of the present disclosure provides a method for data processing, which can be applied to a distributed database system. The distributed database system may include at least one coordination node 110 (also called a computing node), multiple storage nodes 120 and a storage disk 130. It may also include a metadata node 140, a function node 150, etc. The structure of the distributed database system is shown in FIG1 . In FIG1 , all storage nodes 120 are connected to the metadata node 140, and all storage nodes 120 are also connected to the function node 150. Only one connection relationship is drawn in FIG1 for example. Similarly, in FIG1 , all coordination nodes 110 are connected to the metadata node 140, and all coordination nodes 110 are also connected to the function node 150. Only one connection relationship is drawn in FIG1 for example. The following introduces each component of the distributed database system:

[0088] The coordination node 110 can be an independent computer device, a virtual machine, or a computer cluster. The coordination node 110 can receive time series data and perform parsing, format conversion, and preprocessing on the time series data. It can also determine the storage node corresponding to each time series data based on certain distribution rules and send each time series data to the corresponding storage node. The distribution rules can be automatically generated by the coordination node or manually set.

[0089] The storage node 120 can be an independent computer device, a virtual machine, or a computer cluster. The storage node 120 is used to organize time series data for disk flushing, that is, storing the time series data on the storage disk. It can also be used to perform other processing on the time series data, such as aggregation. The distributed database system includes multiple storage nodes, and different storage nodes store the received time series data in different paths on the storage disk.

[0090] The storage disk 130 is an underlying storage medium for storing data. There may be multiple storage disks 130 , which may be located in one device or multiple devices.

[0091] The metadata node 140 can be used to store various types of metadata, such as information recording the above-mentioned distribution rules.

[0092] Function node 150 may be used to store functions edited by a user.

[0093] Distributed database systems can be used to store time series data. In related technologies, real-time aggregation calculations of time series data are generally performed in big data components. The present disclosure provides a method for data aggregation processing, which is applied to a distributed database system. In this way, the distributed database system can not only store time series data but also perform real-time aggregation processing on time series data, saving transmission resources. As shown in Figure 2, the processing flow of this method includes the following steps:

[0094] Step 201: The coordination node receives an aggregation instruction for the original data table.

[0095] The aggregation instruction is used to instruct the generation of an aggregated data table based on the original data table, that is, to aggregate the data units of the original data table received subsequently to obtain aggregated data units, and use the aggregated data units to combine into an aggregated data table. The data in the aggregated data table also belongs to time series data.

[0096] In step 202, the coordinating node obtains the first shard key of the aggregate data table and obtains the second shard key of the original data table.

[0097] A shard key is a collection of several attribute tags. Both aggregated data tables and source data tables have corresponding shard keys. For aggregated data tables, you can set the first shard key using the shard key directive, or you can directly set all attribute tags in the aggregation conditions in the aggregation directive as the first shard key. The attribute tags included in the first shard key are referred to as first attribute tags. For source data tables, you can set the second shard key corresponding to the source data table using the shard key directive. The attribute tags included in the second shard key are referred to as second attribute tags.

[0098] Step 203: When the first sharding key and the second sharding key are the same, the coordination node determines the first allocation information based on the second allocation information of the original data table.

[0099] The first allocation information is used to indicate how the data unit should be distributed during aggregation, and records the storage nodes allocated to the various first attribute value sets corresponding to the first sharding key. The allocated storage nodes are used to perform aggregation processing, and the first attribute value set consists of one attribute value for each first attribute label. The second allocation information is used to indicate how the data unit should be distributed during disk flushing, and records the storage nodes allocated to the various second attribute value sets corresponding to the second sharding key. The allocated storage nodes are used to perform disk flushing processing, and the second attribute value set consists of one attribute value for each second attribute label.

[0100] The second allocation information is described in detail below:

[0101] After receiving the Create Shard Key instruction corresponding to the original data table and completing the setting of the second shard key, the coordinating node can generate second allocation information for the original data table. Specifically, based on the different values ​​corresponding to the second attribute tag of the second shard key of the original data table, the attribute values ​​corresponding to different attribute tags can be combined to generate multiple second attribute value sets. For example, the second attribute tags include three attribute tags, each of which has two different attribute values. Attribute tags A, B, and C each take one attribute value to form a second attribute value set, and a total of eight second attribute value sets can be generated.

[0102] The coordinating node assigns storage nodes to all second attribute value sets, obtaining second assignment information. For example, {a1, b1} is assigned storage node 1, {a1, b2} is assigned storage node 2, and so on, until all second attribute value sets are assigned storage nodes. The storage nodes assigned to each second attribute value set can be the same or different.

[0103] If there are multiple coordinating nodes, after setting the second allocation information, the coordinating node can send the second allocation information to the metadata node for storage. Other coordinating nodes can periodically access the metadata node. After the second allocation information is stored in the metadata node, other coordinating nodes can read the second allocation information for subsequent data distribution.

[0104] After executing step 202 to obtain the first shard key of the aggregate data table and the second shard key of the original data table, the coordinating node may determine whether the first shard key and the second shard key are completely equal, that is, whether the first attribute label in the first shard key and the second attribute label in the second shard key are completely equal. If they are completely equal, the process of determining the first allocation information may be as follows:

[0105] The first shard key and the second shard key are identical, and the multiple first attribute value sets and the multiple second attribute value sets are also identical. When assigning storage nodes to each first attribute value set, the storage nodes corresponding to the same first attribute value set and the second attribute value set can be set to be the same to obtain first allocation information. In this way, the first allocation information can be considered to be the same as the second allocation information.

[0106] The second allocation information and the first allocation information are further explained below by way of example:

[0107] The data of a certain time window of a distributed database is shown in Table 1 above. Assuming that the second sharding keys are tagA and tagB, the value of tagA is a1 or a2, and the value of tagB is b1 or b2, four second attribute value sets {a1, b1}, {a1, b2}, {a2, b1} and {a2, b2} can be obtained. A storage node is assigned to each second attribute value set to obtain second allocation information, as shown in Figure 3. Assuming that the second allocation information is {a1, b1}→storage node 1, {a1, b2}→storage node 2, {a2, b1}→storage node 3 and {a2, b2}→storage node 4. Assuming that the first sharding key is tagA and tagB, we can get the first attribute value set {a1, b1}, {a1, b2}, {a2, b1} and {a2, b2}, and then get the first allocation information as {a1, b1}→storage node 1, {a1, b2}→storage node 2, {a2, b1}→storage node 3 and {a2, b2}→storage node 4. It can be considered that the first allocation information is the same as the second allocation information.

[0108] In the case where there are multiple coordination nodes, after setting the first allocation information, the coordination node can send the first allocation information to the metadata node for storage. Other coordination nodes can obtain the first allocation information by periodically accessing the metadata node for subsequent data distribution.

[0109] Step 204: The coordination node distributes the received data units of the original data table to corresponding storage nodes based on the first allocation information.

[0110] After receiving the data unit, the coordinating node searches the first allocation information for the first attribute value set and the corresponding storage node included in the data unit. It then adds an aggregation flag and a write flag to the data unit. The write flag indicates that the data unit needs to be flushed to disk, and the aggregation flag indicates that the data unit needs to be aggregated. The marked data unit is then sent to the storage node.

[0111] The following is further explained by way of example (continue using the example of step 203):

[0112] The data unit with the attribute value {a1, b1} corresponding to the second attribute label is distributed to storage node 1, the data unit with the attribute value {a1, b2} corresponding to the second attribute label is distributed to storage node 2, the data unit with the attribute value {a2, b1} corresponding to the second attribute label is distributed to storage node 3, and the data unit with the attribute value {a2, b2} corresponding to the second attribute label is distributed to storage node 4. That is, the first and second rows of data in Table 1 are sent to storage node 1, the third and fourth rows of data in Table 1 are sent to storage node 2, the fifth and sixth rows of data in Table 1 are sent to storage node 3, and the seventh and eighth rows of data in Table 1 are sent to storage node 4.

[0113] Step 205: The storage node writes the received data units into the storage disk and performs aggregation processing based on the received data units.

[0114] The storage node establishes a first cache pool (also called a write cache pool) and a second cache pool (also called a computation cache pool). When a storage node receives a data unit with both an aggregation and write tag, it requests a first temporary cache from the write cache pool and writes the received data unit into the first temporary cache. The storage node then writes the data unit in the first temporary cache to the storage disk. After writing, the storage node requests a second temporary cache from the computation cache pool with the same size as the first temporary cache and releases the second temporary cache to the write cache pool. The storage node performs aggregation based on the data units in the first temporary cache, writes the aggregation result (which may be an intermediate or final aggregation result) into the aggregation cache, and then releases the first temporary cache to the computation cache pool. In this way, for the case where the aggregation algorithm is additive, each time a data unit is received, the attribute value used for aggregation in the data unit is added to the previously calculated cumulative value to obtain a new cumulative value. When the aggregation end condition is met, that is, when the delay time (carried in the aggregation instruction) has elapsed since the end time of the time window, the storage node writes the aggregation result in the aggregation cache to the storage disk, as shown in Figure 4.

[0115] Due to the existence of the latency mechanism, the aggregation cache in the storage node may simultaneously maintain aggregation results corresponding to multiple time windows (such as the above-mentioned cumulative value). For example, if each minute is a time window and the latency is 20 seconds, then during the time period from 9:01 to 9:01:20, the aggregation cache simultaneously maintains the aggregation results for the 9:00-9:01 time window and the 9:01-9:02 time window. During this time period, if a data unit with a timestamp in the 9:00-9:01 time window is received, the aggregation results corresponding to the 9:00-9:01 time window in the aggregation cache can be accumulated to update the aggregation result. If a data unit with a timestamp in the 9:01-9:02 time window is received, the aggregation results corresponding to the 9:01-9:02 time window in the aggregation cache can be accumulated to update the aggregation result. At 9:01:20, the aggregation result corresponding to the 9:00-9:01 time window in the aggregation cache is determined as the final aggregation result and written to the storage disk.

[0116] The processing flow in Figure 2 above is applicable to the case where the first sharding key of the aggregated data table is exactly the same as the second sharding key of the original data table. For the case where the first sharding key and the second sharding key are different, and the first attribute label corresponding to the first sharding key is a subset of the second attribute label corresponding to the second sharding key, the processing flow shown in Figure 5 can be used, including the following steps:

[0117] Step 501: The coordination node receives an aggregation instruction for the original data table.

[0118] The specific processing of this step is the same as step 201, and the relevant description of step 201 can be found.

[0119] In step 502, the coordinating node obtains the first shard key of the aggregate data table and obtains the second shard key of the original data table.

[0120] The specific processing of this step is the same as step 202, and the relevant description of step 202 can be referred to.

[0121] Step 503: When the first sharding key is a subset of the second sharding key, the coordination node determines a plurality of first attribute value sets corresponding to the first sharding key and a plurality of second attribute value sets corresponding to the second sharding key.

[0122] The coordinating node determines that the first shard key is a subset of the second shard key, that is, the first shard key and the second shard key are different, and the second attribute tag corresponding to the second shard key completely includes the first attribute tag corresponding to the first shard key. The first shard key corresponds to multiple first attribute value sets, each consisting of one attribute value for each first attribute tag; the second shard key corresponds to multiple second attribute value sets, each consisting of one attribute value for each second attribute tag.

[0123] Step 504: The coordination node determines first allocation information based on the multiple first attribute value sets and the multiple second attribute value sets.

[0124] The coordination node obtains the second allocation information of the distributed database, that is, obtains the storage node corresponding to each second attribute value set. Based on the first sharding key, the coordination node obtains multiple first attribute value sets. For each first attribute value set, at least one second attribute value set that includes the first attribute value set can be found. Then, at least one storage node corresponding to at least one second attribute value set can be determined, and a storage node is selected from the at least one storage node. The selection method can be random selection, selection in a specified order, or selection based on the load balancing principle. Then, the selected storage node can be assigned to the corresponding first attribute value set, and the first allocation information can be obtained.

[0125] The following are examples:

[0126] The data of a time window in a distributed database is shown in Table 1 above. Assuming that the second sharding key is tagA and tagB, the value of tagA is a1 or a2, and the value of tagB is b1 or b2, four second attribute value sets {a1, b1}, {a1, b2}, {a2, b1}, and {a2, b2} can be obtained. A storage node is assigned to each second attribute value set to obtain second allocation information, as shown in Figure 6. Assuming that the second allocation information is {a1, b1}→storage node 1, {a1, b2}→storage node 2, {a2, b1}→storage node 3, and {a2, b2}→storage node 4. Assuming that the first sharding key is tagA, two first attribute value sets {a1} and {a2} can be obtained. The second attribute value sets containing the first attribute value set {a1} include {a1, b1} and {a1, b2}. Among storage nodes 1 and 2 corresponding to {a1, b1} and {a1, b2}, one is arbitrarily selected as the storage node corresponding to {a1}. Suppose storage node 1 is selected as the storage node corresponding to {a1}. The second attribute value sets containing the first attribute value set {a2} include {a2, b1} and {a2, b2}. Among storage nodes 3 and 4 corresponding to {a2, b1} and {a2, b2}, one is arbitrarily selected as the storage node corresponding to {a2}. Suppose storage node 3 is selected as the storage node corresponding to {a2}. The first allocation information is then {a1} → storage node 1 and {a2} → storage node 3.

[0127] Step 505: The coordination node distributes the data units related to the original data table to the corresponding storage nodes based on the first allocation information and the second allocation information.

[0128] The coordination node can add an aggregation mark and / or a write mark to the data units related to the original data table based on the first allocation information and the second allocation information, and distribute them to the corresponding storage nodes. The aggregation mark is used to instruct the storage node to perform aggregation processing on the data unit. The write mark is used to instruct the storage node to perform disk flushing processing on the data unit. The data units related to the original data table can be data units of the original data table directly received by the coordination node, or they can be data units obtained after the coordination node performs certain processing on the data units of the received original data table (for example, deleting some fields, aggregation processing, etc.).

[0129] The above-mentioned process of deleting some fields may be as follows: for a data unit with an aggregation mark but no write mark, before sending the data unit, the attribute values ​​(ie, numerical fields) that need to be aggregated are retained in the data unit and other attribute values ​​are deleted.

[0130] The specific steps for distributing data units will be detailed later. The scheme for coordinating nodes to participate in partial aggregation processing will also be detailed later.

[0131] Step 506: The storage node performs disk flushing, aggregation, and other processing on the received data units.

[0132] After receiving the data unit, the storage node processes the data unit according to the aggregation flag and write flag carried by the data unit. The following describes the processing methods corresponding to the three different marking situations:

[0133] Case 1: For a data unit carrying both a write mark and an aggregation mark, the processing method is the same as step 205 .

[0134] Case 2: For data units that only carry write marks, after the storage node receives the data unit, it applies for a third temporary cache from the write cache pool, writes the received data unit into the third temporary cache, and then the storage node writes the data unit in the third temporary cache into the storage disk, and then releases the third temporary cache to the write cache pool.

[0135] In case three, for a data unit that carries only an aggregation tag, after receiving the data unit, the storage node requests a fourth temporary buffer from the computational cache pool and writes the received data unit to the fourth temporary buffer. The storage node then aggregates the data units in the fourth temporary buffer, writes the aggregated result to the aggregation buffer, and then releases the fourth temporary buffer to the computational cache pool. When the aggregation end condition is met, the storage node writes the aggregated result in the aggregation buffer to the storage disk.

[0136] The present disclosure also provides another data aggregation processing method. This method has a wide range of applications and can be applied to any situation where the first sharding key and the second sharding key are different. This method can also be used when the first sharding key is a subset of the second sharding key. The processing flow of this method can be shown in Figure 7 and includes the following steps:

[0137] Step 701: The coordination node receives an aggregation instruction for the original data table.

[0138] The specific processing of this step is the same as step 201, and the relevant description of step 201 can be found.

[0139] Step 702: The coordinating node obtains the first shard key of the aggregate data table.

[0140] The specific processing of this step is the same as step 202, and the relevant description of step 202 can be referred to.

[0141] Step 703: The coordinating node determines first allocation information based on the first sharding key and the storage node selection rule.

[0142] The coordinating node obtains the first attribute tag and the second attribute tag corresponding to the first sharding key and the second sharding key. There are two situations: one is that the first attribute tag and the second attribute tag are completely different, and the other is that the first attribute tag and the second attribute tag have an intersection but the first attribute tag is not completely contained in the second attribute tag. For both situations, the first allocation information can be determined based on the first sharding key. The specific processing process can be to determine multiple first attribute value sets corresponding to the first sharding key and assign a storage node to each first attribute value set. The storage node selection rule can be random selection, selection in a specified order, or selection based on load balancing principles.

[0143] The following are examples:

[0144] The data of a time window of a distributed database is shown in Table 3. Assuming that the second sharding keys are tagA and tagB, the value of tagA is a1 or a2, and the value of tagB is b1 or b2, four second attribute value sets {a1, b1}, {a1, b2}, {a2, b1}, and {a2, b2} can be obtained. A storage node is assigned to each second attribute value set to obtain the second allocation information, as shown in Figure 8. Assume that the second allocation information is {a1, b1}→storage node 1, {a1, b2}→storage node 2, {a2, b1}→storage node 3, and {a2, b2}→storage node 4. Assuming that the first sharding keys are tagB and tagC, the value of tagB is b1 or b2, and the value of tagC is c1 or c2, four first attribute value sets {b1, c1}, {b1, c2}, {b2, c1}, and {b2, c2} can be obtained. A storage node is assigned to each attribute value set to obtain the first allocation information. Assume that the first allocation information is {b1, c1}→storage node 1, {b1, c2}→storage node 2, {b2, c1}→storage node 3, and {b2, c2}→storage node 4.

[0145] Table 3

[0146] Step 704: The coordination node distributes the data units related to the original data table to the corresponding storage nodes based on the first allocation information and the second allocation information.

[0147] The processing of this step is similar to that of step 505 , and reference may be made to the relevant description of step 505 .

[0148] Step 705: The storage node performs processing such as flushing and aggregation on the received data units.

[0149] The processing of this step is similar to that of step 506 , and reference may be made to the relevant description of step 506 .

[0150] In an embodiment of the present disclosure, for the data units of the received original data table, if the storage node that performs the aggregation processing corresponding to the data unit and the storage node that performs the disk flushing processing are not the same storage node, it can be considered to first perform a first-level aggregation processing on the data unit at the coordination node, and then perform a second-level aggregation processing at the storage node (because one coordination node cannot obtain all the data units that need to be aggregated, so it is necessary to complete the final aggregation, i.e., the second-level aggregation, at the storage node). Whether to perform two-level aggregation can be determined by a pre-determination process, that is, two working modes are set for the coordination node, mode one is a mode in which the coordination node does not participate in the aggregation, and mode two is a mode in which the coordination node participates in the aggregation. For these two modes, selection can be made based on some reference information, and the reference information may include relevant information of the original data table and relevant information of the coordination node. For example, the input-output ratio of the aggregation processing of the original data table, the input-output ratio is the ratio of the number of data before and after aggregation (can be estimated based on experience or can be measured by experiments on the original data table). When the input-output ratio is greater than a specified threshold, mode two is adopted. When the input-output ratio is less than the specified threshold, mode one is adopted. The specified threshold can be set based on experience or experiments.

[0151] In mode 1, the processing of step 505 and step 704 can be as follows:

[0152] After the coordination node receives the data unit of the original data, it determines the first attribute value set that matches the data unit in the first allocation information, and then determines the first storage node corresponding to the first attribute value set. In the second allocation information, it determines the second attribute value set that matches the data unit, and then determines the second storage node corresponding to the second attribute value set. If the first storage node and the second storage node are the same, a write mark and an aggregation mark can be added to the data unit, and the marked data unit can be sent to the storage node; if the first storage node and the second storage node are different, the data unit is copied, an aggregation mark is added to one of the two data units, and the marked data unit is sent to the first storage node, a write mark is added to the other of the two data units, and the marked data unit is sent to the second storage node. Optionally, for data units that have an aggregation mark but no write mark, before sending the data unit, the attribute values ​​(i.e., numerical fields) that need to be aggregated are retained in the data unit and other attribute values ​​are deleted.

[0153] The following examples are used to illustrate (using the examples of step 504 and step 703 respectively):

[0154] The processing of step 505 corresponding to the example of step 504 is as follows:

[0155] The first attribute value set matched by the data unit in the first row of Table 1 is {a1}, and the second attribute value set matched is {a1, b1}. The storage nodes corresponding to {a1} and {a1, b1} are both storage node 1. A write mark and an aggregation mark are added to the data unit, and the marked data unit is distributed to storage node 1. The data unit in the second row is similar to the data unit in the first row. A write mark and an aggregation mark are also added and sent to storage node 1. The first attribute value set matched by the data unit in the third row is {a1}, and the second attribute value set matched is {a1, b2}. , the storage node corresponding to {a1} is storage node 1, and the storage node corresponding to {a1, b2} is storage node 2. Then copy the third row of data, add an aggregation tag to one copy and distribute it to storage node 1, and add a write tag to the other copy and distribute it to storage node 2; the fourth row of data units are similar to the third row of data units. Then copy the fourth row of data, add an aggregation tag to one copy and distribute it to storage node 1, and add a write tag to the other copy and distribute it to storage node 2; the fifth row of data units to the eighth row of data units are the same as the above processing steps, and they are not listed here one by one.

[0156] Optionally, for a data unit that has an aggregation tag but no write tag, before sending the data unit, the attribute value (ie, the numerical field) corresponding to tagA is retained in the data unit and other attribute values ​​are deleted.

[0157] The processing of step 704 corresponding to the example of step 703 is as follows:

[0158] The first attribute value set matched by the first row of data units in Table 3 is {b1, c1}, and the second attribute value set matched is {a1, b1}. The storage node corresponding to {b1, c1} and {a1, b1} is storage node 1. Write marks and aggregation marks are added to the first row of data, and the marked data units are distributed to storage node 1; the first attribute value set matched by the second row of data units is {b1, c2}, and the second attribute value set matched is {a1, b1}. The storage node corresponding to {b1, c2} is storage node 2, and the storage node corresponding to {a1, b1} is storage node 1. Then copy the second row of data units, add the aggregation mark to one of them and distribute it to storage node 2, and add the write mark to the other and distribute it to storage node 1; the above processing steps are the same for the third to eighth row of data units, and they are not listed one by one here.

[0159] Optionally, for a data unit that has an aggregation tag but no write tag, before sending the data unit, the attribute values ​​(ie, numerical fields) corresponding to tagB and tagC are retained in the data unit, and other attribute values ​​are deleted.

[0160] In mode 2, the processing of step 505 and step 704 can be as follows:

[0161] The process of adding aggregation tags and write tags to data units in the original data table is the same as in Mode 1. When there are multiple data units in the original data table that have been added with aggregation tags but not with write tags to be sent, the coordination node can aggregate these data units, retaining the attribute values ​​that need to be aggregated and deleting other attribute values. The coordinating node can then add aggregation tags to the aggregated data units and send them to the storage node. The storage node can perform secondary aggregation based on the received aggregated data units.

[0162] The embodiment of the present disclosure also provides an instruction for viewing aggregated tasks. The instruction may be in the following form:

[0163] SHOW STERAMS

[0164] The characters in this command are fixed and are used to view the running status of all aggregate tasks created by the aggregate command.

[0165] The embodiment of the present disclosure also provides an instruction for deleting an aggregation task. The instruction may be in the following form:

[0166] DROP STREAM stream_name

[0167] DROP STREAM is the command name. The characters in this field are fixed and can be called a fixed field. "stream_name" is a parameter field whose content can be set according to actual needs. "stream_name" is the task name of the streaming aggregation task.

[0168] This command is used to delete the streaming aggregation task named "stream_name". After deleting the streaming aggregation task, the original data table can continue to record time series data, while the aggregated data table stops recording data.

[0169] In the disclosed embodiments, data aggregation is performed through a distributed database system. This allows both data storage and data aggregation to be handled by the distributed database, eliminating the need to build an additional processing system for data aggregation. Furthermore, there is no need to copy data and send it separately to the distributed database system and the additional processing system. This approach can reduce the use of transmission resources.

[0170] Based on the same technical concept, an embodiment of the present disclosure provides a device for data aggregation processing, which can be applied to the above-mentioned coordination node. As shown in FIG9 , the device includes:

[0171] The receiving module 910 is used to receive the aggregation instruction for the original data table, and can specifically implement the processing functions of the above steps 201, 501 and 701, as well as other implicit steps.

[0172] Acquisition module 920 is configured to obtain a first sharding key for the aggregated data table corresponding to the original data table. The aggregated data table is used to record data after the data in the original data table has been aggregated. The first sharding key includes at least one first attribute tag of the aggregated data table. Specifically, this module implements the processing functions of steps 202, 502, and 702 above, as well as other implicit steps.

[0173] Determination module 930 is configured to determine first allocation information, where the first allocation information indicates storage nodes allocated for performing aggregation processing for multiple first attribute value sets corresponding to the first shard key, where the first attribute value sets consist of one attribute value for each first attribute tag. Specifically, this module can implement the processing functions of steps 203, 504, and 703 above, as well as other implicit steps.

[0174] Distribution module 940 is configured to compare the first attribute value set in the first allocation information with the attribute values ​​in the corresponding data units in the original data table, thereby distributing the data units corresponding to the original data table to the corresponding storage nodes. Specifically, this module can implement the processing functions of steps 204, 505, and 704 above, as well as other implicit steps.

[0175] In one possible implementation, determination module 930 is used to determine first allocation information based on second allocation information of the original data table, wherein the second allocation information is used to indicate storage nodes for performing disk flushing that are respectively allocated to multiple second attribute value sets corresponding to the second sharding key, the storage nodes indicated by the second allocation information are used to perform disk flushing, the second attribute value sets are composed of one attribute value for each second attribute label, and in the first allocation information and the second allocation information, the storage nodes corresponding to the same attribute value sets are the same. Specifically, the processing function of the above-mentioned step 203 and other implicit steps can be implemented.

[0176] In one possible implementation, determination module 930 is configured to determine multiple first attribute value sets corresponding to the first sharding key and multiple second attribute value sets corresponding to the second sharding key, where the second attribute value sets consist of one attribute value for each second attribute tag, and to determine first allocation information based on the multiple first attribute value sets and the multiple second attribute value sets. Specifically, this module may implement the processing functionality of step 503 above, as well as other implicit steps.

[0177] In one possible implementation, determination module 930 is configured to, for each first attribute value set, determine at least one second attribute value set that includes the first attribute value set, select one storage node from the storage nodes corresponding to the at least one second attribute value set as the storage node corresponding to the first attribute value set, and determine first allocation information based on the storage node corresponding to each first attribute value set. Specifically, this module may implement the processing functionality of step 504 above, as well as other implicit steps.

[0178] In one possible implementation, determination module 930 is configured to determine the first allocation information based on a storage node selection rule, where the storage node selection rule includes random selection, selection in a specified order, or selection based on resource occupancy. Specifically, this module may implement the processing functionality of step 504 above, as well as other implicit steps.

[0179] In a possible implementation, the first sharding key and the second sharding key are different and are not a subset of the second sharding key, and the second sharding key includes at least one second attribute label of the original data table.

[0180] In one possible implementation, the first shard key is determined based on the aggregation instruction.

[0181] In one possible implementation, the device further includes a marking module 950 for comparing the first attribute value set in the first allocation information and the second attribute value set in the second allocation information with the attribute values ​​in the data units corresponding to the original data table, respectively, to add an aggregation mark and / or a write mark to the data units corresponding to the original data table. Specifically, this can implement the processing functions of steps 204, 505, and 704, as well as other implicit steps. A distribution module 940 is used to distribute the marked data units to the corresponding storage nodes, wherein the second allocation information is used to indicate the storage nodes for performing disk flushing processing respectively assigned to the multiple second attribute value sets corresponding to the second sharding key, the storage nodes indicated by the second allocation information are used to perform disk flushing processing, the second sharding key includes at least one second attribute tag of the original data table, the second attribute value set consists of one attribute value for each second attribute tag, the aggregation mark is used to indicate that the storage node performs aggregation processing on the data unit, and the write mark is used to indicate that the storage node performs disk flushing processing on the data unit. Specifically, this can implement the processing functions of steps 204, 505, and 704, as well as other implicit steps.

[0182] In one possible implementation, the determination module 930 is used to determine, in the first allocation information, a first attribute value set that matches the attribute value of the first data unit of the received original data table and a corresponding first storage node, and to determine, in the second allocation information, a second attribute value set that matches the attribute value of the first data unit of the received original data table and a corresponding second storage node; specifically, it can implement the processing functions of the above steps 203, 504 and 703, as well as other implicit steps.

[0183] The marking module 950 is configured to add an aggregation mark and a write mark to the first data unit when the first storage node and the second storage node are the same storage node, and can specifically implement the processing function of the above step 204 and other implicit steps.

[0184] The distribution module 940 is used to send the marked data unit to the first storage node; specifically, it can implement the processing function of the above step 204 and other implicit steps.

[0185] Marking module 950 is configured to, when the first storage node and the second storage node are different storage nodes, copy the first data unit to obtain a second data unit, aggregate the first data unit with other data units that match the first attribute value set, add an aggregation marker to the aggregated data unit, and add a write marker to the second data unit. Specifically, this module implements the processing functions of steps 505 and 704 above, as well as other implicit steps.

[0186] The distribution module 940 is configured to send the first data unit to the first storage node and the second data unit to the second storage node, and can specifically implement the processing functions of steps 505 and 704 above, as well as other implicit steps.

[0187] In one possible implementation, the determination module 930 is used to determine, in the first allocation information, a first attribute value set that matches the attribute value of the first data unit of the received original data table and a corresponding first storage node, and to determine, in the second allocation information, a second attribute value set that matches the first data unit of the received original data table and a corresponding second storage node; specifically, it can implement the processing function of the above-mentioned step 505 and other implicit steps.

[0188] The marking module 950 is configured to add an aggregation mark and a write mark to the first data unit when the first storage node and the second storage node are the same storage node, and can specifically implement the processing function of the above step 504 and other implicit steps.

[0189] The distribution module 940 is used to send the marked data unit to the first storage node; specifically, it can implement the processing function of the above step 504 and other implicit steps.

[0190] Marking module 950 is configured to, when the first storage node and the second storage node are different storage nodes, copy the first data unit to obtain a second data unit, add an aggregation marker to the first data unit, and add a write marker to the second data unit. Specifically, this module implements the processing functions of steps 505 and 704 above, as well as other implicit steps.

[0191] The distribution module 940 is configured to send the first data unit to the first storage node and the second data unit to the second storage node, and can specifically implement the processing functions of steps 505 and 704 above, as well as other implicit steps.

[0192] In the disclosed embodiments, data aggregation is performed through a distributed database system. This allows both data storage and data aggregation to be handled by the distributed database, eliminating the need to build an additional processing system for data aggregation. Furthermore, there is no need to copy data and send it separately to the distributed database system and the additional processing system. This approach can reduce the use of transmission resources.

[0193] The receiving module 910, the acquiring module 920, the determining module 930, the distributing module 940, and the marking module 950 can all be implemented via software or hardware. For example, the implementation of the receiving module 910 will be described below using the receiving module 910 as an example. Similarly, the implementation of the acquiring module 920, the determining module 930, the distributing module 940, and the marking module 950 can refer to the implementation of the receiving module 910.

[0194] As an example of a software functional unit, the receiving module 910 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the receiving module 910 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0195] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0196] As an example of a hardware functional unit, the receiving module 910 may include at least one computing device, such as a server. Alternatively, the receiving module 910 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0197] The multiple computing devices included in receiving module 910 can be distributed in the same region or in different regions. The multiple computing devices included in receiving module 910 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in receiving module 910 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0198] It should be noted that, in other embodiments, the receiving module 910, the acquisition module 920, the determination module 930, the distribution module 940 and the marking module 950 can be used for any step in the data aggregation processing method in the computing node, and the steps that the receiving module 910, the acquisition module 920, the determination module 930, the distribution module 940 and the marking module 950 are responsible for implementing can be specified as needed. The full functions of the device for performing data aggregation processing on the computing node are realized by respectively implementing different steps in the data aggregation processing method through the receiving module 910, the acquisition module 920, the determination module 930, the distribution module 940 and the marking module 950.

[0199] Based on the same technical concept, embodiments of the present disclosure provide a data aggregation processing apparatus that can be applied to the aforementioned storage node. As shown in FIG10 , the apparatus includes an aggregation module 1010 for performing aggregation processing based on received data units. Specifically, the apparatus can implement the processing functions of steps 205 , 506 , and 705 , as well as other implicit steps.

[0200] In a possible implementation, the device further includes an application module 1020 for applying for a first temporary cache from the first cache pool and writing the received data unit into the first temporary cache; specifically, it can implement the processing function of the above step 205 and other implicit steps.

[0201] The device further includes a writing module 1030 for writing the data units in the first temporary cache into the storage disk; specifically, the writing module 1030 can implement the processing function of the above step 205 and other implicit steps.

[0202] The device also includes a release module 1040 for applying to the second cache pool for a second temporary cache having the same space size as the first temporary cache, and releasing the second temporary cache to the first cache pool; specifically, it can implement the processing function of the above step 205 and other implicit steps.

[0203] Aggregation module 1010 is used to perform aggregation processing based on the data units in the first temporary cache; specifically, it can implement the processing function of step 205 above, as well as other implicit steps.

[0204] The release module 1040 is further configured to release the first temporary buffer to the second buffer pool, and specifically implement the processing function of the above step 205 and other implicit steps.

[0205] In the disclosed embodiments, data aggregation is performed through a distributed database system. This allows both data storage and data aggregation to be handled by the distributed database, eliminating the need to build an additional processing system for data aggregation. Furthermore, there is no need to copy data and send it separately to the distributed database system and the additional processing system. This approach can reduce the use of transmission resources.

[0206] Aggregation module 1010, application module 1020, write module 1030, and release module 1040 can all be implemented in software or hardware. For example, the implementation of aggregation module 1010 will be described below using aggregation module 1010 as an example. Similarly, the implementation of application module 1020, write module 1030, and release module 1040 can refer to the implementation of aggregation module 1010.

[0207] As an example of a software functional unit, the aggregation module 1010 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the aggregation module 1010 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0208] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0209] As an example of a hardware functional unit, the aggregation module 1010 may include at least one computing device, such as a server. Alternatively, the aggregation module 1010 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0210] The multiple computing devices included in the aggregation module 1010 can be distributed in the same region or in different regions. The multiple computing devices included in the aggregation module 1010 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the aggregation module 1010 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0211] It should be noted that, in other embodiments, the aggregation module 1010, the application module 1020, the write module 1030 and the release module 1040 can be used for any step in the data aggregation processing method in the computing node, and the steps that the aggregation module 1010, the application module 1020, the write module 1030 and the release module 1040 are responsible for implementing can be specified as needed. The full functions of the device for performing data aggregation processing on the computing node are realized by respectively implementing different steps in the data aggregation processing method through the aggregation module 1010, the application module 1020, the write module 1030 and the release module 1040.

[0212] The present disclosure also provides a distributed database system, as shown in FIG1 , which includes a coordination node, multiple storage nodes, and a storage disk. The functions of the coordination node, storage node, and storage disk have been introduced and will not be repeated here.

[0213] The coordination node, storage node, and storage disk can all be implemented in software or hardware. For example, the following describes the implementation of the coordination node. Similarly, the implementation of the storage node and storage disk can refer to the implementation of the coordination node.

[0214] As an example of a software functional unit, a module and a coordination node may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container and other computing devices. Furthermore, the above-mentioned computing device may be one or more. For example, a device for performing data aggregation processing may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same AZ or in different AZs, and each AZ includes one data center or multiple data centers with close geographical locations. Generally, a region may include multiple AZs.

[0215] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is located within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0216] A module is an example of a hardware functional unit. The coordination node may include at least one computing device, such as a server. Alternatively, the device for performing data aggregation processing may be implemented using an ASIC or a PLD. The PLD may be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0217] The multiple computing devices included in the coordination node can be distributed in the same region or in different regions. The multiple computing devices included in the coordination node can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the coordination node can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0218] The present disclosure also provides a computing device 100. The computing device 100 can be applied to the aforementioned coordination node or storage node. As shown in FIG11 , the computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other via the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the present disclosure does not limit the number of processors and memories in the computing device 100.

[0219] Bus 102 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG8 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 102 may include a path for transmitting information between various components of computing device 100 (e.g., memory 106, processor 104, and communication interface 108).

[0220] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0221] The memory 106 may include volatile memory, such as random access memory (RAM). The memory 106 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0222] Memory 106 stores executable program code. Processor 104 executes the executable program code to implement the functions of the aforementioned receiving module 910, obtaining module 920, determining module 930, distributing module 940, marking module 950, aggregating module 1010, applying module 1020, writing module 1030, and releasing module 1040, thereby implementing the data aggregation processing method. In other words, memory 106 stores instructions for the data aggregation processing method.

[0223] Alternatively, the memory 106 stores executable code, and the processor 104 executes the executable code to respectively implement the functions of the aforementioned receiving module 910, obtaining module 920, determining module 930, distributing module 940, marking module 950, aggregating module 1010, applying module 1020, writing module 1030, and releasing module 1040, thereby implementing the data aggregation processing method. In other words, the memory 106 stores instructions for the data aggregation processing method.

[0224] The communication interface 108 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0225] The present disclosure also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0226] As shown in Figure 12, the computing device cluster includes at least one computing device 100. The memory 106 of one or more computing devices 100 in the computing device cluster may store the same instructions for the method for data aggregation processing.

[0227] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for the method for data aggregation processing. In other words, the combination of one or more computing devices 100 can jointly execute the instructions for the method for data aggregation processing.

[0228] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions, each used for a portion of the functions of the data aggregation processing apparatus. That is, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more of the aforementioned receiving module 910, obtaining module 920, determining module 930, distributing module 940, marking module 950, aggregating module 1010, applying module 1020, writing module 1030, and releasing module 1040.

[0229] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network (WAN) or a local area network (LAN), and the network may be a Transmission Control Protocol (TCP) network or a Remote Direct Memory Access (RDMA) network. FIG13 illustrates a possible implementation. As shown in FIG13 , two computing devices 100A and 100B are connected via a network. Specifically, the connection to the network is achieved via a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of the receiving module 910, the acquiring module 920, the determining module 930, the distributing module 940, and the marking module 950. Simultaneously, the memory 106 in the computing device 100B stores instructions for executing the functions of the aggregation module 1010, the application module 1020, the writing module 1030, and the releasing module 1040.

[0230] The connection method between the computing device clusters shown in Figure 13 may be that considering that the method for data aggregation processing provided by the present disclosure requires a large amount of data storage, it is considered to entrust the functions implemented by the aggregation module 1010, the application module 1020, the writing module 1030 and the release module 1040 to be executed by the computing device 100B.

[0231] It should be understood that the functions of the computing device 100A shown in FIG13 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0232] The present disclosure also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection relationship between the computing device clusters described in Figures 12 and 13. However, the memory 106 of one or more computing devices 100 in this computing device cluster can store the same instructions for the data aggregation processing method.

[0233] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for the method for data aggregation processing. In other words, the combination of one or more computing devices 100 can jointly execute the instructions for the method for data aggregation processing.

[0234] The present disclosure also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be executed on a computing device or stored in any available medium. When executed on at least one computing device, the computer program product causes the at least one computing device to perform a data aggregation processing method.

[0235] The embodiments of the present disclosure also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disk (DVD)), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform a data aggregation processing method, or instruct a computing device to perform a data aggregation processing method.

[0236] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present disclosure.

Claims

1. A method for data aggregation processing, characterized in that: The method is applied to a distributed database system, the distributed database system includes a coordination node and a plurality of storage nodes, and the method includes: The coordination node receives an aggregation instruction for the original data table; The coordination node obtains a first sharding key of an aggregate data table corresponding to the original data table, wherein the aggregate data table is used to record data after the data in the original data table is aggregated, and the first sharding key includes at least one first attribute label of the aggregate data table; The coordination node determines first allocation information, wherein the first allocation information is used to indicate storage nodes respectively allocated to multiple first attribute value sets corresponding to the first sharding key, and the first attribute value set consists of one attribute value of each first attribute label; The coordination node compares the first attribute value set in the first allocation information with the attribute values ​​in the data units corresponding to the original data table, so as to distribute the data units corresponding to the original data table to the corresponding storage nodes; The storage node performs aggregation processing based on the received data units.

2. The method according to claim 1, characterized in that: The first sharding key is the same as the second sharding key of the original data table, and the second sharding key includes at least one second attribute label of the original data table; The coordinating node determines first allocation information, including: The coordination node determines the first allocation information based on the second allocation information of the original data table, wherein the second allocation information is used to indicate the storage nodes respectively allocated to the multiple second attribute value sets corresponding to the second sharding key, the storage nodes indicated by the second allocation information are used to perform disk flushing, the second attribute value set consists of an attribute value for each second attribute label, and in the first allocation information and the second allocation information, the storage nodes corresponding to the same attribute value set are the same.

3. The method according to claim 1, characterized in that The first sharding key is different from the second sharding key and is a subset of the second sharding key, and the second sharding key includes at least one second attribute label of the original data table; The coordinating node determines first allocation information, including: The coordination node determines a plurality of first attribute value sets corresponding to the first sharding key and a plurality of second attribute value sets corresponding to the second sharding key, wherein the second attribute value set consists of one attribute value for each second attribute label; The coordination node determines first allocation information based on the multiple first attribute value sets and the multiple second attribute value sets.

4. The method according to claim 3, characterized in that The coordinating node determines first allocation information based on the multiple first attribute value sets and the multiple second attribute value sets, including: For each first attribute value set, the coordination node determines at least one second attribute value set including the first attribute value set, and selects one storage node from the storage nodes corresponding to the at least one second attribute value set as the storage node corresponding to the first attribute value set; The coordination node determines first allocation information based on the storage nodes corresponding to each first attribute value set.

5. The method according to claim 1, characterized in that The coordinating node determines first allocation information, including: The coordination node determines the first allocation information based on a storage node selection rule, wherein the storage node selection rule includes random selection, selection in a specified order, or selection based on resource occupancy.

6. The method according to claim 5, characterized in that The first sharding key is different from the second sharding key and is not a subset of the second sharding key, and the second sharding key includes at least one second attribute label of the original data table.

7. The method according to any one of claims 1 to 6, characterized in that: The first shard key is determined based on the aggregation instruction.

8. The method according to any one of claims 1 to 7, characterized in that: The coordination node compares the first attribute value set in the first allocation information with the attribute value in the data unit corresponding to the original data table to distribute the data unit corresponding to the original data table to the corresponding storage node, including: The coordination node compares the first attribute value set in the first allocation information and the second attribute value set in the second allocation information with the attribute values ​​in the data units corresponding to the original data table, respectively, to add aggregation marks and / or write marks to the data units corresponding to the original data table, and distributes them to the corresponding storage nodes, wherein the second allocation information is used to indicate the storage nodes respectively allocated to the multiple second attribute value sets corresponding to the second sharding key, the storage nodes indicated by the second allocation information are used to perform disk flushing, the second sharding key includes at least one second attribute tag of the original data table, the second attribute value set consists of one attribute value for each second attribute tag, the aggregation mark is used to instruct the storage node to perform aggregation processing on the data unit, and the write mark is used to instruct the storage node to perform disk flushing processing on the data unit.

9. The method according to claim 8, characterized in that The coordination node compares the first attribute value set in the first allocation information and the second attribute value set in the second allocation information with the attribute values ​​in the data units corresponding to the original data table, respectively, to add an aggregation mark and / or a write mark to the data units corresponding to the original data table, and distributes them to the corresponding storage nodes, including: The coordinating node determines, in the first allocation information, a first attribute value set that matches the attribute value of the first data unit of the received original data table and a corresponding first storage node, and determines, in the second allocation information, a second attribute value set that matches the attribute value of the first data unit of the received original data table and a corresponding second storage node; When the first storage node and the second storage node are the same storage node, the coordination node adds an aggregation mark and a write mark to the first data unit, and sends it to the first storage node; When the first storage node and the second storage node are not the same storage node, the coordination node copies the first data unit to obtain the second data unit, aggregates the first data unit with other data units that match the first attribute value set, adds the aggregation mark to the data unit obtained by the aggregation and sends it to the first storage node, and adds the write mark to the second data unit and sends it to the second storage node.

10. The method according to claim 8, characterized in that The coordination node compares the first attribute value set in the first allocation information and the second attribute value set in the second allocation information with the attribute values ​​in the data units corresponding to the original data table, respectively, to add an aggregation mark and / or a write mark to the data units corresponding to the original data table, and distributes them to the corresponding storage nodes, including: The coordinating node determines, in the first allocation information, a first attribute value set that matches the attribute value of the first data unit of the received original data table and a corresponding first storage node, and determines, in the second allocation information, a second attribute value set that matches the attribute value of the first data unit of the received original data table and a corresponding second storage node; When the first storage node and the second storage node are the same storage node, the coordination node adds an aggregation mark and a write mark to the first data unit, and sends it to the first storage node; When the first storage node and the second storage node are not the same storage node, the coordination node copies the first data unit to obtain the second data unit, adds the aggregation mark to the first data unit and sends it to the first storage node, and adds the write mark to the second data unit and sends it to the second storage node.

11. The method according to any one of claims 1 to 10, characterized in that: The storage node performs aggregation processing based on the received data units, including: The storage node applies for a first temporary cache from the first cache pool, and writes the received data unit into the first temporary cache; The storage node writes the data unit in the first temporary cache into the storage disk; The storage node applies to the second cache pool for a second temporary cache having the same space size as the first temporary cache, and releases the second temporary cache to the first cache pool; The storage node performs aggregation processing based on the data units in the first temporary cache; The storage node releases the first temporary buffer to the second buffer pool.

12. A distributed database system, characterized in that: The distributed database system includes a coordination node and multiple storage nodes; The coordination node is used to: Receive an aggregation instruction for an original data table; obtain a first sharding key of an aggregated data table corresponding to the original data table, wherein the aggregated data table is used to record data after the data in the original data table is aggregated, and the first sharding key includes at least a first attribute tag; determining first allocation information, wherein the first allocation information is used to indicate storage nodes respectively allocated to multiple first attribute value sets corresponding to the first sharding key, the first attribute value set consisting of one attribute value of each first attribute tag; comparing the first attribute value set in the first allocation information with the attribute value in the data unit corresponding to the original data table, so as to distribute the data unit corresponding to the original data table to the corresponding storage node; The storage node is used to perform aggregation processing based on the received data units.

13. The distributed database system according to claim 12, characterized in that: The first sharding key is the same as the second sharding key of the original data table, and the second sharding key includes at least one second attribute label of the original data table; The coordination node is used to: Based on the second allocation information of the original data table, the first allocation information is determined, wherein the second allocation information is used to indicate storage nodes respectively allocated to multiple second attribute value sets corresponding to the second sharding key, the storage nodes indicated by the second allocation information are used to perform disk flushing, the second attribute value set is composed of an attribute value for each second attribute label, and in the first allocation information and the second allocation information, the storage nodes corresponding to the same attribute value set are the same.

14. The distributed database system according to claim 12, characterized in that: The first sharding key is different from the second sharding key and is a subset of the second sharding key, and the second sharding key includes at least one second attribute label of the original data table; The coordination node is used to: Determine a plurality of first attribute value sets corresponding to the first sharding key and a plurality of second attribute value sets corresponding to the second sharding key, wherein the second attribute value set consists of one attribute value for each second attribute label; First allocation information is determined based on the multiple first attribute value sets and the multiple second attribute value sets.

15. The distributed database system according to claim 14, characterized in that: The coordination node is used to: For each first attribute value set, determine at least one second attribute value set that includes the first attribute value set, and select a storage node from the storage nodes corresponding to the at least one second attribute value set as the storage node corresponding to the first attribute value set; First allocation information is determined based on the storage nodes corresponding to each first attribute value set.

16. The distributed database system according to claim 12, characterized in that: The coordination node is used to: The first allocation information is determined based on a storage node selection rule, wherein the storage node selection rule includes random selection, selection in a specified order, or selection based on resource occupancy.

17. The distributed database system according to claim 16, characterized in that: The first sharding key is different from the second sharding key and is not a subset of the second sharding key, and the second sharding key includes at least one second attribute label of the original data table.

18. The distributed database system according to any one of claims 12 to 17, characterized in that: The first shard key is determined based on the aggregation instruction.

19. The distributed database system according to any one of claims 12 to 18, characterized in that: The coordination node is used to: The first attribute value set in the first allocation information and the second attribute value set in the second allocation information are respectively compared with the attribute values ​​in the data units corresponding to the original data table, so as to add aggregation marks and / or write marks to the data units corresponding to the original data table, and distribute them to the corresponding storage nodes, wherein the second allocation information is used to indicate the storage nodes respectively allocated to the multiple second attribute value sets corresponding to the second sharding key, the storage nodes indicated by the second allocation information are used to perform disk flushing, the second sharding key includes at least one second attribute tag of the original data table, the second attribute value set consists of one attribute value for each second attribute tag, the aggregation mark is used to instruct the storage node to perform aggregation processing on the data unit, and the write mark is used to instruct the storage node to perform disk flushing processing on the data unit.

20. The distributed database system according to claim 19, characterized in that: The coordination node is used to: In the first allocation information, a first attribute value set matching the attribute value of the first data unit of the received original data table is determined. and a corresponding first storage node, and determining, in the second allocation information, a second attribute value set and a corresponding second storage node that match the attribute value of the first data unit of the received original data table; When the first storage node and the second storage node are the same storage node, adding an aggregation mark and a write mark to the first data unit, and sending the first data unit to the first storage node; When the first storage node and the second storage node are not the same storage node, the first data unit is copied to obtain the second data unit, the first data unit is aggregated with other data units matching the first attribute value set, the aggregation mark is added to the data unit obtained by the aggregation and sent to the first storage node, and the write mark is added to the second data unit and sent to the second storage node.

21. The distributed database system according to claim 19, characterized in that: The coordination node is used to: In the first allocation information, a first attribute value set matching the attribute value of the first data unit received in the original data table and a corresponding first storage node are determined, and in the second allocation information, a second attribute value set matching the attribute value of the first data unit received in the original data table and a corresponding second storage node are determined; When the first storage node and the second storage node are the same storage node, adding an aggregation mark and a write mark to the first data unit, and sending the first data unit to the first storage node; When the first storage node and the second storage node are not the same storage node, the first data unit is copied to obtain the second data unit, the aggregation mark is added to the first data unit and sent to the first storage node, and the write mark is added to the second data unit and sent to the second storage node.

22. The distributed database system according to any one of claims 12 to 21, characterized in that: The storage node is used for: Applying for a first temporary cache from a first cache pool, and writing the received data unit into the first temporary cache; Writing the data unit in the first temporary cache into the storage disk; Applying for a second temporary cache having the same space size as the first temporary cache from the second cache pool, and releasing the second temporary cache to the first cache pool; performing aggregation processing based on the data units in the first temporary cache; The first temporary buffer is released to the second buffer pool.

23. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 11.

25. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 11.

Images