Data aggregation processing method, system and equipment

Abstract

The embodiment of the invention provides a data aggregation processing method, system and equipment, and relates to the technical field of databases. In the embodiment of the invention, the data aggregation processing is carried out through the distributed database system, so that the data storage and the data aggregation processing are both completed by the distributed database, and an additional processing system does not need to be built to carry out the data aggregation processing; and the data does not need to be copied and sent to a distributed database system and an additional processing system. In this way, occupation of transmission resources can be reduced.

Description

[0001] This disclosure claims priority to Chinese patent application No. 202311125364.8 filed on September 1, 2023, with invention name “A data processing method, device and computing device cluster”, the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0002] The present disclosure relates to the field of database technology, and in particular to a method, system and device for data aggregation processing. Background Art

[0003] With the rapid development of technologies such as the Internet and the Internet of Things, a large amount of "time series data" needs to be stored in many businesses. For example, "time series data" can be temperature data collected periodically by a temperature sensor, or it can be a user's browsing history, etc. Distributed database systems can store a large amount of time series data. In practical applications, it is often necessary to aggregate time series data. Aggregation processing is to perform specified calculations (for example, summation, etc.) on time series data that meet certain conditions (for example, time series data belonging to the same time period).

[0004] In the related technology, outside the distributed database system, an additional processing system is built to perform aggregation processing. The management platform can send time series data to the distributed database system and the processing system respectively. The distributed database system performs conventional storage on the time series data, and the processing system performs aggregation processing on the time series data.

[0005] In the above technology, the same data needs to be copied and sent to multiple entities for processing, which will waste a lot of transmission resources. Summary of the invention

[0006] The present disclosure provides a method, system and device for data aggregation processing, which can reduce the waste of transmission resources.

[0007] In the first aspect, a method for data aggregation processing is provided, which is applied to a distributed database system, wherein the distributed database system includes a coordination node and multiple storage nodes. The processing process of the method may include: the coordination node receives an aggregation instruction for an original data table; then, the coordination node may obtain a first shard key of an aggregate data table corresponding to the original data table, wherein the aggregate data table is used to record data after the data in the original data table is aggregated, and the first shard key includes at least one first attribute tag of the aggregate data table; further, the coordination node may determine first allocation information, wherein the first allocation information is used to indicate storage nodes respectively allocated to a plurality of first attribute value sets corresponding to the first shard key, and the first attribute value set is composed of an attribute value of each first attribute tag; the coordination node compares the first attribute value set in the first allocation information with the attribute value in the data unit corresponding to the original data table, so as to distribute the data unit corresponding to the original data table to the corresponding storage node; the storage node performs aggregation processing based on the received data unit.

[0008] The data unit corresponding to the original data table may be the data unit of the original data table directly received by the coordination node, or may be the data unit obtained after the coordination node performs certain processing (for example, deleting some fields, aggregating, etc.) on the data unit of the received original data table.

[0009] In this way, data aggregation processing is performed through a distributed database system. Data storage and data aggregation processing are all completed by the distributed database. There is no need to build an additional processing system for data aggregation processing, and there is no need to copy the data and send it to the distributed database system and the additional processing system respectively. This method can reduce the occupation of transmission resources.

[0010] In one possible implementation, the first sharding key is the same as the second sharding key of the original data table, and the second sharding key includes at least one second attribute tag of the original data table. In this case, the coordination node can determine the first allocation information based on the second allocation information of the original data table. The second allocation information is used to indicate the storage nodes respectively allocated to the multiple second attribute value sets corresponding to the second sharding key, the storage nodes indicated by the second allocation information are used to perform disk flushing, the second attribute value set is composed of one attribute value for each second attribute tag, and in the first allocation information and the second allocation information, the storage nodes corresponding to the same attribute value set are the same.

[0011] In this way, the first allocation information for aggregation directly reuses the second allocation information for disk flushing, which can simplify processing, improve efficiency, and reduce the occupation of processing resources.

[0012] In a possible implementation, the first sharding key and the second sharding key are different and are subsets of the second sharding key, and the second sharding key includes at least one second attribute tag of the original data table. In this case, the coordination node can determine multiple first attribute value sets corresponding to the first sharding key and multiple second attribute value sets corresponding to the second sharding key, and then determine the first allocation information based on the multiple first attribute value sets and the multiple second attribute value sets. The second attribute value set consists of one attribute value for each second attribute tag.

[0013] In one possible implementation, for each first attribute value set, the coordination node can determine at least one second attribute value set that includes the first attribute value set, and select a storage node from the storage nodes corresponding to the at least one second attribute value set as the storage node corresponding to the first attribute value set; further, the coordination node can determine the first allocation information based on the storage nodes corresponding to each first attribute value set.

[0014] In this way, when the first shard key is a subset of the second shard key, the data that needs to be flushed to disk in the storage node can be aggregated as much as possible, reducing additional data transmission and improving network resource utilization.

[0015] In a possible implementation, the coordination node may determine the first allocation information based on a storage node selection rule, wherein the storage node selection rule may include random selection, selection in a specified order, or selection based on resource occupancy, etc.

[0016] This method of determining the first allocation information may be performed in any case, and the first shard key and the second shard key may not satisfy any relationship.

[0017] This provides greater flexibility when allocating storage nodes for aggregation processing.

[0018] In a possible implementation, the first shard key is different from the second shard key and is not a subset of the second shard key. In this case, the coordination node may determine the first allocation information based on the storage node selection rule.

[0019] In this way, for the case where the first shard key and the second shard key are the same, and the first shard key is a subset of the second shard key, you can choose the above more resource-saving processing method. For the case where the first shard key and the second shard key are different and are not a subset of the second shard key, you can use this more universal and flexible processing method.

[0020] In a possible implementation, the first shard key is determined based on the aggregation instruction. Alternatively, the first shard key may be obtained in other ways, for example, the configuration information carrying the first shard key may be manually stored in the coordination node, and the first shard key may be obtained from the configuration information. For another example, the first shard key may be set by a create shard key instruction, that is, the first shard key is obtained from the create shard key instruction.

[0021] The above-mentioned setting of the first shard key through the aggregation instruction can complete the aggregation task and the setting of the shard key through only one aggregation instruction, which helps to improve the operation efficiency.

[0022] In one possible implementation, the processing of distributing data units can be as follows: the coordinating node can compare the first attribute value set in the first allocation information and the second attribute value set in the second allocation information with the attribute values ​​in the data units corresponding to the original data table, respectively, to add aggregation marks and / or write marks to the data units corresponding to the original data table, and distribute them to the corresponding storage nodes.

[0023] Among them, the aggregation mark is used to instruct the storage node to perform aggregation processing on the data unit, and the write mark is used to instruct the storage node to perform disk flushing processing on the data unit.

[0024] In this way, through aggregation marking and write marking, the storage node can clearly distinguish which data units need to be flushed to disk, which data units need aggregation processing, and which data units need both aggregation processing and disk flushing processing, so as to improve the processing efficiency of the storage node.

[0025] In one possible implementation, the processing of distributing data units may be as follows: the coordination node may determine, in the first allocation information, a first attribute value set that matches the attribute value of the first data unit received in the original data table and the corresponding first storage node, and may determine, in the second allocation information, a second attribute value set that matches the attribute value of the first data unit received in the original data table and the corresponding second storage node; when the first storage node and the second storage node are the same storage node, the coordination node may add an aggregation mark and a write mark to the first data unit, and send it to the first storage node; when the first storage node and the second storage node are not the same storage node, the coordination node may copy the first data unit to obtain the second data unit, aggregate the first data unit with other data units that match the first attribute value set, add an aggregation mark to the data unit obtained by the aggregation process and send it to the first storage node, and add a write mark to the second data unit and send it to the second storage node.

[0026] In this way, the first-level aggregation can be performed on the coordination node, and then the second-level aggregation can be performed on the storage node. In some application scenarios where the pressure on the coordination node is relatively small, the processing resources of the coordination node can be well utilized to reduce the processing burden of the storage node.

[0027] In one possible implementation, the processing of distributing data units may be as follows: the coordination node may determine, in the first allocation information, a first set of attribute values ​​that matches the attribute values ​​of the first data unit received in the original data table and the corresponding first storage node, and may determine, in the second allocation information, a second set of attribute values ​​that matches the attribute values ​​of the first data unit received in the original data table and the corresponding second storage node; when the first storage node and the second storage node are the same storage node, the coordination node may add an aggregation mark and a write mark to the first data unit, and send it to the first storage node; when the first storage node and the second storage node are not the same storage node, the coordination node may copy the first data unit to obtain a second data unit, add an aggregation mark to the first data unit and send it to the first storage node, and add a write mark to the second data unit and send it to the second storage node.

[0028] In this way, the coordination node only performs distribution work and does not participate in aggregation processing. In some application scenarios where the coordination node is under heavy pressure, it can prevent the coordination node from being stuck or even crashing.

[0029] In a possible implementation, the storage node may apply for a first temporary cache from the first cache pool, and write the received data unit into the first temporary cache; then, the storage node may write the data unit in the first temporary cache into the storage disk; further, the storage node may apply for a second temporary cache having the same space size as the first temporary cache from the second cache pool, and release the second temporary cache to the first cache pool; thereafter, the storage node performs aggregation processing based on the data unit in the first temporary cache; the storage node releases the first temporary cache to the second cache pool.

[0030] In this way, the first cache pool is supplemented by the second temporary cache, so that memory leakage in the first cache pool can be prevented.

[0031] In a second aspect, a distributed database system is provided. The distributed database system includes a coordination node and multiple storage nodes. The coordination node and the multiple storage nodes are used to execute the method provided by the first aspect and possible implementations thereof.

[0032] According to a third aspect, a device for data aggregation processing is provided, the device comprising at least one module, and the at least one module is used to implement the method provided by the first aspect and its possible implementation manner.

[0033] In a fourth aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the method provided by the above-mentioned first aspect and its possible implementation methods.

[0034] In a fifth aspect, a computer device is provided, the computer device comprising a memory and a processor, the memory being used to store computer instructions. The processor executes the computer instructions stored in the memory, so that the computer device executes the method provided in the first aspect and possible implementations thereof.

[0035] In a sixth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program code. When the computer program code is executed by a computer device, the computer device executes the method provided by the first aspect and its possible implementation methods.

[0036] In a seventh aspect, a computer program product comprising instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the method provided by the first aspect and possible implementations thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a structural diagram of a distributed database system provided by an embodiment of the present disclosure;

[0038] Figure 2 is a flow chart of a method for data aggregation processing provided by an embodiment of the present disclosure;

[0039] Figure 3 is a schematic diagram of allocation information provided by an embodiment of the present disclosure;

[0040] Figure 4 is a schematic diagram of a dual-pool exchange process provided by an embodiment of the present disclosure;

[0041] Figure 5 is a flow chart of a method for data aggregation processing provided by an embodiment of the present disclosure;

[0042] Figure 6 is a schematic diagram of allocation information provided by an embodiment of the present disclosure;

[0043] Figure 7 is a flow chart of a method for data aggregation processing provided by an embodiment of the present disclosure;

[0044] Figure 8 is a schematic diagram of allocation information provided by an embodiment of the present disclosure;

[0045] Fig. 9is a structural diagram of a data aggregation processing device provided by an embodiment of the present disclosure;

[0046] Fig.10 is a structural diagram of a data aggregation processing device provided by an embodiment of the present disclosure;

[0047] Fig.11 is a schematic diagram of a computing device provided by an embodiment of the present disclosure;

[0048] Fig.12 is a schematic diagram of a computing device cluster provided by an embodiment of the present disclosure;

[0049] Fig.13 It is a schematic diagram of a computing device cluster provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0051] The concepts involved in this disclosure are explained below:

[0052] Time Series Data

[0053] Time series data is a series of data generated continuously over time. Time series data carries a timestamp (used to identify the time when the time series data was sent). For example, time series data can be data periodically collected by IoT sensors, transaction data in the financial market, and so on.

[0054] polymerization

[0055] Aggregation can also be called aggregate calculation or aggregate processing. It is to divide the data in the database into several groups according to certain rules, perform specified calculations (for example, sum, etc.) on multiple rows of data in each group, and obtain a calculation result for each group as the aggregation result. Aggregation of time series data can be to perform specified calculations on time series data that meet certain time conditions (for example, time series data belonging to the same time period).

[0056] Original data table

[0057] The original data table is used to record the original data that has not been aggregated. The original data can be data directly obtained through data collection or data generated in real time during business execution. A row of data in the original data table can be called a data unit.

[0058] The embodiment of the present disclosure takes an original data table for recording time series data as an example to illustrate the solution.

[0059] Aggregate Data Table

[0060] An aggregate data table is a data table obtained by aggregating the data in the original data table. A row of data in an aggregate data table can be called an aggregate data unit. When the data stored in the original data table is time series data, the data in the aggregate data table also belongs to time series data.

[0061] Shard key

[0062] The shard key consists of at least one attribute tag of the data table. The data units can be grouped based on the attribute value corresponding to the at least one attribute tag in the data unit. The purpose of grouping can be to partition and store multiple data units or to group and aggregate them. The shard key of the aggregate data table is used to group and aggregate the data units. The shard key of the original data table (assuming that the original data table is not generated by aggregation) is used to partition and store the data units.

[0063] Take an example to illustrate the usage of shard keys: for example, the two data units of an original data table are M and N, and the shard keys of the original data table are tagA and tagB. tagA and tagB are two attribute labels in the original data table. The attribute values ​​corresponding to tagA and tagB in data unit M are am and bm, and the attribute values ​​corresponding to tagA and tagB in data unit N are an and bn. If am is equal to an and bm is equal to bn, then data units M and N should be grouped together and stored in the same path. If am is not equal to an or bm is not equal to bn, then data units M and N should not be grouped together and should be stored in different paths.

[0064] The attribute tags or tags used by the shard key of the data table can be set manually or by using the create shard key command. The format of the create shard key command is as follows:

[0065] CREAT MEASUREMENT table_name WITH SHARDKEY tagA,tagB……

[0066] In this instruction, "CREATE MEASUREMENT" is the instruction name, and "WITH SHARDKEY" is used to extract the attribute labels contained in the shard key. The characters of these two fields are fixed and can be called fixed fields.

[0067] "table_name" and "tagA, tagB..." are parameter fields, and the content in the parameter fields can be set according to actual needs. "table_name" is the name of the data table for creating the shard key, and "tagA, tagB..." are the attribute tags that make up the shardkey.

[0068] Aggregation directives

[0069] The aggregation instruction is used to instruct the generation of an aggregated data table based on the original data table, that is, to aggregate the data units of the original data table received after the aggregation instruction to obtain aggregated data units, and to use the aggregated data units to combine into an aggregated data table. The data in the aggregated data table also belongs to time series data.

[0070] Aggregate directives can take the following forms:

[0071] CREATE STREAM stream_name INTO agg_table_name AS SELECT sum("value_tag")AS"agg_value_tag"FROM table_name GROUP BY TIME(time_1),"group_tagA","group_tagB"DELAY time_2

[0072] In this instruction, "CREATE STREAM" is the instruction name, "INTO" is used to extract the table name of the aggregate data table, "ASSELECT" is used to extract the attribute label of the numeric field of the original data table, and this numeric field is the input data of the aggregate calculation. The subsequent "AS" is used to extract the attribute label of the numeric field of the aggregate data table, and this numeric field is the aggregation result. "FROM" is used to extract the table name of the original data table, "GROUP BY" is used to extract the aggregation condition, "TIME" is used to extract the duration corresponding to the time window, and "DELAY" is used to extract the delay duration. The characters of these fields are fixed and can be called fixed fields.

[0073] "stream_name", "agg_table_name", "sum("value_tag")", "agg_value_tag", "table_name", "time_1", "group_tagA", "group_tagB" and "time_2" are parameter fields. The content in the parameter fields can be set according to actual needs. The following is an introduction to the fields one by one:

[0074] "stream_name" is the task name of the streaming aggregation task, "agg_table_name" is the table name of the aggregation data table, which is the data table that records the result data after the aggregation calculation of the time series data, and "table_name" is the table name of the original data table, which is the data table that records the time series data.

[0075] "TIME(time_1), "group_tagA", "group_tagB"" can be generally referred to as the aggregation condition. Among them, "time_1" is the duration of the time window, and the time window is the time unit of the aggregation processing. For example, when time_1 is 1m, it means that the time window is created with 1 minute as the time unit, which means that the data units within 1 minute are aggregated to obtain aggregated data units. "group_tagA" and "group_tagB" are the attribute labels of the original data table, and are also the attribute labels of the columns to be added to the aggregated data table (here, 2 attribute labels are taken as an example, and the number can be any positive integer). When the aggregated data table is generated, these two columns of data in the original data table are added to the aggregated data table. Based on the setting of the aggregation instruction parsing logic, "group_tagA" and "group_tagB" can be set as the sharding key of the aggregated data table by default. The above aggregation condition means: based on the duration of the time window "time_1", for multiple data units within a time window, data units with the same attribute values ​​corresponding to "group_tagA" and "group_tagB" are grouped together, and data units in the same group are aggregated to obtain an aggregated data unit.

[0076] "time_2" is the delay duration, which is the allowed delay duration for data unit transmission. For example, the time window duration is 5 minutes, the delay duration is 1m (1 minute), and 9:00-9:05 is a time window. When a data unit with a timestamp of 9:04 is received at 9:05:30, and the data unit is received before 9:06 (9:05 plus 1 minute), the data unit can be aggregated with other data units in the 9:00-9:05 time window.

[0077] "value_tag" is the attribute tag corresponding to the numeric field of the original data table (input data for aggregate calculation). "sum()" indicates the algorithm for aggregate calculation. Here, addition is used as an example, but other algorithms can also be used. In addition to "sum", calculation tasks can also be average, maximum, minimum, or variance, etc. Technical personnel can set them according to actual business needs. "agg_value_tag" is the attribute tag corresponding to the numeric field of the aggregate data table (aggregation result).

[0078] In order to explain the above aggregation instructions in more detail, an example is given below:

[0079] Assume the aggregation instruction is:

[0080] CREATE STREAM test INTO table_2AS SELECT sum("value")AS"agg_value"FROM table_1GROUP BY TIME(1m),"tagA","tagB"DELAY 20s

[0081] The table name corresponding to the original data table is table_1. Assume that the set of time series data in a certain time window in table_1 is as shown in Table 1:

[0082] Table 1

[0083] tagA tagB value a1 b1 11 a1 b1 35 a1 b2 25 a1 b2 8 a2 b1 71 a2 b1 19 a2 b2 33 a2 b2 52

[0084] According to the values ​​of tagA and tagB in the aggregation condition, the time series data can be divided into 4 groups, namely: the first group, the data units whose attribute value corresponding to tagA is a1 and the attribute value corresponding to tagB is b1; the second group, the data units whose attribute value corresponding to tagA is a1 and the attribute value corresponding to tagB is b2; the third group, the data units whose attribute value corresponding to tagA is a2 and the attribute value corresponding to tagB is b1; the fourth group, the data units whose attribute value corresponding to tagA is a2 and the attribute value corresponding to tagB is b2. {a1, b1}, {a1, b2}, {a2, b1} and {a2, b2} are four attribute value sets of table_1, and each attribute value set corresponds to a group.

[0085] The value attribute values ​​of the four groups of data units are summed up to obtain four aggregated data units, which are recorded in the aggregated data table, that is, table_2, as shown in Table 2:

[0086] Table 2

[0087] tagA tagB value a1 b1 46 a1 b2 33 a2 b1 90 a2 b2 85

[0088] The embodiment of the present disclosure provides a method for data processing, which can be applied to a distributed database system. The distributed database system may include at least one coordination node 110 (also referred to as a computing node), multiple storage nodes 120 and a storage disk 130, and may also include a metadata node 140, a function node 150, etc. The structure of the distributed database system is as follows: Figure 1 As shown, Figure 1 All storage nodes 120 in the system are connected to the metadata node 140, and all storage nodes 120 are also connected to the function node 150. Figure 1 Only one connection relationship is drawn for example. Similarly, Figure 1All coordination nodes 110 are connected to metadata nodes 140, and all coordination nodes 110 are also connected to function nodes 150. Figure 1 Only one connection relationship is drawn for example. The following is an introduction to each component of the distributed database system:

[0089] The coordination node 110 may be an independent computer device, a virtual machine, or a computer cluster, etc. The coordination node 110 may receive time series data and parse, convert, preprocess, etc. the time series data, and may also determine the storage node corresponding to each time series data based on a certain distribution rule, and send each time series data to the corresponding storage node. The distribution rule may be automatically generated by the coordination node or manually set.

[0090] The storage node 120 can be an independent computer device, a virtual machine, or a computer cluster. The storage node 120 is used to organize the time series data for disk flushing, that is, to store the time series data in the storage disk, and can also be used to perform other processing on the time series data, such as aggregation processing. The distributed database system includes multiple storage nodes, and different storage nodes store the time series data they receive in different paths of the storage disk.

[0091] The storage disk 130 is an underlying storage medium for storing data. There may be multiple storage disks 130 , which may be located in one device or multiple devices.

[0092] The metadata node 140 may be used to store various types of metadata, such as information recording the above-mentioned distribution rules.

[0093] Function node 150 may be used to store functions edited by a user.

[0094] Distributed database systems can be used to store time series data. In related technologies, real-time aggregation calculations of time series data are generally performed in big data components. The present disclosure provides a method for data aggregation processing, which is applied in a distributed database system. In this way, the distributed database system can not only store time series data, but also perform real-time aggregation processing on time series data, saving transmission resources. Figure 2 As shown, the processing flow of the method includes the following steps:

[0095] Step 201: The coordination node receives an aggregation instruction for the original data table.

[0096] The aggregation instruction is used to instruct the generation of an aggregated data table based on the original data table, that is, to aggregate the data units of the original data table received subsequently to obtain aggregated data units, and to use the aggregated data units to combine into an aggregated data table. The data in the aggregated data table also belongs to time series data.

[0097] Step 202: The coordination node obtains the first shard key of the aggregate data table and obtains the second shard key of the original data table.

[0098] The shard key is a set of several attribute tags. Both the aggregate data table and the original data table have corresponding shard keys. For the aggregate data table, you can set the first shard key through the shard key instruction, or you can directly set all the attribute tags in the aggregation conditions in the aggregation instruction as the first shard key. The attribute tags included in the first shard key can be called the first attribute tags. For the original data table, you can set the second shard key corresponding to the original data table through the shard key instruction. All the attribute tags included in the second shard key are called second attribute tags.

[0099] Step 203: When the first sharding key and the second sharding key are the same, the coordination node determines the first allocation information based on the second allocation information of the original data table.

[0100] The first allocation information is used to indicate how the data unit should be distributed when performing aggregation, and records the storage nodes respectively allocated to the various first attribute value sets corresponding to the first sharding key. The allocated storage nodes are used to perform aggregation processing, and the first attribute value set consists of one attribute value for each first attribute label. The second allocation information is used to indicate how the data unit should be distributed when performing disk flushing, and records the storage nodes respectively allocated to the various second attribute value sets corresponding to the second sharding key. The allocated storage nodes are used to perform disk flushing processing, and the second attribute value set consists of one attribute value for each second attribute label.

[0101] The second allocation information is described in detail below:

[0102] The coordination node can generate the second allocation information of the original data table after receiving the creation shard key instruction corresponding to the original data table and completing the setting of the second shard key. Specifically, based on the different values ​​corresponding to the second attribute label of the second shard key of the original data table, the attribute values ​​corresponding to different attribute labels can be combined respectively, thereby generating multiple second attribute value sets. For example, the second attribute label includes 3 attribute labels, each of which has two different attribute values, and attribute label A, attribute label B, and attribute label C each take an attribute value to form a second attribute value set, and a total of 8 second attribute value sets can be generated.

[0103] The coordination node allocates storage nodes to all second attribute value sets respectively to obtain second allocation information. For example, storage node 1 is allocated to {a1, b1}, storage node 2 is allocated to {a1, b2}, and so on, until storage nodes are allocated to all second attribute value sets. The storage nodes allocated to each second attribute value set may be the same or different.

[0104] In the case where there are multiple coordination nodes, after setting the second allocation information, the coordination node can send the second allocation information to the metadata node for storage. Other coordination nodes can periodically access the metadata node. After the second allocation information is stored in the metadata node, other coordination nodes can read the second allocation information for subsequent data distribution.

[0105] After executing step 202 to obtain the first sharding key of the aggregate data table and the second sharding key of the original data table, the coordination node can determine whether the first sharding key and the second sharding key are completely equal, that is, determine whether the first attribute label in the first sharding key and the second attribute label in the second sharding key are completely equal. If they are completely equal, the process of determining the first allocation information can be as follows:

[0106] The first sharding key and the second sharding key are completely identical, and the multiple first attribute value sets and the multiple second attribute value sets are also completely identical. When allocating storage nodes to each first attribute value set, the storage nodes corresponding to the same first attribute value set and the second attribute value set can be set to be the same, so as to obtain first allocation information. In this way, the first allocation information can be considered to be the same as the second allocation information.

[0107] The second allocation information and the first allocation information are further described below by way of example:

[0108] The data of a time window of a distributed database is shown in Table 1 above. Assuming that the second sharding key is tagA and tagB, the value of tagA is a1 or a2, and the value of tagB is b1 or b2, four second attribute value sets {a1, b1}, {a1, b2}, {a2, b1} and {a2, b2} can be obtained. A storage node is allocated to each second attribute value set to obtain the second allocation information, such as Figure 3 As shown, assuming that the second allocation information is {a1, b1}→storage node 1, {a1, b2}→storage node 2, {a2, b1}→storage node 3 and {a2, b2}→storage node 4. Assuming that the first sharding key is tagA and tagB, the first attribute value set {a1, b1}, {a1, b2}, {a2, b1} and {a2, b2} can be obtained, and then the first allocation information is {a1, b1}→storage node 1, {a1, b2}→storage node 2, {a2, b1}→storage node 3 and {a2, b2}→storage node 4. It can be considered that the first allocation information is the same as the second allocation information.

[0109] In the case where there are multiple coordination nodes, after setting the first allocation information, the coordination node may send the first allocation information to the metadata node for storage. Other coordination nodes may obtain the first allocation information by periodically accessing the metadata node for subsequent data distribution.

[0110] Step 204: The coordination node distributes the received data units of the original data table to corresponding storage nodes based on the first allocation information.

[0111] After receiving the data unit, the coordination node searches for the first attribute value set and the corresponding storage node included in the data unit in the first allocation information, and adds an aggregation mark and a write mark to the data unit. The write mark is used to indicate that the data unit needs to be flushed, and the aggregation mark is used to indicate that the data unit needs to be aggregated. Then the marked data unit is sent to the storage node.

[0112] The following is further described by way of example (the example of step 203 is continued):

[0113] The data unit with the attribute value {a1, b1} corresponding to the second attribute label is distributed to storage node 1, the data unit with the attribute value {a1, b2} corresponding to the second attribute label is distributed to storage node 2, the data unit with the attribute value {a2, b1} corresponding to the second attribute label is distributed to storage node 3, and the data unit with the attribute value {a2, b2} corresponding to the second attribute label is distributed to storage node 4. That is, the first row of data and the second row of data in Table 1 are sent to storage node 1, the third row of data and the fourth row of data in Table 1 are sent to storage node 2, the fifth row of data and the sixth row of data in Table 1 are sent to storage node 3, and the seventh row of data and the eighth row of data in Table 1 are sent to storage node 4.

[0114] Step 205: The storage node writes the received data unit into the storage disk, and performs aggregation processing based on the received data unit.

[0115] The storage node establishes a first cache pool (also called a write cache pool) and a second cache pool (also called a computing cache pool). When the storage node receives a data unit with an aggregation tag and a write tag, it applies to the write cache pool for a first temporary cache and writes the received data unit into the first temporary cache. The storage node then writes the data unit in the first temporary cache to the storage disk. After writing, the storage node applies to the computing cache pool for a second temporary cache with the same space size as the first temporary cache, and releases the second temporary cache to the write cache pool. The storage node performs aggregation processing based on the data unit in the first temporary cache, writes the aggregation result (the aggregation result may be an intermediate aggregation result or a final aggregation result) into the aggregation cache, and then releases the first temporary cache to the computing cache pool. In this way, for the case where the aggregation processing algorithm is addition, it is equivalent to adding the attribute value used for aggregation in the data unit to the previously calculated cumulative value each time a data unit is received to obtain a new cumulative value. When the aggregation end condition is met, that is, when the delay time (carried in the aggregation instruction) passes after the end time point of the time window, the storage node writes the aggregation result in the aggregation cache to the storage disk, such as Figure 4 shown.

[0116] Due to the existence of the delay mechanism, the aggregation cache in the storage node may maintain the aggregation results corresponding to multiple time windows (such as the above-mentioned cumulative value) at the same time. For example, each minute is a time window, and 20s (20 seconds) is the delay length. In the time period from 9:01 to 9:01:20, the aggregation cache simultaneously maintains the aggregation results of the 9:00-9:01 time window and the 9:01-9:02 time window. In this time period, if a data unit with a timestamp in the 9:00-9:01 time window is received, the aggregation results corresponding to the 9:00-9:01 time window in the aggregation cache can be accumulated to update the aggregation results. If a data unit with a timestamp in the 9:01-9:02 time window is received, the aggregation results corresponding to the 9:01-9:02 time window in the aggregation cache can be accumulated to update the aggregation results. At 9:01:20, the aggregation results corresponding to the 9:00-9:01 time window in the aggregation cache are determined as the final aggregation results and written to the storage disk.

[0117] Above Figure 2 The processing flow is applicable to the case where the first sharding key of the aggregate data table is exactly the same as the second sharding key of the original data table. For the case where the first sharding key and the second sharding key are different, and the first attribute label corresponding to the first sharding key is a subset of the second attribute label corresponding to the second sharding key, the following can be used Figure 5 The processing flow shown includes the following steps:

[0118] Step 501: The coordination node receives an aggregation instruction for the original data table.

[0119] The specific processing of this step is the same as step 201, and the relevant description of step 201 can be referred to.

[0120] Step 502: The coordination node obtains the first shard key of the aggregate data table and obtains the second shard key of the original data table.

[0121] The specific processing of this step is the same as step 202, and the relevant description of step 202 can be referred to.

[0122] Step 503: When the first shard key is a subset of the second shard key, the coordination node determines a plurality of first attribute value sets corresponding to the first shard key and a plurality of second attribute value sets corresponding to the second shard key.

[0123] The coordination node determines that the first shard key is a subset of the second shard key, that is, the first shard key is different from the second shard key, and the second attribute tag corresponding to the second shard key completely includes the first attribute tag corresponding to the first shard key. The first shard key corresponds to multiple first attribute value sets, and the first attribute value set consists of one attribute value for each first attribute tag; the second shard key corresponds to multiple second attribute value sets, and the second attribute value set consists of one attribute value for each second attribute tag.

[0124] Step 504: The coordination node determines first allocation information based on the multiple first attribute value sets and the multiple second attribute value sets.

[0125] The coordination node obtains the second allocation information of the distributed database, that is, obtains the storage node corresponding to each second attribute value set. Based on the first sharding key, the coordination node obtains multiple first attribute value sets. For each first attribute value set, at least one second attribute value set including the first attribute value set can be found. Then, at least one storage node corresponding to at least one second attribute value set can be determined, and a storage node is selected from the at least one storage node. The selection method can be random selection, selection in a specified order, or selection based on a load balancing principle. Then, the selected storage node can be assigned to the corresponding first attribute value set, and the first allocation information can be obtained.

[0126] The following is an example:

[0127] The data of a time window of a distributed database is shown in Table 1 above. Assuming that the second sharding key is tagA and tagB, the value of tagA is a1 or a2, and the value of tagB is b1 or b2, four second attribute value sets {a1, b1}, {a1, b2}, {a2, b1} and {a2, b2} can be obtained. A storage node is allocated to each second attribute value set to obtain the second allocation information, such as Figure 6As shown, assume that the second allocation information is {a1, b1}→storage node 1, {a1, b2}→storage node 2, {a2, b1}→storage node 3 and {a2, b2}→storage node 4. Assume that the first shard key is tagA, and two first attribute value sets {a1} and {a2} can be obtained. The second attribute value sets that include the first attribute value set {a1} are {a1, b1} and {a1, b2}. Among the storage nodes 1 and 2 corresponding to {a1, b1} and {a1, b2}, any one is selected as the storage node corresponding to {a1}. Assume that storage node 1 is selected as the storage node corresponding to {a1}. The second attribute value set including the first attribute value set {a2} includes {a2, b1} and {a2, b2}. Among the storage nodes 3 and 4 corresponding to {a2, b1} and {a2, b2}, one is randomly selected as the storage node corresponding to {a2}. Suppose that storage node 3 is selected as the storage node corresponding to {a2}. Then, the first allocation information is {a1}→storage node 1 and {a2}→storage node 3.

[0128] Step 505: The coordination node distributes the data units related to the original data table to the corresponding storage nodes based on the first allocation information and the second allocation information.

[0129] Based on the first allocation information and the second allocation information, the coordination node can add an aggregation mark and / or a write mark to the data units related to the original data table, and distribute them to the corresponding storage nodes. The aggregation mark is used to instruct the storage node to perform aggregation processing on the data unit. The write mark is used to instruct the storage node to perform disk flushing processing on the data unit. The data units related to the original data table can be data units of the original data table directly received by the coordination node, or they can be data units obtained after the coordination node performs certain processing on the data units of the received original data table (for example, deleting some fields, aggregation processing, etc.).

[0130] The above-mentioned process of deleting some fields may be as follows: for a data unit with an aggregation mark added but no write mark added, before sending the data unit, the attribute values ​​(ie, numerical fields) that need to be aggregated are retained in the data unit and other attribute values ​​are deleted.

[0131] The specific steps of distributing data units will be described in detail later. The scheme of coordinating nodes participating in partial aggregation processing will also be described in detail later.

[0132] Step 506: The storage node performs disk flushing, aggregation, and other processing on the received data units.

[0133] After receiving the data unit, the storage node processes the data unit according to the aggregation mark and write mark carried by the data unit. The following describes the processing methods corresponding to the three different marking situations:

[0134] Case 1: For a data unit that carries both a write mark and an aggregation mark, the processing method is the same as step 205.

[0135] In the second case, for data units that only carry write marks, when the storage node receives the data unit, it applies for a third temporary cache from the write cache pool, writes the received data unit into the third temporary cache, and then the storage node writes the data unit in the third temporary cache into the storage disk, and then releases the third temporary cache to the write cache pool.

[0136] Case 3: For data units that only carry aggregation marks, after receiving the data unit, the storage node applies for the fourth temporary cache from the computing cache pool, writes the received data unit into the fourth temporary cache, performs aggregation processing based on the data units in the fourth temporary cache, writes the aggregation result into the aggregation cache, and then releases the fourth temporary cache to the computing cache pool. When the aggregation end condition is met, the storage node writes the aggregation result in the aggregation cache into the storage disk.

[0137] The disclosed embodiment also provides another data aggregation processing method, which has a wide range of applications and can be applied to any situation where the first sharding key and the second sharding key are different. This processing method can also be used when the first sharding key is a subset of the second sharding key. The processing flow of this method can be as follows: Figure 7 As shown, the following steps are included:

[0138] Step 701: The coordination node receives an aggregation instruction for the original data table.

[0139] The specific processing of this step is the same as step 201, and the relevant description of step 201 can be referred to.

[0140] Step 702: The coordinating node obtains the first shard key of the aggregate data table.

[0141] The specific processing of this step is the same as step 202, and the relevant description of step 202 can be referred to.

[0142] Step 703: The coordinating node determines first allocation information based on the first sharding key and the storage node selection rule.

[0143] The coordination node obtains the first attribute label and the second attribute label corresponding to the first sharding key and the second sharding key. There are two situations: one is that the first attribute label and the second attribute label are completely different, and the other is that the first attribute label and the second attribute label have an intersection but the first attribute label is not completely contained in the second attribute label. For these two situations, the first allocation information can be determined based on the first sharding key. The specific processing process can be to determine multiple first attribute value sets corresponding to the first sharding key, allocate a storage node for each first attribute value set, and the storage node selection rule can be random selection, selection in a specified order, or selection based on a load balancing principle.

[0144] The following is an example:

[0145] The data of a time window of a distributed database is shown in Table 3. Assuming that the second sharding key is tagA and tagB, the value of tagA is a1 or a2, and the value of tagB is b1 or b2, four second attribute value sets {a1, b1}, {a1, b2}, {a2, b1} and {a2, b2} can be obtained. A storage node is allocated to each second attribute value set to obtain the second allocation information, such as Figure 8 As shown, assuming that the second allocation information is {a1, b1}→storage node 1, {a1, b2}→storage node 2, {a2, b1}→storage node 3 and {a2, b2}→storage node 4. Assuming that the first sharding keys are tagB and tagC, the value of tagB is b1 or b2, and the value of tagC is c1 or c2, four first attribute value sets {b1, c1}, {b1, c2}, {b2, c1} and {b2, c2} can be obtained. A storage node is allocated to each attribute value set to obtain the first allocation information. Assuming that the first allocation information is {b1, c1}→storage node 1, {b1, c2}→storage node 2, {b2, c1}→storage node 3 and {b2, c2}→storage node 4.

[0146] Table 3

[0147] tagA tagB tagC value a1 b1 c1 11 a1 b1 c2 35 a1 b2 c1 25 a1 b2 c2 8 a2 b1 c1 71 a2 b1 c2 19 a2 b2 c1 33 a2 b2 c2 52

[0148] Step 704: The coordination node distributes the data units related to the original data table to the corresponding storage nodes based on the first allocation information and the second allocation information.

[0149] The processing of this step is similar to that of step 505 , and reference may be made to the relevant description of step 505 .

[0150] Step 705: The storage node performs disk flushing, aggregation, and other processing on the received data units.

[0151] The processing of this step is similar to that of step 506 , and reference may be made to the relevant description of step 506 .

[0152] In the embodiment of the present disclosure, for the data unit of the received original data table, if the storage node that performs the aggregation processing and the storage node that performs the disk flushing processing corresponding to the data unit are not the same storage node, it can be considered to first perform the first-level aggregation processing on the data unit at the coordination node, and then perform the second-level aggregation processing at the storage node (because a coordination node cannot get all the data units that need to be aggregated, so it is necessary to complete the final aggregation at the storage node, that is, the second-level aggregation). Whether to perform two-level aggregation can be determined by a pre-determination process, that is, two working modes are set for the coordination node, mode one is a mode in which the coordination node does not participate in the aggregation, and mode two is a mode in which the coordination node participates in the aggregation. For these two modes, selection can be made based on some reference information, and the reference information may include relevant information of the original data table, and may also include relevant information of the coordination node. For example, the input-output ratio of the aggregation processing of the original data table, the input-output ratio is the ratio of the number of data before and after the aggregation (it can be estimated based on experience or can be determined by experiments on the original data table). When the input-output ratio is greater than the specified threshold, mode two is adopted, and when the input-output ratio is less than the specified threshold, mode one is adopted. The specified threshold can be set based on experience or experiments.

[0153] In mode 1, the processing of step 505 and step 704 may be as follows:

[0154] After receiving the data unit of the original data, the coordination node determines the first attribute value set that matches the data unit in the first allocation information, and then determines the first storage node corresponding to the first attribute value set. In the second allocation information, it determines the second attribute value set that matches the data unit, and then determines the second storage node corresponding to the second attribute value set. If the first storage node and the second storage node are the same, a write mark and an aggregation mark can be added to the data unit, and the marked data unit can be sent to the storage node; if the first storage node and the second storage node are different, the data unit is copied, an aggregation mark is added to one of the two data units, and the marked data unit is sent to the first storage node, a write mark is added to the other of the two data units, and the marked data unit is sent to the second storage node. Optionally, for data units that have an aggregation mark but no write mark, before sending the data unit, the attribute values ​​(i.e., numerical fields) that need to be aggregated are retained in the data unit and other attribute values ​​are deleted.

[0155] The following is an example (using the examples of step 504 and step 703 respectively):

[0156] The processing of step 505 corresponding to the example of step 504 is as follows:

[0157] The first attribute value set matched by the data unit in the first row of Table 1 is {a1}, and the second attribute value set matched is {a1, b1}. The storage nodes corresponding to {a1} and {a1, b1} are both storage node 1. A write mark and an aggregation mark are added to the data unit, and the marked data unit is distributed to storage node 1. The data unit in the second row is similar to the data unit in the first row. A write mark and an aggregation mark are also added and sent to storage node 1. The first attribute value set matched by the data unit in the third row is {a1}, and the second attribute value set matched is {a1, b2} , the storage node corresponding to {a1} is storage node 1, and the storage node corresponding to {a1, b2} is storage node 2. Copy the third row of data again, add an aggregation tag to one of them and distribute it to storage node 1, and add a write tag to the other and distribute it to storage node 2; the fourth row of data units are similar to the third row of data units. Copy the fourth row of data again, add an aggregation tag to one of them and distribute it to storage node 1, and add a write tag to the other and distribute it to storage node 2; the fifth row of data units to the eighth row of data units are the same as the above processing steps, and they are not listed one by one here.

[0158] Optionally, for a data unit to which an aggregation tag is added but a write tag is not added, before the data unit is sent, the attribute value (ie, the numerical field) corresponding to tagA is retained in the data unit and other attribute values ​​are deleted.

[0159] The processing of step 704 corresponding to the example of step 703 is as follows:

[0160] The first attribute value set matched by the first row of data units in Table 3 is {b1, c1}, and the second attribute value set matched is {a1, b1}. The storage node corresponding to {b1, c1} and {a1, b1} is storage node 1. Write marks and aggregation marks are added to the first row of data, and the marked data units are distributed to storage node 1. The first attribute value set matched by the second row of data units is {b1, c2}, and the second attribute value set matched is {a1, b1}. The storage node corresponding to {b1, c2} is storage node 2, and the storage node corresponding to {a1, b1} is storage node 1. Make another copy of the second row of data units, add an aggregation mark to one of them and distribute it to storage node 2, and add a write mark to the other copy and distribute it to storage node 1. The above processing steps are the same for the third row of data units to the eighth row of data units, and they are not listed one by one here.

[0161] Optionally, for a data unit to which an aggregation mark is added but a write mark is not added, before the data unit is sent, the attribute values ​​(ie, numerical fields) corresponding to tagB and tagC are retained in the data unit, and other attribute values ​​are deleted.

[0162] In mode 2, the processing of step 505 and step 704 may be as follows:

[0163] The process of adding aggregation marks and write marks to the data units of the original data table is the same as that of mode 1. When there are multiple data units of the original data table to be sent with aggregation marks added but without write marks added, after retaining the attribute values ​​that need to be aggregated and deleting other attribute values ​​for these data units, the coordination node can perform aggregation processing on these data units, add aggregation marks to the data units obtained by the aggregation processing, and then send them to the storage node. The storage node can perform secondary aggregation processing based on the received data units obtained by the aggregation processing.

[0164] The embodiment of the present disclosure also provides an instruction for viewing an aggregated task, and the instruction may be in the following form:

[0165] SHOW STERAMS

[0166] The characters in this command are fixed and are used to view the running status of all aggregate tasks created by the aggregate command.

[0167] The embodiment of the present disclosure also provides an instruction for deleting an aggregation task, and the instruction may be in the following form:

[0168] DROP STREAM stream_name

[0169] DROP STREAM is the command name. The characters in this field are fixed and can be called a fixed field. "stream_name" is a parameter field. The content in the parameter field can be set according to actual needs. "stream_name" is the task name of the streaming aggregation task.

[0170] This command is used to delete the streaming aggregation task named "stream_name". After deleting the streaming aggregation task, the original data table can continue to record time series data, and the aggregation data table stops recording data.

[0171] In the disclosed embodiment, data aggregation processing is performed through a distributed database system, so that data storage and data aggregation processing are both completed by the distributed database, and there is no need to build an additional processing system for data aggregation processing, and there is no need to copy the data and send it to the distributed database system and the additional processing system respectively. In this way, the occupation of transmission resources can be reduced.

[0172] Based on the same technical concept, the embodiment of the present disclosure provides a device for data aggregation processing, which can be applied to the above-mentioned coordination node, such as Fig. 9 As shown, the device comprises:

[0173] The receiving module 910 is used to receive the aggregation instruction for the original data table, and can specifically implement the processing functions of the above steps 201, 501 and 701, as well as other implicit steps.

[0174] The acquisition module 920 is used to obtain the first sharding key of the aggregated data table corresponding to the original data table, wherein the aggregated data table is used to record the data after the data in the original data table is aggregated, and the first sharding key includes at least one first attribute label of the aggregated data table. Specifically, the processing functions of the above steps 202, 502 and 702, as well as other implicit steps can be implemented.

[0175] The determination module 930 is used to determine the first allocation information, wherein the first allocation information is used to indicate the storage nodes for performing the aggregation processing respectively allocated to the multiple first attribute value sets corresponding to the first sharding key, and the first attribute value set is composed of one attribute value of each first attribute label. Specifically, the processing functions of the above steps 203, 504 and 703, as well as other implicit steps, can be realized.

[0176] The distribution module 940 is used to compare the first attribute value set in the first allocation information with the attribute value in the data unit corresponding to the original data table, so as to distribute the data unit corresponding to the original data table to the corresponding storage node. Specifically, the processing functions of the above steps 204, 505 and 704, as well as other implicit steps, can be realized.

[0177] In a possible implementation, the determination module 930 is used to determine the first allocation information based on the second allocation information of the original data table, wherein the second allocation information is used to indicate the storage nodes for performing the disk flushing process respectively allocated to the multiple second attribute value sets corresponding to the second sharding key, the storage nodes indicated by the second allocation information are used to perform the disk flushing process, the second attribute value set is composed of one attribute value for each second attribute label, and in the first allocation information and the second allocation information, the storage nodes corresponding to the same attribute value set are the same. Specifically, the processing function of the above step 203 and other implicit steps can be implemented.

[0178] In a possible implementation, the determination module 930 is used to determine a plurality of first attribute value sets corresponding to the first sharding key and a plurality of second attribute value sets corresponding to the second sharding key, wherein the second attribute value set is composed of an attribute value of each second attribute tag and based on the plurality of first attribute value sets and the plurality of second attribute value sets, determine the first allocation information. Specifically, the processing function of the above step 503 and other implicit steps can be implemented.

[0179] In a possible implementation, the determination module 930 is used to determine, for each first attribute value set, at least one second attribute value set including the first attribute value set, select a storage node from the storage nodes corresponding to the at least one second attribute value set as the storage node corresponding to the first attribute value set; and determine the first allocation information based on the storage node corresponding to each first attribute value set. Specifically, the processing function of the above step 504 and other implicit steps can be implemented.

[0180] In a possible implementation, the determination module 930 is used to determine the first allocation information based on a storage node selection rule, wherein the storage node selection rule includes random selection, selection in a specified order, or selection based on resource occupancy. Specifically, the processing function of the above step 504 and other implicit steps can be implemented.

[0181] In a possible implementation, the first shard key and the second shard key are different and are not a subset of the second shard key, and the second shard key includes at least one second attribute label of the original data table.

[0182] In one possible implementation, the first shard key is determined based on the aggregation instruction.

[0183] In a possible implementation, the device further includes a marking module 950, which is used to compare the first attribute value set in the first allocation information and the second attribute value set in the second allocation information with the attribute values ​​in the data unit corresponding to the original data table, respectively, so as to add an aggregation mark and / or a write mark to the data unit corresponding to the original data table, which can specifically implement the processing functions of the above steps 204, 505 and 704, as well as other implicit steps. A distribution module 940 is used to distribute the marked data unit to the corresponding storage node, wherein the second allocation information is used to indicate the storage nodes for performing disk flushing processing respectively assigned to the multiple second attribute value sets corresponding to the second sharding key, the storage node indicated by the second allocation information is used to perform disk flushing processing, the second sharding key includes at least one second attribute tag of the original data table, the second attribute value set consists of one attribute value of each second attribute tag, the aggregation mark is used to indicate that the storage node performs aggregation processing on the data unit, and the write mark is used to indicate that the storage node performs disk flushing processing on the data unit. Specifically, the processing functions of the above steps 204, 505 and 704, as well as other implicit steps, can be implemented.

[0184] In one possible implementation, the determination module 930 is used to determine, in the first allocation information, a first attribute value set that matches the attribute value of the first data unit of the received original data table and a corresponding first storage node, and to determine, in the second allocation information, a second attribute value set that matches the attribute value of the first data unit of the received original data table and a corresponding second storage node; specifically, it can implement the processing functions of the above steps 203, 504 and 703, as well as other implicit steps.

[0185] The marking module 950 is used to add an aggregation mark and a write mark to the first data unit when the first storage node and the second storage node are the same storage node. Specifically, it can realize the processing function of the above step 204 and other implicit steps.

[0186] The distribution module 940 is used to send the marked data unit to the first storage node; specifically, it can implement the processing function of the above step 204 and other implicit steps.

[0187] The marking module 950 is used to, when the first storage node and the second storage node are not the same storage node, copy the first data unit to obtain the second data unit, aggregate the first data unit with other data units matching the first attribute value set, add an aggregation mark to the data unit obtained by the aggregation process, and add a write mark to the second data unit. Specifically, the processing functions of the above steps 505 and 704, as well as other implicit steps, can be realized.

[0188] The distribution module 940 is used to send the first data unit to the first storage node and send the second data unit to the second storage node. Specifically, it can implement the processing functions of the above steps 505 and 704, as well as other implicit steps.

[0189] In one possible implementation, the determination module 930 is used to determine, in the first allocation information, a first attribute value set that matches the attribute value of the first data unit of the received original data table and a corresponding first storage node, and to determine, in the second allocation information, a second attribute value set that matches the first data unit of the received original data table and a corresponding second storage node; specifically, it can implement the processing function of the above-mentioned step 505, as well as other implicit steps.

[0190] The marking module 950 is used to add an aggregation mark and a write mark to the first data unit when the first storage node and the second storage node are the same storage node. Specifically, it can implement the processing function of the above step 504 and other implicit steps.

[0191] The distribution module 940 is used to send the marked data unit to the first storage node; specifically, it can implement the processing function of the above step 504 and other implicit steps.

[0192] The marking module 950 is used to copy the first data unit to obtain the second data unit, add an aggregation mark to the first data unit, and add a write mark to the second data unit when the first storage node and the second storage node are not the same storage node. Specifically, it can realize the processing functions of the above steps 505 and 704, as well as other implicit steps.

[0193] The distribution module 940 is used to send the first data unit to the first storage node and send the second data unit to the second storage node. Specifically, it can implement the processing functions of the above steps 505 and 704, as well as other implicit steps.

[0194] In the disclosed embodiment, data aggregation processing is performed through a distributed database system, so that data storage and data aggregation processing are both completed by the distributed database, and there is no need to build an additional processing system for data aggregation processing, and there is no need to copy the data and send it to the distributed database system and the additional processing system respectively. In this way, the occupation of transmission resources can be reduced.

[0195] Among them, the receiving module 910, the acquisition module 920, the determination module 930, the distribution module 940 and the marking module 950 can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the receiving module 910 is introduced below by taking the receiving module 910 as an example. Similarly, the implementation of the acquisition module 920, the determination module 930, the distribution module 940 and the marking module 950 can refer to the implementation of the receiving module 910.

[0196] As an example of a software functional unit, the receiving module 910 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the receiving module 910 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0197] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0198] As an example of a hardware functional unit, the receiving module 910 may include at least one computing device, such as a server, etc. Alternatively, the receiving module 910 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0199] The multiple computing devices included in the receiving module 910 can be distributed in the same region or in different regions. The multiple computing devices included in the receiving module 910 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the receiving module 910 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0200] It should be noted that, in other embodiments, the receiving module 910, the acquisition module 920, the determination module 930, the distribution module 940 and the marking module 950 can be used for any step in the data aggregation processing method in the computing node, and the steps that the receiving module 910, the acquisition module 920, the determination module 930, the distribution module 940 and the marking module 950 are responsible for implementing can be specified as needed. The full functions of the device for performing data aggregation processing on the computing node are realized by respectively implementing different steps in the data aggregation processing method through the receiving module 910, the acquisition module 920, the determination module 930, the distribution module 940 and the marking module 950.

[0201] Based on the same technical concept, the embodiment of the present disclosure provides a device for data aggregation processing, which can be applied to the above storage nodes, such as Fig.10 As shown, the device includes: an aggregation module 1010, which is used to perform aggregation processing based on the received data units. Specifically, it can realize the processing functions of the above steps 205, 506 and 705, as well as other implicit steps.

[0202] In a possible implementation, the device further includes an application module 1020, which is used to apply for a first temporary cache from the first cache pool and write the received data unit into the first temporary cache; specifically, it can implement the processing function of the above step 205 and other implicit steps.

[0203] The device also includes a writing module 1030, which is used to write the data units in the first temporary cache into the storage disk; specifically, it can realize the processing function of the above step 205, as well as other implicit steps.

[0204] The device also includes a release module 1040, which is used to apply to the second cache pool for a second temporary cache having the same space size as the first temporary cache, and release the second temporary cache to the first cache pool; specifically, it can implement the processing function of the above step 205 and other implicit steps.

[0205] Aggregation module 1010 is used to perform aggregation processing based on the data units in the first temporary cache; specifically, it can implement the processing function of the above step 205 and other implicit steps.

[0206] The release module 1040 is also used to release the first temporary buffer to the second buffer pool, and can specifically implement the processing function of the above step 205 and other implicit steps.

[0207] In the disclosed embodiment, data aggregation processing is performed through a distributed database system, so that data storage and data aggregation processing are both completed by the distributed database, and there is no need to build an additional processing system for data aggregation processing, and there is no need to copy the data and send it to the distributed database system and the additional processing system respectively. In this way, the occupation of transmission resources can be reduced.

[0208] Among them, the aggregation module 1010, the application module 1020, the writing module 1030 and the release module 1040 can all be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the aggregation module 1010 is introduced below by taking the aggregation module 1010 as an example. Similarly, the implementation of the application module 1020, the writing module 1030 and the release module 1040 can refer to the implementation of the aggregation module 1010.

[0209] As an example of a software functional unit, the aggregation module 1010 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the aggregation module 1010 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Among them, usually a region can include multiple AZs.

[0210] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0211] As an example of a hardware functional unit, the aggregation module 1010 may include at least one computing device, such as a server, etc. Alternatively, the aggregation module 1010 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0212] The multiple computing devices included in the aggregation module 1010 can be distributed in the same region or in different regions. The multiple computing devices included in the aggregation module 1010 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the aggregation module 1010 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0213] It should be noted that, in other embodiments, the aggregation module 1010, the application module 1020, the write module 1030 and the release module 1040 can be used for any step in the data aggregation processing method in the computing node, and the steps that the aggregation module 1010, the application module 1020, the write module 1030 and the release module 1040 are responsible for implementing can be specified as needed. The aggregation module 1010, the application module 1020, the write module 1030 and the release module 1040 respectively implement different steps in the data aggregation processing method to realize the full functions of the device for performing data aggregation processing on the computing node.

[0214] The present disclosure also provides a distributed database system, such as Figure 1 As shown, it includes a coordination node, multiple storage nodes and a storage disk. The functions of the coordination node, storage node and storage disk have been introduced and will not be repeated here.

[0215] The coordination node, storage node and storage disk can be implemented by software or hardware. As an example, the implementation of the coordination node is introduced below. Similarly, the implementation of the storage node and storage disk can refer to the implementation of the coordination node.

[0216] As an example of a software functional unit, a coordination node may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container, and other computing devices. Furthermore, the computing device may be one or more. For example, a device for performing data aggregation processing may include code running on multiple hosts / virtual machines / containers. It should be noted that multiple hosts / virtual machines / containers for running the application may be distributed in the same region or in different regions. Multiple hosts / virtual machines / containers for running the code may be distributed in the same AZ or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Typically, a region may include multiple AZs.

[0217] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same VPC or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway must be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0218] As an example of a hardware functional unit, the coordination node may include at least one computing device, such as a server, etc. Alternatively, the device for performing data aggregation processing may also be a device implemented by ASIC or PLD, etc. The PLD may be implemented by CPLD, FPGA, GAL or any combination thereof.

[0219] The multiple computing devices included in the coordination node can be distributed in the same region or in different regions. The multiple computing devices included in the coordination node can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the coordination node can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0220] The present disclosure also provides a computing device 100. The computing device 100 can be applied to the above-mentioned coordination node or storage node. Fig.11 As shown, the computing device 100 includes: a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, the memory 106, and the communication interface 108 communicate with each other through the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that the present disclosure does not limit the number of processors and memories in the computing device 100.

[0221] The bus 102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus 102 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the computing device 100 (eg, the memory 106, the processor 104, and the communication interface 108).

[0222] The processor 104 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0223] The memory 106 may include a volatile memory, such as a random access memory (RAM). The memory 106 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0224] The memory 106 stores executable program codes, and the processor 104 executes the executable program codes to respectively implement the functions of the aforementioned receiving module 910, obtaining module 920, determining module 930, distributing module 940, marking module 950, aggregating module 1010, applying module 1020, writing module 1030, and releasing module 1040, thereby implementing the method for data aggregation processing. That is, the memory 106 stores instructions for the method for data aggregation processing.

[0225] Alternatively, the memory 106 stores executable codes, and the processor 104 executes the executable codes to respectively implement the functions of the aforementioned receiving module 910, obtaining module 920, determining module 930, distributing module 940, marking module 950, aggregating module 1010, applying module 1020, writing module 1030, and releasing module 1040, thereby implementing the method for data aggregation processing. That is, the memory 106 stores instructions for the method for data aggregation processing.

[0226] The communication interface 108 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0227] The present disclosure also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0228] like Fig.12 As shown, the computing device cluster includes at least one computing device 100. The memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for the method for data aggregation processing.

[0229] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions of the method for data aggregation processing. In other words, the combination of one or more computing devices 100 may jointly execute the instructions of the method for data aggregation processing.

[0230] It should be noted that the memory 106 in different computing devices 100 in the computing device cluster can store different instructions, which are respectively used for part of the functions of the data aggregation processing device. That is, the instructions stored in the memory 106 in different computing devices 100 can implement the functions of one or more of the aforementioned receiving module 910, acquisition module 920, determination module 930, distribution module 940, marking module 950, aggregation module 1010, application module 1020, writing module 1030 and release module 1040.

[0231] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network, etc. The network may be a Transmission Control Protocol (TCP) network or a Remote Direct Memory Access (RDMA) network, etc. Fig.13 A possible implementation is shown. Fig.13 As shown, two computing devices 100A and 100B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 106 in the computing device 100A stores instructions for executing the functions of the receiving module 910, the acquiring module 920, the determining module 930, the distributing module 940, and the marking module 950. At the same time, the memory 106 in the computing device 100B stores instructions for executing the functions of the aggregation module 1010, the application module 1020, the writing module 1030, and the releasing module 1040.

[0232] Fig.13The connection method between the computing device clusters shown may be that considering that the method for data aggregation processing provided by the present disclosure requires a large amount of data storage, it is considered that the functions implemented by the aggregation module 1010, the application module 1020, the writing module 1030 and the release module 1040 are handed over to the computing device 100B for execution.

[0233] It should be understood that Fig.13 The functions of the computing device 100A shown in FIG. 1 may also be completed by multiple computing devices 100. Similarly, the functions of the computing device 100B may also be completed by multiple computing devices 100.

[0234] The present disclosure also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.12 and Fig.13 The connection mode of the computing device cluster is different in that the memory 106 in one or more computing devices 100 in the computing device cluster may store the same instructions for the method of data aggregation processing.

[0235] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions of the method for data aggregation processing. In other words, the combination of one or more computing devices 100 may jointly execute the instructions of the method for data aggregation processing.

[0236] The present disclosure also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device is caused to perform a method for data aggregation processing.

[0237] The embodiments of the present disclosure also provide a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disk (DVD)), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct a computing device to perform a data aggregation processing method, or instruct a computing device to perform a data aggregation processing method.

[0238] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for data aggregation processing, characterized in that: The method is applied to a distributed database system, the distributed database system includes a coordination node and a plurality of storage nodes, and the method includes: The coordination node receives an aggregation instruction for the original data table; The coordination node obtains a first sharding key of an aggregate data table corresponding to the original data table, wherein the aggregate data table is used to record data after the data in the original data table is aggregated, and the first sharding key includes at least one first attribute label of the aggregate data table; The coordination node determines first allocation information, wherein the first allocation information is used to indicate storage nodes respectively allocated to multiple first attribute value sets corresponding to the first sharding key, and the first attribute value set consists of one attribute value of each first attribute label; The coordination node compares the first attribute value set in the first allocation information with the attribute values ​​in the data units corresponding to the original data table, so as to distribute the data units corresponding to the original data table to the corresponding storage nodes; The storage node performs aggregation processing based on the received data units.

2. The method according to claim 1, characterized in that The first sharding key is the same as the second sharding key of the original data table, and the second sharding key includes at least one second attribute label of the original data table; The coordinating node determines first allocation information, including: The coordination node determines the first allocation information based on the second allocation information of the original data table, wherein the second allocation information is used to indicate the storage nodes respectively allocated to the multiple second attribute value sets corresponding to the second sharding key, the storage nodes indicated by the second allocation information are used to perform disk flushing, the second attribute value set consists of an attribute value for each second attribute label, and in the first allocation information and the second allocation information, the storage nodes corresponding to the same attribute value set are the same.

3. The method according to claim 1, characterized in that The first sharding key is different from the second sharding key and is a subset of the second sharding key, and the second sharding key includes at least one second attribute label of the original data table; The coordinating node determines first allocation information, including: The coordination node determines a plurality of first attribute value sets corresponding to the first sharding key and a plurality of second attribute value sets corresponding to the second sharding key, wherein the second attribute value set consists of one attribute value for each second attribute label; The coordination node determines first allocation information based on the multiple first attribute value sets and the multiple second attribute value sets.

4. The method according to claim 3, characterized in that The coordinating node determines first allocation information based on the multiple first attribute value sets and the multiple second attribute value sets, including: For each first attribute value set, the coordination node determines at least one second attribute value set including the first attribute value set, and selects one storage node from the storage nodes corresponding to the at least one second attribute value set as the storage node corresponding to the first attribute value set; The coordination node determines first allocation information based on the storage nodes corresponding to each first attribute value set.

5. The method according to claim 1, characterized in that: The coordinating node determines first allocation information, including: The coordination node determines the first allocation information based on a storage node selection rule, wherein the storage node selection rule includes random selection, selection in a specified order, or selection based on resource occupancy.

6. The method according to claim 5, characterized in that The first sharding key is different from the second sharding key and is not a subset of the second sharding key, and the second sharding key includes at least one second attribute label of the original data table.

7. The method according to any one of claims 1 to 6, characterized in that: The first shard key is determined based on the aggregation instruction.

8. The method according to any one of claims 1 to 7, characterized in that: The coordination node compares the first attribute value set in the first allocation information with the attribute value in the data unit corresponding to the original data table to distribute the data unit corresponding to the original data table to the corresponding storage node, including: The coordination node compares the first attribute value set in the first allocation information and the second attribute value set in the second allocation information with the attribute values ​​in the data units corresponding to the original data table, respectively, to add aggregation marks and / or write marks to the data units corresponding to the original data table, and distributes them to the corresponding storage nodes, wherein the second allocation information is used to indicate the storage nodes respectively allocated to the multiple second attribute value sets corresponding to the second sharding key, the storage nodes indicated by the second allocation information are used to perform disk flushing, the second sharding key includes at least one second attribute tag of the original data table, the second attribute value set consists of one attribute value for each second attribute tag, the aggregation mark is used to instruct the storage node to perform aggregation processing on the data unit, and the write mark is used to instruct the storage node to perform disk flushing processing on the data unit.

9. The method according to claim 8, characterized in that The coordination node compares the first attribute value set in the first allocation information and the second attribute value set in the second allocation information with the attribute values ​​in the data units corresponding to the original data table, respectively, to add an aggregation mark and / or a write mark to the data units corresponding to the original data table, and distributes them to the corresponding storage nodes, including: The coordinating node determines, in the first allocation information, a first attribute value set that matches the attribute value of the first data unit of the received original data table and a corresponding first storage node, and determines, in the second allocation information, a second attribute value set that matches the attribute value of the first data unit of the received original data table and a corresponding second storage node; When the first storage node and the second storage node are the same storage node, the coordination node adds an aggregation mark and a write mark to the first data unit, and sends it to the first storage node; When the first storage node and the second storage node are not the same storage node, the coordination node copies the first data unit to obtain the second data unit, aggregates the first data unit with other data units that match the first attribute value set, adds the aggregation mark to the data unit obtained by the aggregation and sends it to the first storage node, and adds the write mark to the second data unit and sends it to the second storage node.

10. The method according to claim 8, characterized in that The coordination node compares the first attribute value set in the first allocation information and the second attribute value set in the second allocation information with the attribute values ​​in the data units corresponding to the original data table, respectively, to add an aggregation mark and / or a write mark to the data units corresponding to the original data table, and distributes them to the corresponding storage nodes, including: The coordinating node determines, in the first allocation information, a first attribute value set that matches the attribute value of the first data unit of the received original data table and a corresponding first storage node, and determines, in the second allocation information, a second attribute value set that matches the attribute value of the first data unit of the received original data table and a corresponding second storage node; When the first storage node and the second storage node are the same storage node, the coordination node adds an aggregation mark and a write mark to the first data unit, and sends it to the first storage node; When the first storage node and the second storage node are not the same storage node, the coordination node copies the first data unit to obtain the second data unit, adds the aggregation mark to the first data unit and sends it to the first storage node, and adds the write mark to the second data unit and sends it to the second storage node.

11. The method according to any one of claims 1 to 10, characterized in that: The storage node performs aggregation processing based on the received data units, including: The storage node applies for a first temporary cache from the first cache pool, and writes the received data unit into the first temporary cache; The storage node writes the data unit in the first temporary cache into the storage disk; The storage node applies to the second cache pool for a second temporary cache having the same space size as the first temporary cache, and releases the second temporary cache to the first cache pool; The storage node performs aggregation processing based on the data units in the first temporary cache; The storage node releases the first temporary buffer to the second buffer pool.

12. A distributed database system, characterized in that: The distributed database system includes a coordination node and multiple storage nodes; The coordination node is used to: Receive an aggregation instruction for an original data table; obtain a first sharding key of an aggregated data table corresponding to the original data table, wherein the aggregated data table is used to record data after the data in the original data table is aggregated, and the first sharding key includes at least one first attribute tag of the aggregated data table; determine first allocation information, wherein the first allocation information is used to indicate storage nodes respectively allocated to multiple first attribute value sets corresponding to the first sharding key, and the first attribute value set consists of one attribute value for each first attribute tag; compare the first attribute value set in the first allocation information with the attribute value in the data unit corresponding to the original data table, so as to distribute the data unit corresponding to the original data table to the corresponding storage node; The storage node is used to perform aggregation processing based on the received data units.

13. The distributed database system according to claim 12, characterized in that: The first sharding key is the same as the second sharding key of the original data table, and the second sharding key includes at least one second attribute label of the original data table; The coordination node is used to: Based on the second allocation information of the original data table, the first allocation information is determined, wherein the second allocation information is used to indicate storage nodes respectively allocated to multiple second attribute value sets corresponding to the second sharding key, the storage nodes indicated by the second allocation information are used to perform disk flushing, the second attribute value set is composed of an attribute value for each second attribute label, and in the first allocation information and the second allocation information, the storage nodes corresponding to the same attribute value set are the same.

14. The distributed database system according to claim 12, characterized in that: The first sharding key is different from the second sharding key and is a subset of the second sharding key, and the second sharding key includes at least one second attribute label of the original data table; The coordination node is used to: Determine a plurality of first attribute value sets corresponding to the first sharding key and a plurality of second attribute value sets corresponding to the second sharding key, wherein the second attribute value set consists of one attribute value for each second attribute label; First allocation information is determined based on the multiple first attribute value sets and the multiple second attribute value sets.

15. The distributed database system according to claim 14, characterized in that: The coordination node is used to: For each first attribute value set, determine at least one second attribute value set that includes the first attribute value set, and select a storage node from the storage nodes corresponding to the at least one second attribute value set as the storage node corresponding to the first attribute value set; First allocation information is determined based on the storage nodes corresponding to each first attribute value set.

16. The distributed database system according to claim 12, characterized in that: The coordination node is used to: The first allocation information is determined based on a storage node selection rule, wherein the storage node selection rule includes random selection, selection in a specified order, or selection based on resource occupancy.

17. The distributed database system according to claim 16, characterized in that: The first sharding key is different from the second sharding key and is not a subset of the second sharding key, and the second sharding key includes at least one second attribute label of the original data table.

18. The distributed database system according to any one of claims 12 to 17, characterized in that: The first shard key is determined based on the aggregation instruction.

19. The distributed database system according to any one of claims 12 to 18, characterized in that: The coordination node is used to: The first attribute value set in the first allocation information and the second attribute value set in the second allocation information are respectively compared with the attribute values ​​in the data units corresponding to the original data table, so as to add aggregation marks and / or write marks to the data units corresponding to the original data table, and distribute them to the corresponding storage nodes, wherein the second allocation information is used to indicate the storage nodes respectively allocated to the multiple second attribute value sets corresponding to the second sharding key, the storage nodes indicated by the second allocation information are used to perform disk flushing, the second sharding key includes at least one second attribute tag of the original data table, the second attribute value set consists of one attribute value for each second attribute tag, the aggregation mark is used to instruct the storage node to perform aggregation processing on the data unit, and the write mark is used to instruct the storage node to perform disk flushing processing on the data unit.

20. The distributed database system according to claim 19, characterized in that: The coordination node is used to: In the first allocation information, a first attribute value set matching the attribute value of the first data unit received in the original data table and a corresponding first storage node are determined, and in the second allocation information, a second attribute value set matching the attribute value of the first data unit received in the original data table and a corresponding second storage node are determined; When the first storage node and the second storage node are the same storage node, adding an aggregation mark and a write mark to the first data unit, and sending the first data unit to the first storage node; When the first storage node and the second storage node are not the same storage node, the first data unit is copied to obtain the second data unit, the first data unit is aggregated with other data units matching the first attribute value set, the aggregation mark is added to the data unit obtained by the aggregation and sent to the first storage node, and the write mark is added to the second data unit and sent to the second storage node.

21. The distributed database system according to claim 19, characterized in that: The coordination node is used to: In the first allocation information, a first attribute value set matching the attribute value of the first data unit received in the original data table and a corresponding first storage node are determined, and in the second allocation information, a second attribute value set matching the attribute value of the first data unit received in the original data table and a corresponding second storage node are determined; When the first storage node and the second storage node are the same storage node, adding an aggregation mark and a write mark to the first data unit, and sending the first data unit to the first storage node; When the first storage node and the second storage node are not the same storage node, the first data unit is copied to obtain the second data unit, the aggregation mark is added to the first data unit and sent to the first storage node, and the write mark is added to the second data unit and sent to the second storage node.

22. The distributed database system according to any one of claims 12 to 21, characterized in that: The storage node is used for: Applying for a first temporary cache from a first cache pool, and writing the received data unit into the first temporary cache; Writing the data unit in the first temporary cache into the storage disk; Applying for a second temporary cache having the same space size as the first temporary cache from the second cache pool, and releasing the second temporary cache to the first cache pool; performing aggregation processing based on the data units in the first temporary cache; The first temporary buffer is released to the second buffer pool.

23. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 11.

25. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 11.

Images