Data processing method and related equipment

Abstract

The embodiment of the invention provides a data processing method which is used for reducing occupation of storage resources and saving the storage resources. The method is applied to network equipment and comprises the steps that a base table and M aggregation tables are obtained, the base table comprises data actually stored by the network equipment, the M aggregation tables are obtained by processing the data included in the base table according to M aggregation definitions, the aggregation definitions corresponding to the M aggregation tables are different, each aggregation definition comprises a key-value pair, and the key-value pair is a key-value pair; the method is used for indicating processing modes of data included in the base table. And determining N aggregation tables to be merged from the M aggregation tables according to keys included in the M aggregation definitions. The to-be-merged N aggregation tables are merged, a merged aggregation table is obtained, and the data size of the merged aggregation table is smaller than the sum of the data sizes of the to-be-merged N aggregation tables. Wherein M and N are integers larger than or equal to 2, and N is smaller than or equal to M.

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office on September 28, 2023, with application number 202311287944.7 and invention name “A data aggregation method, device and computing device cluster”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of computers, and in particular to a data processing method and related equipment. Background Art

[0003] In data management, the data in the base table can be aggregated according to certain rules to obtain an aggregate table. The rules here can be understood as aggregation definitions, which include key-value pairs. The aggregate table can also be understood as preprocessing of the base table for further calls.

[0004] In a data processing method, different aggregation definitions are used to aggregate data in a base table to obtain different aggregation tables. In this method, different aggregation tables may have duplicate data, that is, multiple copies of the same data are stored, wasting storage resources. Summary of the invention

[0005] The present application provides a data processing method and related equipment for reducing the occupancy of storage resources and saving storage resources.

[0006] In a first aspect, the present application provides a data processing method, comprising:

[0007] The network device obtains a base table and M aggregation tables. The base table includes data actually stored in the network device. The M aggregation tables are obtained by processing the data included in the base table according to the M aggregation definitions. The M aggregation tables correspond to the M aggregation definitions one by one, and the M aggregation definitions are different, which means that the data included in each of the M aggregation tables is different. The aggregation definition includes a key-value pair. Therefore, different aggregation definitions mean that at least one of the keys and values ​​included in the aggregation definition is different. When the aggregation definition includes multiple keys or multiple values, any one of them is different, which is also considered to be a different aggregation definition. In other words, the same aggregation definition means that each of the keys and values ​​included in the aggregation definition is the same. The aggregation definition is used to indicate the way to process the data of the base table including the aggregation table to obtain the aggregation table. According to the keys included in the M aggregation definitions, the N aggregation tables to be merged are determined from the M aggregation tables, and the N aggregation tables to be merged are merged to obtain the merged aggregation table. The amount of data in the merged aggregation table is less than the sum of the data amounts of the N aggregation tables to be merged. Among them, M and N are both integers greater than or equal to 2, and N is less than or equal to M. That is to say, the aggregate table to be merged may be part or all of the aggregate tables in the M aggregate tables. In addition, the N aggregate tables to be merged may be merged into one or more aggregate tables, that is, the number of the merged aggregate tables is one or more, which is determined by the specific merging operation, which will be described in detail later. It should also be noted that the M aggregate tables here refer to all the aggregate tables corresponding to the base table at the current moment. These M aggregate tables may include the original aggregate table of the base table, that is, the aggregate table obtained by processing the data in the base table for the first time; it may also include the aggregate table obtained by aggregating the data in the base table multiple times; it may also include the aggregate tables shown in the above two schemes, which are not limited here.

[0008] In the present application, after obtaining M aggregation tables corresponding to the same base table, according to the aggregation definition of the aggregation table, N aggregation tables to be merged are determined from the M aggregation tables, and the N aggregation tables are merged to obtain a merged aggregation table. The amount of data in the merged aggregation table is less than the sum of the amount of data in the N aggregation tables to be merged, that is, the data processing method provided by the present application reduces the number of times the same data in different aggregation tables is stored, thereby reducing the amount of data in the aggregation table and reducing the occupancy of storage resources.

[0009] In some optional implementations of the first aspect, the network device may merge different aggregation tables including the same key in the aggregation definition. Specifically, in the case where there are multiple first aggregation definitions including the same key in the M aggregation definitions, it is determined that the N aggregation tables to be merged include multiple first aggregation tables corresponding to the multiple first aggregation definitions. The same key mentioned here means that each key included in the aggregation definition is the same. For example, the key included in aggregation definition 1 is A+B, the key included in aggregation definition 2 is A+B, and the key included in aggregation definition 3 is A+C, then aggregation definition 1 and aggregation definition 2 are the multiple first aggregation tables mentioned here. In addition, the first aggregation table corresponding to the first aggregation definition mentioned here refers to the aggregation definition adopted by the first aggregation table as the first aggregation definition. In the case of determining that the N aggregation tables to be merged include the aforementioned multiple first aggregation tables, merging the N aggregation tables to be merged is actually merging the keys and values ​​corresponding to the multiple first aggregation tables. Since the keys corresponding to the multiple first aggregation tables are the same, merging the keys corresponding to the multiple first aggregation tables is to deduplicate the keys and only save one key. In addition, if there are duplicate data in the value, deduplication can also be performed. After merging multiple first aggregate tables, a merged aggregate table is obtained. In practical applications, there may be multiple groups of first aggregate tables. In this case, the merge is performed in groups. For example, the keys corresponding to aggregate tables 1 to 3 are the same, key1; the keys corresponding to aggregate tables 4 and 5 are the same, key2. Then the merge is to merge aggregate tables 1 to 3 to obtain a merged aggregate table. Merge aggregate tables 4 and 5 to obtain a merged aggregate table.

[0010] In the present application, for multiple first aggregation tables corresponding to the same key in the M aggregation tables, multiple first aggregation tables can be directly determined as the aggregation tables to be merged, which is simple and convenient to operate, and improves the practicality of the technical solution of the present application. At the same time, when merging the aggregation tables corresponding to the same key, the same key will be deduplicated, reducing the number of storage copies of the same key, further reducing the waste of storage resources, and saving the user's usage cost.

[0011] In some optional implementations of the first aspect, the network device may merge different aggregation tables with different keys included in the aggregation definition. Specifically, in the case where M aggregation definitions include X second aggregation definitions with different keys, the network device can determine whether there is an aggregation table that can be merged in the X aggregation tables by estimating the aggregation rate of the merged aggregation table. That is, determine whether there are N aggregation tables to be merged in the X aggregation tables. Wherein, X is an integer greater than or equal to 2 and less than or equal to M. The network device estimates at least one aggregation rate generated by merging any 2 to X second aggregation tables in the X second aggregation tables. The at least one aggregation rate generated here refers to the respective aggregation rates of at least one new aggregation table obtained by merging multiple second aggregation tables. If there is a first target aggregation rate greater than the first threshold value in the at least one aggregation rate, the network device determines that the N aggregation tables to be merged include at least one group of second aggregation tables corresponding to at least one first target aggregation rate. Wherein, each group of second aggregation tables in at least one group of second aggregation tables includes multiple second aggregation tables. Exemplarily, it is assumed that the aggregation rate of aggregation table A obtained by the network device from aggregation table 1 and aggregation table 2 with different merge keys is greater than the first threshold, and the aggregation rate of aggregation table B obtained by the network device from aggregation table 1 and aggregation table 3 with different merge keys is greater than the first threshold, that is, the network device determines that the first target aggregation rate includes the aggregation rate of aggregation table A and the aggregation rate of aggregation table B. Then, the network device can determine a group of second aggregation tables (i.e., aggregation table 1 and aggregation table 2) corresponding to the aggregation rate of the merged aggregation table A, and a group of second aggregation tables (i.e., aggregation table 1 and aggregation table 3) corresponding to the aggregation rate of the merged aggregation table B. Among them, the value of the first threshold can be set according to the needs of the actual application, and is not specifically limited here. Optionally, the fewer the storage resources (or idle storage resources) of the network device, the higher the value of the first threshold can be set; optionally, the better the computing performance of the network device, the lower the value of the first threshold can be set.

[0012] In this application, by comparing the estimated aggregation rate generated by merging different aggregation tables corresponding to different keys with the first threshold, it is determined whether the aggregation tables corresponding to different keys can be merged, thereby balancing computing resources and usage costs. In the scheme that can be merged, if there are identical values ​​in the aggregation tables of different keys, deduplication is performed during the merge, which further reduces the occupancy of storage resources and saves storage resources.

[0013] In some optional implementations of the first aspect, the multiple second aggregate tables included in each group of second aggregate tables are also different from the other aggregate tables in the X second aggregate tables. The other aggregate tables mentioned here refer to aggregate tables outside this group of second aggregate tables in the X second aggregate tables. In other words, if there is a second aggregate table whose estimated aggregation rates with different aggregate tables are greater than the first threshold, then a method is selected to merge the second aggregate tables. For example, assuming that the aggregation rate generated by merging aggregate table 1 and aggregate table 2, and the aggregation rate generated by merging aggregate table 1 and aggregate table 3, are both greater than the first threshold, then a group of second aggregate tables is determined to be "aggregate table 1 and aggregate table 2", or "aggregate table 1 and aggregate table 3". Correspondingly, a method is selected to merge from merging aggregate table 1 and aggregate table 2, and merging aggregate table 1 and aggregate table 3.

[0014] In the present application, it is defined that the multiple second aggregation tables included in each group of second aggregation tables are different from other aggregation tables in the X second aggregation tables. Therefore, when merging, for the second aggregation tables whose estimated aggregation rates with different aggregation tables are greater than the first threshold, only one method can be selected, and it is not necessary to merge all of them, which simplifies the processing flow and improves processing efficiency.

[0015] In some optional implementations of the first aspect, in the case where there are multiple aggregation rates in at least one aggregation rate that are greater than the first threshold and correspond to the same second aggregation table, the network device has multiple ways to determine the first target aggregation rate.

[0016] Optionally, the network device determines that the highest of the multiple aggregation rates is the first target aggregation rate. That is to say, if the estimated aggregation rates of a second aggregation table with different aggregation tables are all greater than the first threshold, and when a method is selected to merge the second aggregation tables, the highest of the estimated aggregation rates with different aggregation tables is selected as the first target aggregation rate, then the second aggregation table to be merged is determined to be the second aggregation table corresponding to the first target aggregation rate. For example, the estimated aggregation rate A of aggregation table 1 and aggregation table 2 is the highest among all the corresponding aggregation rates of aggregation table 1, then it can be determined that the first target aggregation rate is aggregation rate A, and the aggregation tables that need to be merged are aggregation table 1 and aggregation table 2.

[0017] Optionally, the network device determines that the first target aggregation rate is the one with the largest number of second aggregation tables among multiple aggregation rates. That is to say, if the estimated aggregation rates of a second aggregation table with different aggregation tables are all greater than the first threshold, and when a method is selected to merge the second aggregation tables, the one with the largest number of corresponding aggregation tables is selected as the first target aggregation rate, then the second aggregation table to be merged is determined to be the second aggregation table corresponding to the first target aggregation rate. For example, if the estimated aggregation rates of aggregation table 1 and aggregation table 2, aggregation table 1 and aggregation table 3, and aggregation tables 1 to 3 are all greater than the first threshold, then it can be determined that the first target aggregation rate is the aggregation rate corresponding to aggregation tables 1 to 3, and the aggregation tables that need to be merged are aggregation table 1, aggregation table 2, and aggregation table 3.

[0018] In the present application, when there are multiple aggregation rates in the estimated aggregation table that are greater than the first threshold and correspond to the same second aggregation table, there are multiple ways to determine the first target aggregation rate corresponding to the N aggregation tables to be merged, which enriches the implementation method and application scenario of the technical solution of the present application and improves the flexibility of the technical solution. In the scheme of selecting the highest of multiple aggregation rates as the first target aggregation rate, according to aggregation rate = (number of base table rows - number of aggregation table rows) / number of base table rows, it can be seen that the higher the aggregation rate, the less data in the aggregate table obtained after the merger, which further reduces the amount of storage and saves storage resources. In the scheme of selecting the largest number of second aggregation tables corresponding to multiple aggregation rates as the first target aggregation rate, the largest number of second aggregation tables are merged, so that the merged aggregation table can reduce the amount of stored data in addition to including data of more dimensions as much as possible, saving storage resources.

[0019] In some optional implementations of the first aspect, the N aggregate tables to be merged include at least one group of second aggregate tables corresponding to at least one first target aggregation rate. The N aggregate tables to be merged are merged, that is, the keys and values ​​corresponding to each group of second aggregate tables in at least one group of second aggregate tables are merged respectively. It can be understood that merging a group of second aggregate tables can obtain a merged aggregate table. Exemplarily, it is assumed that the N aggregate tables to be merged include 2 groups of second aggregate tables, the first group of second aggregate tables includes aggregate table 1 and aggregate table 2, and the second group of second aggregate tables includes aggregate table 3 to aggregate table 5. Then, when the merge is performed, the corresponding keys and values ​​of aggregate table 1 and aggregate table 2 are merged to obtain a merged aggregate table; and aggregate table 3 to aggregate table 5 are merged to obtain a merged aggregate table. When merging, deduplication is performed on the same values.

[0020] In this application, not only can the aggregation tables corresponding to the same key be merged, but also the aggregation tables corresponding to different keys can be merged to reduce the amount of stored data as much as possible. In addition, when merging the aggregation tables corresponding to different keys, if the same value exists in different aggregations, deduplication operation is performed to reduce the number of stored copies of the same value, further reducing the waste of storage resources and saving the user's usage cost.

[0021] In some optional implementations of the first aspect, after merging the N aggregation tables to be merged, that is, after obtaining the merged aggregation table, the network device can also determine whether data broadcasting is required. Specifically, the network device obtains the aggregation rate of all aggregation tables corresponding to the base table. All aggregation tables corresponding to the base table mentioned here refer to all aggregation tables corresponding to the base table after the N aggregation tables are merged. In other words, it includes the aggregation tables that have not been merged in the M aggregation tables, and the aggregation table obtained after the N aggregation tables are merged. The network device determines a second target aggregation rate that is greater than the second threshold from the aggregation rates of all aggregation tables, and broadcasts the data included in the aggregation table corresponding to the second target aggregation rate. Among them, the second threshold can be set according to the needs of the actual application, and is not limited here.

[0022] In this application, the data in the aggregation table with an aggregation rate greater than the second threshold can be broadcast, so that when querying data, users can query at any node without the need for data transmission between nodes, which reduces the use of network resources and improves the reliability and convenience of the technical solution. In addition, if the query is directly performed on a node with less load, the efficiency of data query can also be improved.

[0023] In some optional implementations of the first aspect, the network device executes the aggregation calculations of all aggregation tables corresponding to the base table in parallel. Specifically, the execution plan in the aggregation calculation layer may be modified to multi-threaded parallel execution, so that when the base table corresponds to multiple aggregation tables, the aggregation calculations of the multiple aggregation tables are executed in parallel.

[0024] In this application, aggregation calculations are performed in parallel, which reduces the impact on the data import process and thus reduces the data processing delay.

[0025] In a second aspect, the present application provides a data processing device, including:

[0026] An acquisition unit is used to acquire a base table and M aggregation tables. The base table includes data actually stored in a data processing device. The M aggregation tables are obtained by processing the data included in the base table according to M aggregation definitions. The M aggregation tables correspond one-to-one to the M aggregation definitions, and the M aggregation definitions are different. The aggregation definitions include key-value pairs. The aggregation definitions are used to indicate the method of processing the data included in the base table. M is an integer greater than or equal to 2.

[0027] The processing unit is used to determine N aggregation tables to be merged from the M aggregation tables according to the keys included in the M aggregation definitions, where N is an integer greater than or equal to 2 and less than or equal to M. The N aggregation tables to be merged are merged to obtain a merged aggregation table, and the data volume of the merged aggregation table is less than the sum of the data volumes of the N aggregation tables to be merged.

[0028] The data processing device is used to implement the method shown in the aforementioned first aspect or any possible implementation manner of the first aspect.

[0029] In a third aspect, the present application provides a computing device, the computing device comprising a processor and a memory. The processor of the computing device is used to execute instructions stored in the memory, so that the computing device implements the method shown in the first aspect or any possible implementation of the first aspect.

[0030] In a fourth aspect, the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster implements the method disclosed in the first aspect and any possible implementation of the first aspect.

[0031] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a processor, implement the method shown in the aforementioned first aspect or any possible implementation of the first aspect; or, when the instruction is executed by a computer device cluster, the computer device cluster implements the method disclosed in the first aspect and any possible implementation of the first aspect.

[0032] In a sixth aspect, the present application provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are executed on a processor, the method shown in the aforementioned first aspect or any possible implementation of the first aspect is implemented; or, when the computer program instructions are executed by a computer device cluster, the computer device cluster implements the method disclosed in the first aspect and any possible implementation of the first aspect.

[0033] The beneficial effects shown in any one of the second to sixth aspects are similar to those of the first aspect or any possible implementation method of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0035] Figure 2 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0036] Figure 3 A schematic diagram of a flow chart of a data processing method provided in an embodiment of the present application;

[0037] Figure 4 A schematic diagram provided for an embodiment of the present application;

[0038] Figure 5 Another schematic diagram provided for an embodiment of the present application;

[0039] Figure 6 Another schematic diagram provided for an embodiment of the present application;

[0040] Figure 7 Another flowchart of the data processing method provided in the embodiment of the present application;

[0041] Figure 8 A schematic diagram of the structure of a communication device provided in an embodiment of the present application;

[0042] Fig. 9 Another schematic diagram of the structure of a communication device provided in an embodiment of the present application;

[0043] Fig.10 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0044] Fig.11 Another structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The embodiment of the present application provides a data processing method for reducing the occupancy of storage resources and saving storage resources.

[0046] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0047] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged in appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attribute in the embodiments of the present application when describing. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment containing a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment. In addition, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or", describes the association relationship of associated objects, indicating that three relationships can exist, for example, A and / or B, can represent: A exists alone, A and B exist simultaneously, and B exists alone, wherein A, B can be singular or plural. The character " / " generally represents that the associated objects before and after are a kind of "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0048] First, the proper nouns and related concepts that may be involved in the embodiments of the present application are explained.

[0049] 1. Cloud Data Warehouse:

[0050] Cloud data warehouse, also known as cloud data warehouse, is a data storage and management service based on cloud computing technology, which stores and processes the big data of enterprises on the cloud platform. Compared with traditional data warehouses, cloud data warehouses have lower costs, higher scalability and faster response speed. It stores data in the cloud, and through the elasticity and flexibility of cloud computing, it can quickly adjust storage capacity and computing resources as needed. The data processing method provided in the embodiment of the present application can be applied in the cloud data warehouse scenario.

[0051] 2. Base table, aggregate table, and aggregation rate:

[0052] The base table is a table permanently stored in the database and is a logical representation of the actual stored data. For network devices that use the database, the data included in the base table is the data actually stored in the network device.

[0053] Aggregate tables are obtained by processing the data in the base table according to certain rules. The idea of ​​aggregate tables is that when importing data, the data is grouped according to the defined aggregate key by creating an aggregate definition in advance, and then the aggregate value is calculated according to the defined aggregate key to obtain the aggregate key and aggregate value. The aggregate key and aggregate value are written to the same table, that is, the aggregate table. In this way, the data set is compressed. Subsequent query tasks can scan and query from the aggregate table to improve query performance.

[0054] Aggregation rate = (number of base table rows - number of aggregate table rows) / number of base table rows, which means that the higher the aggregation rate, the smaller the amount of data in the aggregated data set and the better the performance.

[0055] 3. Cloud management platform and infrastructure:

[0056] The cloud management platform is used to manage the cloud vendor's infrastructure, which is multiple cloud data centers located in different regions, where each region is equipped with at least one cloud data center. The cloud management platform can provide interfaces related to cloud computing services, such as configuration pages (i.e., interfaces) or APIs for tenants to access cloud services. Tenants can log in to the cloud management platform using a pre-registered account and password, and after a successful login, select and purchase cloud services provided by the cloud data center in the predetermined region, such as object storage services, virtual machine services, container services, or other known cloud services.

[0057] 4. Tenants:

[0058] The top-level object used to manage cloud services and / or cloud resources. Tenants register their tenant accounts and set their tenant passwords on the cloud management platform through a local client (such as a browser). The local client remotely logs in to the cloud management platform through the tenant account and the set tenant password. The cloud management platform provides a configuration interface or API for tenants to configure and use cloud services, wherein the cloud services are specifically provided by the infrastructure managed by the cloud management platform as described above.

[0059] See below. Figure 1 and Figure 2 , Figure 1 and Figure 2 All of them are schematic diagrams of system architecture provided in embodiments of the present application.

[0060] like Figure 1 As shown, the tenant logs into the cloud service platform 30 through the client 10 via the Internet 20, using the account and password registered on the cloud service platform 30. The cloud service platform 30 manages the infrastructure, which includes multiple data centers set up in different regions, such as Figure 1The region 1 shown includes cloud data center 1 and cloud data center 2, and region 2 includes cloud data center 3 and cloud data center 4. Each cloud data center is provided with multiple servers, and service instances (including at least one of virtual machines, containers, and dedicated hosts) are run on the servers.

[0061] In the embodiment of the present application, a data management service is deployed in the business instance, and the tenant purchases the cloud service through the client on the cloud service platform 30, and the tenant sends a call request to the cloud service platform 30, and the call request is used to request the cloud service from the cloud service platform 30. The specific content of the cloud service includes providing data management services for the tenant, including data storage, data transmission, and data processing described later.

[0062] like Figure 2 As shown, the system includes a coordinator node (CN) and Z computing nodes (DN). Z is an integer greater than 2. Among them, CN is used to send execution plans and data to DN; DN is used to perform calculations, transmit data, etc. The data processing method provided in the embodiment of the present application can be used for both CN and DN. The application scenarios include interaction between DN and CN, or interaction between DNs under the same CN, which is not limited here.

[0063] The embodiments of the present application do not limit the product form of CN and DN. They can be virtual machines or physical machines on the cloud, etc., and can also be applied in non-cloud scenarios. No specific limitation is made here.

[0064] It should be noted that Figure 2 The embodiment shown is based on the example of a system including Z+1 nodes, which does not constitute a limitation on the number of nodes in the system. In practical applications, the system may also include a greater or lesser number of network devices, which is not specifically limited here.

[0065] See below. Figure 3 , Figure 3 A flow chart of a data processing method provided in an embodiment of the present application includes:

[0066] 301. The network device obtains a base table and M aggregation tables. The base table includes the data actually stored in the network device. The M aggregation tables are obtained by processing the data included in the base table according to M aggregation definitions. The M aggregation tables correspond to the M aggregation definitions one by one, and the M aggregation definitions are different. The aggregation definitions include key-value pairs. The aggregation definitions are used to indicate the method of processing the data included in the base table. M is an integer greater than or equal to 2.

[0067] During the data import process, the network device obtains the base table and M aggregation tables. These aggregation tables are obtained by processing the data included in the base table according to the aggregation definition. The data included in the base table mentioned here can be part or all of the data in the base table. In other words, the data source of the aggregation table is part or all of the data in the base table.

[0068] The aggregation definition includes a key-value pair, which is used to define the key and value of the aggregation table, which indicates how the network device processes the data included in the base table to obtain the aggregation table. Each of the M aggregation tables has a corresponding aggregation definition, that is, each aggregation table is obtained by processing the data included in the base table according to an aggregation definition.

[0069] The M aggregation tables each correspond to a different aggregation definition. Here, the different aggregation definitions refer to that at least one of the keys and values ​​included in the aggregation definition is different. When the aggregation definition includes multiple keys or multiple values, any one of them is different, which is also considered to be a different aggregation definition. In other words, the same aggregation definition means that each of the keys and values ​​included in the aggregation definition is the same.

[0070] For example, it is assumed that the base table is as shown in Table 1 below:

[0071] Table 1

[0072] suject Fraction Name age …… math 100 Li Si 12 …… Language 95 Zhang San 12 …… English 94 Wang Wu 11 …… Language 93 Li Si 12 …… math 91 Wang Wu 11 …… English 90 Li Si 12 …… English 88 Zhang San 12 …… math 88 Zhang San 12 …… Language 85 Wang Wu 11 ……

[0073] If the aggregation definition of the aggregation table 1 includes a key-value pair of "name-sum(score)", the aggregation table 1 is shown in the following Table 2. In this case, sum(score) indicates the sum of the scores.

[0074] Table 2

[0075] Name Sum(score) Li Si 283 Zhang San 271 Wang Wu 270

[0076] If the aggregation definition of aggregation table 2 includes the key-value pair "name-age", then aggregation table 2 is as shown in Table 3 below:

[0077] Table 3

[0078] Name age Li Si 12 Zhang San 12 Wang Wu 11

[0079] As shown in Table 2 and Table 3, the aggregate table groups the data in the base table according to the key included in the aggregate definition to achieve data compression. In subsequent data query tasks, you can scan and query from the aggregate table to improve query performance.

[0080] It should be noted that the aforementioned Tables 1 to 3 are only examples of base tables and aggregate tables, and do not constitute a limitation on the representation form of the base tables or aggregate tables. In actual applications, the base tables or aggregate tables may also be in other representation forms, such as tree structures, key-value pairs, etc., which are not specifically limited here.

[0081] 302. The network device determines N aggregation tables to be merged from the M aggregation tables according to the keys included in the M aggregation definitions, where N is an integer greater than or equal to 2 and less than or equal to M.

[0082] Different aggregation definitions may include the same key or different keys. Therefore, different aggregation tables may correspond to the same key or different keys. For different situations, network devices have different processing methods, which are explained below.

[0083] 1. N aggregate tables to be merged, including aggregate tables corresponding to the same key.

[0084] The aggregation definition of each aggregation table includes the key of the aggregation table. When there are multiple first aggregation definitions including the same key in the M aggregation definitions, the network device determines that the N aggregation tables to be merged include multiple first aggregation tables corresponding to the multiple first aggregation definitions. In addition, the first aggregation table corresponding to the first aggregation definition mentioned here refers to the aggregation definition adopted by the first aggregation table as the first aggregation definition.

[0085] Since an aggregation definition can include one or more keys, the multiple first aggregation definitions mentioned here include the same key, which means that each key included in different aggregation table definitions is the same. For example, the keys included in aggregation definition 1 and aggregation definition 2 are A+B, and the key corresponding to aggregation definition 3 is A+C, then aggregation definition 1 and aggregation definition 2 are the multiple first aggregation definitions mentioned here. Aggregation table 1 corresponding to aggregation definition 1 and aggregation table table 2 corresponding to aggregation definition 2 are the multiple first aggregation tables mentioned here.

[0086] In some optional implementations, there may be multiple groups of first aggregation definitions, that is, the N aggregation tables to be merged include multiple groups of first aggregation tables, wherein the keys included in the aggregation definitions corresponding to each group of first aggregation tables are the same.

[0087] Exemplarily, assume that aggregation definitions 1 to 3 include the same key, which is A; and aggregation definitions 4 and 5 include the same key, which is B. Then aggregation table 1 corresponding to aggregation definition 1 to aggregation table 3 corresponding to aggregation definition 3 are multiple first aggregation tables corresponding to the same key A, and aggregation table 4 corresponding to aggregation definition 4 and aggregation table 5 corresponding to aggregation definition 5 are multiple first aggregation tables corresponding to the same key B.

[0088] In the present application, multiple first aggregation tables corresponding to the same key in the M aggregation tables can be directly determined as the aggregation tables to be merged, which is simple and convenient to operate and improves the practicality of the technical solution of the present application.

[0089] 2. N aggregate tables to be merged, including aggregate tables corresponding to different keys.

[0090] In summary, corresponding to the aggregation tables with different keys, the network device estimates the aggregation rate generated by merging the different aggregation tables, and compares the aggregation rate with the first threshold, so as to determine whether these different aggregation tables are included in the N aggregation tables to be merged.

[0091] Specifically, in the case where M aggregation definitions include X second aggregation definitions with different corresponding keys, the network device estimates at least one aggregation rate generated by merging any 2 to X second aggregation tables in the X second aggregation tables corresponding to the X second aggregation definitions. Wherein, X is an integer greater than or equal to 2 and less than or equal to M. If there is at least one first target aggregation rate greater than the first threshold value in the at least one aggregation rate, then the network device determines that the N aggregation tables to be merged include at least one group of second aggregation tables corresponding to at least one first target aggregation rate, and each group of second aggregation tables in the at least one group of second aggregation tables includes multiple second aggregation tables.

[0092] It should be noted that the at least one aggregation rate generated by merging any 2 to X second aggregation tables mentioned above refers to the estimated aggregation rate of at least one new aggregation table obtained by merging multiple second aggregation tables. Exemplarily, assuming that the keys corresponding to aggregation tables 1 to 4 are all different, then the network device estimates the aggregation rate generated by combining any 2 to 4 aggregation tables in these 4 aggregation tables, including the aggregation rate of the merged aggregation table shown in Table 4 below:

[0093] Table 4

[0094]

[0095] As shown in Table 4, the network device estimates that the aggregation table 1 and the aggregation table 2 are merged, and the aggregation rate of the merged aggregation table is the aggregation rate A.

[0096] Exemplarily, assuming that among the aggregation rates A to K shown in Table 4, the aggregation rates of aggregation rate A, aggregation rate F and aggregation rate I are all greater than the first threshold, then the network device determines N aggregation tables to be merged, including: a group of aggregation tables corresponding to aggregation rate A (that is, aggregation table 1 and aggregation table 2), a group of aggregation tables corresponding to aggregation rate F (that is, aggregation table 3 and aggregation table 4), and a group of aggregation tables corresponding to aggregation rate I (that is, aggregation table 1, aggregation table 3 and aggregation table 4).

[0097] It should also be noted that the value of the first threshold can be set according to the needs of the actual application, and is not specifically limited here. Optionally, the less the storage resources (or free storage resources) of the device, the higher the value of the first threshold can be set; optionally, the better the computing performance of the network device, the lower the value of the first threshold can be set.

[0098] In some optional implementations, the network device may store at least one aggregation rate. When obtaining a new aggregation table corresponding to a base table, when determining whether the aggregation table can be merged with a previously obtained aggregation table, it is unnecessary to repeatedly calculate the aggregation table corresponding to the estimated merging scheme, thereby saving computing resources.

[0099] Exemplarily, assuming that based on the example of Table 4 above, the network device has obtained aggregation table 5, and the keys of aggregation table 5 are different from those of aggregation table 1 and aggregation table 2. The network device only needs to estimate the aggregation rate generated by merging aggregation table 1 and aggregation table 5, merging aggregation table 2 and aggregation table 2, and merging aggregation table 1, aggregation table 2 and aggregation table 5, and does not need to estimate the aggregation rate generated by merging aggregation table 1 and aggregation table 2.

[0100] In this application, by comparing the estimated aggregation rate generated by merging different aggregation tables corresponding to different keys with the first threshold, it is determined whether the aggregation tables with different keys can be merged, thereby balancing the computing resources and usage costs. In the scheme that can be merged, if there are identical values ​​in the aggregation tables with different keys, deduplication is performed during the merge, which further reduces the occupancy of storage resources and saves storage resources.

[0101] In the above solution, there is no limit on the number of times each aggregation table is merged, as long as the aggregation rate of the new aggregation table generated by the merger is greater than the first threshold. In other words, the same second aggregation table can be included in different groups of second aggregation tables. For example, the aggregation table 1 illustrated in Table 4 above is included in both a group of aggregation tables corresponding to the aggregation rate A and a group of aggregation tables corresponding to the aggregation rate I.

[0102] In some optional implementations, the multiple second aggregation tables included in each group of second aggregation tables are different from other aggregation tables in the X second aggregation tables. The other aggregation tables mentioned here refer to aggregation tables outside this group of second aggregation tables in the X second aggregation tables. In other words, if there is a second aggregation table whose estimated aggregation rates with different aggregation tables are all greater than the first threshold, then a method is selected from them to merge the second aggregation tables.

[0103] Exemplarily, assuming that in the example shown in Table 4 above, both the aggregation rate A and the aggregation rate I are greater than the first threshold, the network device will select a set of second aggregation tables as the aggregation tables to be merged from a set of aggregation tables corresponding to the aggregation rate A and a set of aggregation tables corresponding to the aggregation rate I. In other words, the network device determines that the N aggregation tables to be merged include "aggregation table 1 and aggregation table 2"; or include "aggregation table 1, aggregation table 3, and aggregation table 4".

[0104] In the present application, it is defined that the multiple second aggregation tables included in each group of second aggregation tables are different from other aggregation tables in the X second aggregation tables. Therefore, when merging, for the second aggregation tables whose estimated aggregation rates with different aggregation tables are greater than the first threshold, only one method can be selected, and it is not necessary to merge all of them, which simplifies the processing flow and improves processing efficiency.

[0105] When there are multiple aggregation rates that are greater than the first threshold and correspond to the same second aggregation table in at least one estimated aggregation rate, the network device has multiple ways to determine the first target aggregation rate from these multiple aggregation rates, thereby determining the N aggregation tables to be merged, which are explained below respectively.

[0106] In some optional embodiments, when there are multiple aggregation rates that are greater than a first threshold and correspond to the same second aggregation table among at least one estimated aggregation rate, the network device determines that the highest of the multiple aggregation rates is the first target aggregation rate, and thus determines that the aggregation table to be merged includes the aggregation table corresponding to the first target aggregation rate.

[0107] Exemplarily, assuming that in the example shown in Table 4 above, the aggregation rate B, the aggregation rate D, and the aggregation rate F are all greater than the first threshold, and these aggregation rates all correspond to the aggregation table 3. Then the network device determines the highest aggregation rate as the first target aggregation rate from the aggregation rate B, the aggregation rate D, and the aggregation rate F. Assuming that the aggregation rate F is the highest, the network device determines the aggregation rate F as the first target aggregation rate, and the aggregation table 3 and the aggregation table 4 are included in the N aggregation tables to be merged.

[0108] In this application, when there are multiple aggregation rates in the estimated aggregation table that are greater than the first threshold and correspond to the same second aggregation table, the highest of the multiple aggregation rates is selected as the first target aggregation rate. According to aggregation rate = (number of base table rows - number of aggregation table rows) / number of base table rows, the higher the aggregation rate, the less data in the aggregation table after merging, which further reduces the amount of storage and saves storage resources.

[0109] In some optional embodiments, when there are multiple aggregation rates that are greater than a first threshold and correspond to the same second aggregation table among at least one estimated aggregation rate, the network device can determine that the first target aggregation rate is the one with the largest number of corresponding second aggregation tables among the multiple aggregation rates, and thus determines that the aggregation tables to be merged include the aggregation table corresponding to the first target aggregation rate.

[0110] Exemplarily, it is assumed that in the example shown in Table 4 above, the aggregation rate C, the aggregation rate F, and the aggregation rate I are all greater than the first threshold, and these aggregation rates all correspond to the aggregation table 1. Since the aggregation rate C and the aggregation rate F each correspond to 2 aggregation tables, the aggregation rate I corresponds to 3 aggregation tables, and the number of aggregation tables corresponding to the aggregation rate I is the largest among these aggregation rates, the network device determines that the aggregation rate I is the first target aggregation rate, and the aggregation tables 1, 3, and 4 corresponding to the aggregation rate I are a group of second aggregation tables, which are included in the N aggregation tables to be merged.

[0111] In the present application, when there are multiple aggregation rates in the estimated aggregation table that are greater than the first threshold and correspond to the same second aggregation table, the one with the largest number of corresponding second aggregation tables among the multiple aggregation rates is selected as the first target aggregation rate, that is, the second aggregation tables with the largest number of merged numbers are merged, so that the merged aggregation table can not only include data of more dimensions as much as possible, but also reduce the amount of stored data and save storage resources.

[0112] 3. N aggregate tables to be merged, including aggregate tables corresponding to the same key and aggregate tables corresponding to different keys.

[0113] Combined with the above description, it can be seen that for multiple first aggregation tables corresponding to the same key, the network device can directly determine that these multiple aggregation tables are included in the N aggregation tables to be merged. The network device can also estimate the aggregation rate of merging these multiple first aggregation tables with other aggregation tables with different corresponding keys, so as to determine whether these multiple first aggregation tables can be merged with other aggregation tables. There are many possible ways for the network device to estimate the aggregation table generated by the merger, which are explained below:

[0114] In some optional implementations, multiple first aggregation tables corresponding to the same key may be merged to obtain a merged aggregation table. The aggregation rate of merging the merged aggregation table with other aggregation tables with different keys is estimated, and the aggregation rate is compared with the first threshold to determine whether these aggregation tables can be merged, that is, to determine whether these aggregation tables are included in the N aggregation tables to be merged.

[0115] Exemplarily, assume that the keys of aggregation tables 1 to 3 are all key A, the key of aggregation table 4 is key B, and the key of aggregation table 5 is key C. The network device merges aggregation tables 1 to 3 to obtain the merged aggregation table 6. Then, the four aggregation rates generated by merging aggregation table 4 and aggregation table 6, aggregation table 5 and aggregation table 6, aggregation table 4 and aggregation table 5, aggregation table 4, aggregation table 4 and aggregation table 6 are estimated. Then, based on the relevant instructions in the previous text "II. N aggregation tables to be merged, including aggregation tables corresponding to different keys", the first target aggregation table and multiple second aggregation tables corresponding to the first target aggregation rate are determined from these four aggregation rates. It should be noted that if any of the four aggregation rates corresponding to the aggregation rate of aggregation table 6 is not greater than the first threshold, the network device will still determine that the N aggregation tables to be merged include aggregation tables 1 to aggregation tables 3 that are merged to obtain aggregation table 6 and correspond to the same key.

[0116] In some optional implementations, the network device can also directly estimate the aggregation rate generated by merging the aggregation tables with different keys in the obtained aggregation tables, and determine the N aggregation tables to be merged. Exemplarily, assume that the keys of aggregation tables 1 to 3 are all key A, the key of aggregation table 4 is key B, and the key of aggregation table 5 is key C. The network device can estimate the aggregation rate generated by merging any multiple aggregation tables with different corresponding keys in these five aggregation tables, and then determine the first target aggregation table and multiple second aggregation tables corresponding to the first target aggregation rate from the estimated aggregation rate based on the relevant instructions in the previous "II. N aggregation tables to be merged, including aggregation tables with different keys".

[0117] In general, the network device can directly determine whether multiple first aggregation tables corresponding to the same key are included in the N aggregation tables to be merged. For the aggregation tables corresponding to different keys, it is also necessary to combine the merger estimation of multiple aggregation tables corresponding to different keys and compare the estimated aggregation rate with the first threshold to determine whether they are included in the N aggregation tables to be merged.

[0118] 303. The network device merges the N aggregation tables to be merged to obtain a merged aggregation table, and the data volume of the merged aggregation table is less than the sum of the data volumes of the N aggregation tables to be merged.

[0119] After determining the N aggregation tables to be merged, the network device will merge the N aggregation tables to be merged. Since the N aggregation tables to be merged may include multiple possible aggregation tables, the specific operations of the network device are also different.

[0120] In some optional implementations, the N aggregation tables to be merged include multiple first aggregation tables corresponding to the same key, and the network device directly merges the keys and values ​​corresponding to the multiple first aggregation tables. Figure 4 , Figure 4A schematic diagram provided for an embodiment of the present application:

[0121] First, we briefly explain the creation process of the aggregate table. Figure 4 As shown, the network device obtains multiple values, establishes a base table, or inserts the obtained values ​​into an already established base table. The network device can also perform an aggregation operation, aggregate the data in the base table according to the aggregation definition, and obtain an aggregate table. Both the base table and the aggregate table can respond to the query instruction of the subquery, thereby outputting the query result.

[0122] exist Figure 4 In the illustrated embodiment, it is assumed that two aggregate tables (aggtable) are created on the base table (table), wherein the aggregation definition of aggregate table 1 is "key-value pair: a--sum(b), min(text)"; and the aggregation definition of aggregate table 2 is "key-value pair: a--sum(b), avg(age)".

[0123] The aggregation definitions of these two aggregation tables include the same key: a; and the same value: sum(b), so the network device directly merges aggregation table 1 and aggregation table 2. That is, the same key a and the same value sum(b) are deduplicated, and the aggregation definition of the merged aggregation table is "key-value pair: a--sum(b), min(text), avg(age)".

[0124] By comparison Figure 4 As can be seen from the arrows on both sides, after the merge, the aggregation operation sum(b) only needs to be calculated once, instead of twice as before the merge. At the same time, the aggregation key a and the aggregation value sum(b) only need to be stored once, instead of twice as before the merge. While saving computing resources, storage resources are also reduced.

[0125] Figure 4 The illustrated embodiment takes different aggregation tables corresponding to the same key and corresponding to the same value as an example. In this scheme, the same values ​​are deduplicated during the merge. In actual applications, different aggregation tables corresponding to the same key may correspond to different values, and deduplication is not required in this case. For example, suppose that the aggregation definition of aggregation table 1 is "key-value pair: a--sum(b)"; the aggregation definition of aggregation table 2 is "key-value pair: a--avg(age)". The network device merges aggregation table 1 and aggregation table 2, and obtains the aggregation definition of the merged aggregation table as "key-value pair: a--sum(b), avg(age)".

[0126] in addition, Figure 4The embodiment shown takes a group of first aggregation tables corresponding to the same key as an example. In actual applications, there may be more groups of first aggregation tables, and the network devices merge them separately. For example, the aggregation definitions of aggregation tables 1 to 3 all include key A, the aggregation definitions of aggregation tables 4 to 7 all include key B, and the aggregation definitions of aggregation tables 8 to 10 all include key C. Then the network device merges aggregation tables 1 to 3, aggregation tables 4 to 7, and aggregation tables 8 to 10 respectively to obtain three merged aggregation tables.

[0127] in addition, Figure 4 The embodiment shown is based on the example that different aggregation tables have only one key. In actual applications, the aggregation table can also include more keys, which is not limited here. For example, assuming that the keys included in the aggregation definitions of aggregation tables 1 to 3 are all key A and key B, then the keys in the aggregation definition of the merged aggregation table obtained by merging aggregation table 1 and aggregation table 2 are still key A and key B.

[0128] In the present application, for multiple first aggregation tables corresponding to the same key in the M aggregation tables, multiple first aggregation tables can be directly determined as the aggregation tables to be merged, which is simple and convenient to operate, and improves the practicality of the technical solution of the present application. At the same time, when merging the aggregation tables corresponding to the same key, the same key will be deduplicated, reducing the number of storage copies of the same key, further reducing the waste of storage resources, and saving the user's usage cost.

[0129] In some optional implementations, the N aggregate tables to be merged include at least one group of second aggregate tables, and each group of second aggregate tables includes multiple second aggregate tables corresponding to different keys. In this solution, merging the N aggregate tables to be merged includes: merging the keys and values ​​corresponding to each group of second aggregate tables in at least one group of second aggregate tables, respectively. It can be understood that merging a group of second aggregate tables can obtain a merged aggregate table.

[0130] Exemplarily, it is assumed that the N aggregation tables to be merged include 2 groups of second aggregation tables, which are a first group of second aggregation tables including aggregation table 1 and aggregation table 2, and a second group of second aggregation tables including aggregation table 3 to aggregation table 5. The network device merges aggregation table 1 and aggregation table 2 to obtain a merged aggregation table; and merges aggregation table 3 to aggregation table 5 to obtain a merged aggregation table.

[0131] For example, suppose the network device determines that it needs to merge two aggregation tables corresponding to different keys, where the aggregation definition of aggregation table 1 is "key-value pair: a--sum(b), min(text)"; the aggregation definition of aggregation table 2 is "key-value pair: c--sum(b), avg(age)". These two aggregation definitions include the same value: sum(b). The network device will deduplicate the value, and the aggregation definition of the merged aggregation table is "key-value pair: a, c--sum(b), min(text), avg(age)".

[0132] Exemplarily, different aggregation tables with different keys and different values ​​may be merged as long as the network device determines in step 302 that these different aggregation tables are included in the N aggregation tables to be merged.

[0133] In the present application, by comparing the estimated aggregation rate generated by merging different aggregation tables corresponding to different keys with the first threshold, it is determined whether the aggregation tables corresponding to different keys can be merged, thereby balancing the computing power resources and usage costs. In the scheme that can be merged, if there are identical values ​​in the aggregation tables of different keys, deduplication is performed during the merge, which further reduces the occupancy of storage resources and saves storage resources. In general, after the network device obtains M aggregation tables corresponding to the same base table, it determines N aggregation tables to be merged from the M aggregation tables according to the aggregation definition of the aggregation table, and merges the N aggregation tables to obtain the merged aggregation table. The amount of data in the merged aggregation table is less than the sum of the data amounts of the N aggregation tables to be merged, that is, the data processing method provided by the present application reduces the number of times the same data in different aggregation tables is stored, thereby reducing the amount of data in the aggregation table and reducing the occupancy of storage resources.

[0134] In some optional implementations, after merging the N aggregation tables to be merged, the network device may also determine whether to change the data distribution method. In summary, the network device may obtain the aggregation rates of all the aggregation tables corresponding to the current moment, determine a second target aggregation rate greater than a second threshold from the aggregation rates of all the aggregation tables, and broadcast the data included in the aggregation table corresponding to the second target aggregation rate.

[0135] Among them, there are multiple possible situations for all the aggregation tables corresponding to the current moment: they may include the original aggregation table that has not been merged and the merged aggregation table, or include the merged aggregation table, or include the original aggregation table that has not been merged. The following uses the M mentioned above as 5, that is, the original aggregation tables obtained by the network device are 5, and takes aggregation tables 1 to 5 as examples to explain the possible situations respectively:

[0136] For example, if during data processing, the network device merges aggregation tables 1 to 3 to obtain a merged aggregation table 6, and merges aggregation tables 4 and 5 to obtain a merged aggregation table 7, then the network device obtains two aggregation tables after the merger, namely, aggregation table 6 and aggregation table 7.

[0137] For example, if during the data processing, the network device merges aggregation table 1 and aggregation table 2 to obtain the merged aggregation table 8, and determines that aggregation tables 3 to 5 cannot be merged. Then the network device finally obtains 4 aggregation tables, namely, aggregation table 8, aggregation table 3, aggregation table 4, and aggregation table 5.

[0138] Exemplarily, if during the data processing process, the network device determines that all aggregation tables cannot be merged, then the aggregation tables finally obtained by the network device are the original five aggregation tables, namely, aggregation table 1 to aggregation table 5.

[0139] The network device broadcasts the data included in the aggregation table corresponding to the second target aggregation rate, specifically, distributes the data of the aggregation table in the form of replicas. If the data of the original aggregation table is hash distributed or Rosin-Rammler distribution (RR distribution), the present application changes the data distribution form of the aggregation table.

[0140] In general, data broadcasting can be done by adding a broadcast operator after the aggregation (agg) operation to distribute the data in the aggregation table to each node in the system, so that each node has the full data. If the role of the network device in the system is CN, then the network device will broadcast the data in the aggregation table to all DNs that communicate with the CN; if the role of the network device in the system is DN, then the network device will broadcast the data in the aggregation table to all DNs that correspond to the same CN as the DN, as well as the CN corresponding to the DN.

[0141] The following diagram illustrates the replica distribution. Figure 5 , Figure 5 A schematic diagram provided for an embodiment of the present application.

[0142] exist Figure 5 In the illustrated embodiment, if the aggregation rates of the aggregation tables stored on DN1 and DN2 are both greater than the second threshold, the data in the aggregation tables stored on each DN will be broadcast. Figure 5In the "after broadcast" scenario, DN1 and DN2 execute the broadcast operator after the aggregation operation: for DN1, the data A stored in itself is broadcast to DN2, and data A refers to the data in the aggregation table stored in DN1 with an aggregation rate greater than the second threshold; for DN1, the data B stored in itself is broadcast to DN2, and data B refers to the data in the aggregation table stored in DN2 with an aggregation rate greater than the second threshold.

[0143] Assume that Figure 5 In the "before broadcast" scenario shown, the aggregation tables included in DN1 and DN2 are shown in Table 5 and Table 6 respectively:

[0144] Table 5

[0145] Key: a Value: sum_b 1 30 10 30 30 100

[0146] Table 6

[0147] Key: a Value: sum_b 20 90 40 200

[0148] If data broadcasting is not performed, only part of the data set is available on each DN. When executing a query (select) statement, it is necessary to scan all DNs and then summarize them on CN. If data broadcasting is performed, the newly generated data (30, 100) on DN1 is broadcast to DN2; the newly generated data (40, 200) on DN2 is broadcast to DN1. After the broadcast, both DN1 and DN2 have full data.

[0149] After the broadcast, the aggregation tables included in DN1 and DN2 are shown in Table 7 below:

[0150] Table 7

[0151] Key: a Value: sum_b 1 30 10 30 20 90 30 100 40 200

[0152] After broadcasting, each node in the system stores the full amount of aggregate table data. When querying data, you can select any node for data query. Optionally, you can select a node with a smaller load for data query, or the CN can specify a node with a smaller load to complete the data query. The load mentioned here can be understood as the idle resources of the node, which can be determined by the node's central processing unit (CPU) resources, memory resources, number of tasks, etc. For example, the more idle resources the CPU has, or the more idle memory resources, or the fewer tasks, the smaller the node load is.

[0153] In addition, it should be noted that the value of the second threshold can be set according to the needs of the actual application, and is not limited here. Optionally, the second threshold can be greater than the first threshold, or the second threshold can also be less than or equal to the first threshold, which is not specifically limited here.

[0154] In this application, the data in the aggregation table with an aggregation rate greater than the second threshold can be broadcast, so that when querying data, users can query at any node without the need for data transmission between nodes, which reduces the use of network resources and improves the reliability and convenience of the technical solution. In addition, if the query is directly performed on a node with less load, the efficiency of data query can also be improved.

[0155] In some optional implementations, the network device may perform the aggregation calculations of all aggregation tables corresponding to the base table in parallel. Specifically, the execution plan in the aggregation layer may be modified to multi-threaded parallel execution, so that when the base table corresponds to multiple aggregation tables, the aggregation calculations of the multiple aggregation tables are performed in parallel.

[0156] The following is a further explanation with the help of the schematic diagram. Figure 6 , Figure 6 A schematic diagram provided for an embodiment of the present application.

[0157] like Figure 6 As shown, it is assumed that the base table corresponds to two aggregation tables, where the aggregation definition of aggregation table 1 is "key-value pair: c--sum(b), min(text)"; the aggregation definition of aggregation table 2 is "key-value pair: a--sum(b), avg(age)".

[0158] According to the traditional method, Figure 6 As shown in "Before", the aggregation calculations of the two aggregation tables are executed serially, so the aggregation calculation of aggregation table 2 can only be performed after the aggregation calculation of aggregation table 1 is completed, resulting in slow aggregation calculation speed of the aggregation table. If there are many aggregation tables, slow aggregation calculation will lead to performance bottlenecks.

[0159] In the implementation of this application, Figure 6 In the parallel mode shown in "After", set the number of parallel executions to perform parallel calculations on multiple aggregation tables in the aggregation calculation layer. For example, parallelism can be achieved through thread pools or multithreading.

[0160] In this application, aggregation calculations are performed in parallel, which reduces the impact on the data import process and thus reduces the data processing delay.

[0161] In general, the data processing method introduced above can be summarized as follows: Figure 7 For the process shown, see Figure 7 , Figure 7A flowchart of a data processing method provided in an embodiment of the present application.

[0162] like Figure 7 As shown, the data processing method includes the following steps:

[0163] The network device obtains metadata of the aggregate table, which can indicate whether the obtained table is an aggregate table. The network device executes step ①. Determine whether there are more than one "aggregate table" in the base table of the imported data. If not, or there is only one, then generate an execution plan normally.

[0164] ②. If there are multiple "aggregate tables", when merging multiple "aggregate tables", it is determined whether the aggregate tables have the same aggregate key. If so, they are merged, and then the aggregate calculation operator is generated based on the key and value of the new "aggregate table" after the merger. The aggregate calculation operator is used to indicate the way to merge aggregate tables.

[0165] ③. If there is no identical aggregation key, the aggregation rate is evaluated after the merge. If the aggregation rate is greater than the set merge threshold (merge_threshold), the two "aggregation tables" can be merged, and then the aggregation calculation operator is generated based on the key and value of the new "aggregation table" after the merge.

[0166] ④. If the evaluated aggregation rate is less than the set merging threshold, merging is not possible and the execution plan process is generated normally. The merging threshold is the first threshold mentioned above.

[0167] ⑤. If there are multiple aggregations, get the number of aggregation tables, and then set the corresponding number of parallelism (set dop to n). When the operator is executed, start multiple threads and execute in parallel. Among them, n is an integer greater than or equal to 2, indicating that there are n threads / processes.

[0168] ⑥ Before generating the aggregation calculation operator, first evaluate the aggregation rate. If the aggregation rate is higher than the set aggregation threshold (agg_threshold), an additional broadcast operator is generated to broadcast the data to other computing nodes, that is, the data distribution is determined as a replica distribution. Among them, the merging threshold is the second threshold mentioned above.

[0169] Figure 7 The relevant steps of the embodiment shown in the previous text Figures 3 to 5 The illustrated embodiment has been described in detail, see above, and will not be described again here.

[0170] Next, the related equipment provided in the embodiments of the present application is described.

[0171] See also Figure 8 , Figure 8A schematic diagram of the structure of a data processing device provided in an embodiment of the present application. Figure 8 As shown, the data processing device 800 includes an acquisition unit 801 and a processing unit 802 .

[0172] In some optional embodiments, the acquisition unit 801 is used to obtain a base table and M aggregation tables, where the base table includes data actually stored in the data processing device, and the M aggregation tables are obtained by processing the data included in the base table according to M aggregation definitions. The M aggregation tables correspond one-to-one to the M aggregation definitions, and the M aggregation definitions are different. The aggregation definitions include key-value pairs, and the aggregation definitions are used to indicate the method of processing the data included in the base table. M is an integer greater than or equal to 2.

[0173] The processing unit 802 is used to determine N aggregation tables to be merged from the M aggregation tables according to the keys included in the M aggregation definitions, where N is an integer greater than or equal to 2 and less than or equal to M. The N aggregation tables to be merged are merged to obtain a merged aggregation table, and the data volume of the merged aggregation table is less than the sum of the data volumes of the N aggregation tables to be merged.

[0174] In some optional embodiments, the processing unit 802 is specifically used to: if there are multiple first aggregation definitions including the same key in the M aggregation definitions, then the data processing device determines from the M aggregation tables that the N aggregation tables to be merged include multiple first aggregation tables corresponding to the multiple first aggregation definitions; the data processing device merges the N aggregation tables to be merged, including: merging the keys and values ​​corresponding to the multiple first aggregation tables.

[0175] In some optional implementations, the processing unit 802 is specifically configured to: if the M aggregation definitions include X second aggregation definitions with different corresponding keys, the data processing device estimates at least one aggregation ratio generated by merging any 2 to X second aggregation tables corresponding to the X second aggregation definitions, where X is an integer greater than or equal to 2 and less than or equal to M;

[0176] The data processing device determines that the N aggregation tables to be merged include at least one group of second aggregation tables corresponding to at least one first target aggregation rate;

[0177] Among them, the first target aggregation rate is an aggregation rate that is greater than the first threshold among at least one aggregation rate, each group of second aggregation tables in at least one group of second aggregation tables includes multiple second aggregation tables, and the multiple second aggregation tables included in each group of second aggregation tables are different from other aggregation tables in the X second aggregation tables.

[0178] In some optional embodiments, the processing unit 802 is specifically used to: if among at least one aggregation rate, there are multiple aggregation rates that are greater than the first threshold and correspond to the same second aggregation table, then determine that the highest of the multiple aggregation rates is the first target aggregation rate; or, if among at least one aggregation rate, there are multiple aggregation rates that are greater than the first threshold and correspond to the same second aggregation table, then determine that among the multiple aggregation rates, the one with the largest number of corresponding second aggregation tables is the first target aggregation rate.

[0179] In some optional implementations, the processing unit 802 is specifically configured to: respectively merge the keys and values ​​corresponding to each group of second aggregate tables in at least one group of second aggregate tables.

[0180] In some optional implementations, the acquisition unit 801 is further configured to acquire the respective aggregation rates of all the aggregation tables corresponding to the base table;

[0181] The processing unit 802 is further configured to: determine a second target aggregation rate greater than a second threshold from the aggregation rates of all the aggregation tables; and broadcast the data included in the aggregation table corresponding to the second target aggregation rate.

[0182] In some optional implementations, the processing unit 802 is further configured to execute aggregation calculations of all aggregation tables corresponding to the base table in parallel.

[0183] The acquisition unit 801 and the processing unit 802 may be implemented by software or hardware. For example, the implementation of the processing unit 802 is described below by taking the processing unit 802 as an example. Similarly, the implementation of the acquisition unit 801 may refer to the implementation of the processing unit 802.

[0184] As an example of a software functional unit, the processing unit 802 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the processing unit 802 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Typically, a region may include multiple AZs.

[0185] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0186] As an example of a hardware functional unit, the processing unit 802 may include at least one computing device, such as a server, etc. Alternatively, the processing unit 802 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), etc. The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0187] The multiple computing devices included in the processing unit 802 can be distributed in the same region or in different regions. The multiple computing devices included in the processing unit 802 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the processing unit 802 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0188] It should be noted that the acquisition unit 801 and the processing unit 802 respectively implement different steps in the data processing method to achieve all the functions of the data processing device 800. The data processing device 800 is used for the aforementioned Figures 2 to 7 The operations performed by the network device in the illustrated embodiment are to implement the data processing method provided in the embodiment of the present application, which will not be described in detail here.

[0189] See also Fig. 9 , Fig. 9A schematic diagram of the structure of a computing device provided in an embodiment of the present application. The computing device 900 includes a processor 901, a communication interface 902, a bus 903, and a memory 904. The processor 901, the communication interface 902, and the memory 904 communicate with each other via the bus 903. In practical applications, communication can also be achieved through other means such as wireless transmission, which is not limited here.

[0190] The computing device 900 may be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 900 .

[0191] The processor 901 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP).

[0192] The communication interface 902 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or a communication network.

[0193] The bus 903 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 The bus 903 may include a path for transmitting information between various components of the computing device 900 (eg, the memory 904, the processor 901, and the communication interface 902).

[0194] The memory 904 may include a volatile memory, such as a random access memory (RAM). The memory 904 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0195] The memory 904 stores executable program codes, and the processor 901 executes the executable program codes to respectively implement the functions of the aforementioned acquisition unit 801 and processing unit 802, thereby implementing the data processing method. That is, the memory 904 stores instructions for executing the data processing method.

[0196] Alternatively, the memory 904 stores executable codes, and the processor 901 executes the executable codes to respectively implement the functions of the aforementioned acquisition unit 801 and processing unit 802, thereby implementing the data processing method. That is, the memory 904 stores instructions for executing the data processing method.

[0197] The embodiment of the present application also provides a computing device cluster, which includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some optional implementations, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0198] See also Fig.10 and Fig.11 , Fig.10 and Fig.11 All of them are structural schematic diagrams of the computing device cluster provided in embodiments of the present application.

[0199] like Fig.10 As shown, the computing device cluster includes at least one computing device 900. The memory 904 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing the data processing method provided in the embodiment of the present application.

[0200] In some possible implementations, the memory 904 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 904 may jointly execute instructions for executing the data processing method.

[0201] It should be noted that the memory 904 in different computing devices 900 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the data processing device. That is, the instructions stored in the memory 904 in different computing devices 900 can implement the functions of one or more units in the acquisition unit 801 and the processing unit 802.

[0202] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.11 A possible implementation is shown. Fig.11As shown, two computing devices 900A and 900B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 904 in the computing device 900A stores instructions for executing the functions of the acquisition unit 801. At the same time, the memory 904 in the computing device 900B stores instructions for executing the functions of the processing unit 802.

[0203] Fig.11 The connection method between the computing device clusters shown can be based on the consideration that in the data processing method provided in the present application, processing operations and operations other than processing operations are performed separately, that is, the function of the acquisition unit 801 is considered to be executed by the computing device 900A, and the function of the processing unit 802 is considered to be executed by the computing device 900B.

[0204] It should be understood that Fig.11 The functions of the computing device 900A shown in FIG. 9A may also be completed by multiple computing devices 900. Similarly, the functions of the computing device 900B may also be completed by multiple computing devices 900.

[0205] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig.10 and Fig.11 The connection method of the computing device cluster will not be described in detail here.

[0206] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computer device, the at least one computer device executes the above data processing method.

[0207] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned data processing method.

[0208] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0209] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: include: The network device obtains a base table and M aggregation tables, wherein the base table includes data actually stored by the network device, the M aggregation tables are obtained by processing the data included in the base table according to M aggregation definitions, the M aggregation tables correspond to the M aggregation definitions one by one, and the M aggregation definitions are different, the aggregation definitions include key-value pairs, and the aggregation definitions are used to indicate a method for processing the data included in the base table, and M is an integer greater than or equal to 2; The network device determines, according to the keys included in the M aggregation definitions, N aggregation tables to be merged from the M aggregation tables, where N is an integer greater than or equal to 2 and less than or equal to M; The network device merges the N aggregation tables to be merged to obtain a merged aggregation table, and the data volume of the merged aggregation table is less than the sum of the data volumes of the N aggregation tables to be merged.

2. The method according to claim 1, characterized in that The network device determines, according to the keys included in the M aggregation definitions, N aggregation tables to be merged from the M aggregation tables, including: If there are multiple first aggregation definitions including the same key in the M aggregation definitions, the network device determines, from the M aggregation tables, that the N aggregation tables to be merged include multiple first aggregation tables corresponding to the multiple first aggregation definitions; The network device merges the N aggregation tables to be merged, including: merging keys and values ​​corresponding to the multiple first aggregation tables.

3. The method according to claim 1 or 2, characterized in that: The network device determines, according to the keys included in the M aggregation definitions, N aggregation tables to be merged from the M aggregation tables, including: If the M aggregation definitions include X second aggregation definitions with different keys, the network device estimates at least one aggregation rate generated by merging any 2 to X second aggregation tables corresponding to the X second aggregation definitions, where X is an integer greater than or equal to 2 and less than or equal to M; The network device determines that the N aggregation tables to be merged include at least one group of second aggregation tables corresponding to at least one first target aggregation rate; The first target aggregation rate is an aggregation rate among the at least one aggregation rate that is greater than a first threshold value, each group of second aggregation tables in the at least one group of second aggregation tables includes multiple second aggregation tables, and the multiple second aggregation tables included in each group of second aggregation tables are different from other aggregation tables in the X second aggregation tables.

4. The method according to claim 3, characterized in that If, among the at least one aggregation rate, there are multiple aggregation rates that are greater than the first threshold and correspond to the same second aggregation table, the network device determines the highest of the multiple aggregation rates as the first target aggregation rate; or, If there are multiple aggregation rates among the at least one aggregation rate that are greater than the first threshold and correspond to the same second aggregation table, the network device determines that the first target aggregation rate is the one with the largest number of corresponding second aggregation tables among the multiple aggregation rates.

5. The method according to claim 3 or 4, characterized in that: The merging of the N aggregation tables to be merged includes: The network device respectively merges the keys and values ​​corresponding to each group of second aggregation tables in the at least one group of second aggregation tables.

6. The method according to any one of claims 1 to 5, characterized in that After merging the N aggregation tables to be merged, the method further includes: The network device obtains the respective aggregation rates of all the aggregation tables corresponding to the base table; The network device determines a second target aggregation rate greater than a second threshold from the respective aggregation rates of all the aggregation tables; The network device broadcasts the data included in the aggregation table corresponding to the second target aggregation rate.

7. A data processing device, characterized in that: include: an acquisition unit, configured to acquire a base table and M aggregation tables, wherein the base table includes data actually stored by the data processing device, the M aggregation tables are obtained by processing the data included in the base table according to M aggregation definitions, the M aggregation tables correspond to the M aggregation definitions one by one, and the M aggregation definitions are different, the aggregation definitions include key-value pairs, and the aggregation definitions are used to indicate a method for processing the data included in the base table, and M is an integer greater than or equal to 2; a processing unit, configured to determine N aggregation tables to be merged from the M aggregation tables according to the keys included in the M aggregation definitions, where N is an integer greater than or equal to 2 and less than or equal to M; The processing unit is further used to merge the N aggregation tables to be merged to obtain a merged aggregation table, and the data volume of the merged aggregation table is less than the sum of the data volumes of the N aggregation tables to be merged.

8. The data processing device according to claim 7, characterized in that: The processing unit is specifically used for: If there are multiple first aggregation definitions including the same key in the M aggregation definitions, the data processing device determines from the M aggregation tables that the N aggregation tables to be merged include multiple first aggregation tables corresponding to the multiple first aggregation definitions; The data processing device merges the N aggregation tables to be merged, including: merging keys and values ​​corresponding to the multiple first aggregation tables.

9. The data processing device according to claim 7 or 8, characterized in that: The processing unit is specifically used for: If the M aggregation definitions include X second aggregation definitions with different corresponding keys, the data processing device estimates at least one aggregation rate generated by merging any 2 to X second aggregation tables corresponding to the X second aggregation definitions, where X is an integer greater than or equal to 2 and less than or equal to M; The data processing device determines that the N aggregation tables to be merged include at least one group of second aggregation tables corresponding to at least one first target aggregation rate; The first target aggregation rate is an aggregation rate among the at least one aggregation rate that is greater than a first threshold value, each group of second aggregation tables in the at least one group of second aggregation tables includes multiple second aggregation tables, and the multiple second aggregation tables included in each group of second aggregation tables are different from other aggregation tables in the X second aggregation tables.

10. The data processing device according to claim 9, characterized in that: The processing unit is specifically used for: If, among the at least one aggregation rate, there are multiple aggregation rates that are greater than the first threshold and correspond to the same second aggregation table, then determining the highest of the multiple aggregation rates as the first target aggregation rate; or, If there are multiple aggregation rates among the at least one aggregation rate that are greater than the first threshold and correspond to the same second aggregation table, it is determined that the first target aggregation rate is the one with the largest number of corresponding second aggregation tables among the multiple aggregation rates.

11. The data processing device according to claim 9 or 10, characterized in that: The processing unit is specifically used for: The keys and values ​​corresponding to each group of second aggregate tables in the at least one group of second aggregate tables are merged respectively.

12. The data processing device according to any one of claims 7 to 11, characterized in that: The acquisition unit is further used to acquire the respective aggregation rates of all the aggregation tables corresponding to the base table; The processing unit is further used for: Determine a second target aggregation rate greater than a second threshold from the respective aggregation rates of all the aggregation tables; The data included in the aggregation table corresponding to the second target aggregation rate is broadcast.

13. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 6.

14. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer program instructions, which, when executed by a computing device cluster, cause the computing device cluster to perform the method according to any one of claims 1 to 6.

Images