Data processing method and related device

Abstract

A data processing method, used for reducing the occupation of storage resources and saving storage resources. The method is applied to a network device, and comprises: acquiring a base table and M aggregation tables, the base table comprising data actually stored by the network device, the M aggregation tables being obtained by processing, on the basis of M aggregation definitions, data comprised in the base table, the aggregation definitions respectively corresponding to the M aggregation tables being different, and each aggregation definition comprising a key-value pair and being used for indicating the means of processing the data comprised in the base table (301); on the basis of the keys comprised in the M aggregation definitions, determining N aggregation tables to be merged among the M aggregation tables (302); and merging the N aggregation tables to be merged to obtain a merged aggregation table, the data volume of the merged aggregation table being less than the sum of the data volumes of the N aggregation tables to be merged (303). M and N are both integers greater than or equal to 2, and N is less than or equal to M.

Description

A data processing method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 28, 2023, with application number 202311287944.7, and invention name “A data aggregation method, device and computing device cluster”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on December 26, 2023, with application number 202311824307.9, and invention name “A data processing method and related equipment”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of computers, and in particular to a data processing method and related equipment. Background Art

[0003] In data management, you can aggregate data in base tables according to specific rules to create aggregate tables. These rules can be understood as aggregation definitions, which consist of key-value pairs. Aggregate tables can also be considered preprocessing of base tables to facilitate further use.

[0004] In one data processing method, different aggregation definitions are used to aggregate data in a base table to generate different aggregate tables. In this method, different aggregate tables may contain duplicate data, that is, multiple copies of the same data are stored, wasting storage resources.

[0005] Summary of the Invention

[0006] The present application provides a data processing method and related equipment for reducing the occupation of storage resources and saving storage resources.

[0007] In a first aspect, the present application provides a data processing method, comprising:

[0008] The network device obtains a base table and M aggregation tables. The base table includes the data actually stored in the network device, and the M aggregation tables are obtained by processing the data included in the base table according to the M aggregation definitions. The M aggregation tables correspond one-to-one to the M aggregation definitions, and the M aggregation definitions are different, which means that the data included in each of the M aggregation tables is different. The aggregation definition includes a key-value pair. Therefore, different aggregation definitions mean that at least one of the keys and values ​​included in the aggregation definition is different. When the aggregation definition includes multiple keys or multiple values, any difference in any of them is also considered to be a different aggregation definition. In other words, the same aggregation definition means that each of the keys and values ​​included in the aggregation definition is the same. The aggregation definition is used to indicate the method of processing the data included in the base table including the aggregation table to obtain the aggregation table. Based on the keys included in the M aggregation definitions, N aggregation tables to be merged are determined from the M aggregation tables, and the N aggregation tables to be merged are merged to obtain the merged aggregation table. The amount of data in the merged aggregation table is less than the sum of the data amounts of the N aggregation tables to be merged. Among them, M and N are both integers greater than or equal to 2, and N is less than or equal to M. That is to say, the aggregate table to be merged may be part or all of the aggregate tables in the M aggregate tables. In addition, the N aggregate tables to be merged may be merged into one or more aggregate tables, that is, the number of aggregate tables after the merger is one or more, which is determined by the specific merging operation, which will be described in detail later. It should also be noted that the M aggregate tables here refer to all the aggregate tables corresponding to the base table at the current moment. These M aggregate tables may include the original aggregate table of the base table, that is, the aggregate table obtained by processing the data in the base table for the first time; it may also include the aggregate table obtained by aggregating the data in the base table multiple times; it may also include the aggregate tables shown in the above two schemes, which are not limited here.

[0009] In the present application, after obtaining M aggregate tables corresponding to the same base table, N aggregate tables to be merged are determined from the M aggregate tables according to the aggregation definition of the aggregate tables, and these N aggregate tables are merged to obtain a merged aggregate table. The amount of data in the merged aggregate table is less than the sum of the amount of data in the N aggregate tables to be merged. In other words, the data processing method provided by the present application reduces the number of times the same data is stored in different aggregate tables, thereby reducing the amount of data in the aggregate table and reducing the occupation of storage resources.

[0010] In some optional implementations of the first aspect, the network device may merge different aggregation tables whose aggregation definitions include the same key. Specifically, when there are multiple first aggregation definitions including the same key in the M aggregation definitions, it is determined that the N aggregation tables to be merged include multiple first aggregation tables corresponding to the multiple first aggregation definitions. The same key mentioned here means that each key included in the aggregation definition is the same. For example, aggregation definition 1 includes the key A+B, aggregation definition 2 includes the key A+B, and aggregation definition 3 includes the key A+C, then aggregation definition 1 and aggregation definition 2 are the multiple first aggregation tables mentioned here. In addition, the first aggregation table corresponding to the first aggregation definition mentioned here means that the aggregation definition adopted by the first aggregation table is the first aggregation definition. When it is determined that the N aggregation tables to be merged include the aforementioned multiple first aggregation tables, merging the N aggregation tables to be merged is actually merging the keys and values ​​corresponding to the multiple first aggregation tables. Since the keys corresponding to the multiple first aggregation tables are the same, merging the keys corresponding to the multiple first aggregation tables is to deduplicate the keys and only save one key. In addition, if there is duplicate data in the value, deduplication can also be performed. After merging multiple first aggregate tables, a merged aggregate table is obtained. In practical applications, there may be multiple groups of first aggregate tables. In this case, the merge is performed in groups. For example, if Aggregate Tables 1 through 3 have the same key, key1, and Aggregate Tables 4 and 5 have the same key, key2, then the merge is performed by merging Aggregate Tables 1 through 3 to obtain a merged aggregate table. Aggregate Tables 4 and 5 are then merged to obtain a merged aggregate table.

[0011] In this application, for multiple first aggregate tables corresponding to the same key in M ​​aggregate tables, multiple first aggregate tables can be directly determined as aggregate tables to be merged, which is simple and convenient to operate, and improves the practicality of the technical solution of this application. At the same time, when merging aggregate tables corresponding to the same key, the same key will be deduplicated, reducing the number of stored copies of the same key, further reducing the waste of storage resources, and saving users' usage costs.

[0012] In some optional implementations of the first aspect, the network device may merge different aggregation tables with different keys included in the aggregation definition. Specifically, in the case where M aggregation definitions include X second aggregation definitions with different keys, the network device may determine whether there is an aggregation table that can be merged among these X aggregation tables by estimating the aggregation rate of the aggregate table obtained by the merger. That is, determine whether there are N aggregation tables to be merged among these X aggregation tables. Wherein, X is an integer greater than or equal to 2 and less than or equal to M. The network device estimates at least one aggregation rate generated by merging any 2 to X second aggregation tables among the X second aggregation tables. The at least one aggregation rate generated here refers to the estimated aggregation rate of each new aggregation table obtained by merging multiple second aggregation tables. If there is a first target aggregation rate greater than a first threshold among the at least one aggregation rate, then the network device determines that the N aggregation tables to be merged include at least one group of second aggregation tables corresponding to at least one first target aggregation rate. Wherein, each group of second aggregation tables in the at least one group of second aggregation tables includes multiple second aggregation tables. Exemplarily, it is assumed that the aggregation rate of aggregate table A obtained by merging aggregate table 1 and aggregate table 2 with different merge keys estimated by the network device is greater than the first threshold, and the aggregation rate of aggregate table B obtained by merging aggregate table 1 and aggregate table 3 with different merge keys estimated by the network device is greater than the first threshold, that is, the network device determines that the first target aggregation rate includes the aggregation rate of aggregate table A and the aggregation rate of aggregate table B. Then, the network device can determine a group of second aggregate tables (i.e., aggregate table 1 and aggregate table 2) corresponding to the aggregation rate of the merged aggregate table A, and a group of second aggregate tables (i.e., aggregate table 1 and aggregate table 3) corresponding to the aggregation rate of the merged aggregate table B. Among them, the value of the first threshold can be set according to the needs of the actual application, and is not specifically limited here. Optionally, the fewer the storage resources (or idle storage resources) of the network device, the higher the value of the first threshold can be set; optionally, the better the computing performance of the network device, the lower the value of the first threshold can be set.

[0013] In this application, by comparing the estimated aggregation rate generated by merging different aggregation tables corresponding to different keys with a first threshold, it is determined whether the aggregation tables corresponding to different keys can be merged, thereby balancing computing power resources and usage costs. In the scheme where merging is possible, if there are identical values ​​in the aggregation tables of different keys, deduplication is performed during the merging, further reducing the occupancy of storage resources and saving storage resources.

[0014] In some optional implementations of the first aspect, the multiple second aggregate tables included in each group of second aggregate tables are also different from the other aggregate tables in the X second aggregate tables. The other aggregate tables mentioned here refer to aggregate tables outside this group of second aggregate tables in the X second aggregate tables. That is to say, if there is a second aggregate table whose estimated aggregation rates with different aggregate tables are all greater than the first threshold, then a method is selected from them to merge the second aggregate tables. For example, assuming that the aggregation rate generated by merging aggregate table 1 and aggregate table 2, and the aggregation rate generated by merging aggregate table 1 and aggregate table 3, are both greater than the first threshold, then a group of second aggregate tables is determined to be "aggregate table 1 and aggregate table 2", or "aggregate table 1 and aggregate table 3". Correspondingly, a method is selected from merging aggregate table 1 and aggregate table 2, and merging aggregate table 1 and aggregate table 3 to merge.

[0015] In the present application, it is defined that the multiple second aggregation tables included in each group of second aggregation tables are different from other aggregation tables in the X second aggregation tables. Therefore, when merging, for the second aggregation tables whose estimated aggregation rates with different aggregation tables are greater than the first threshold, only one method can be selected, and it is not necessary to merge all of them, which simplifies the processing flow and improves processing efficiency.

[0016] In some optional implementations of the first aspect, when there are multiple aggregation rates in at least one aggregation rate that are greater than the first threshold and correspond to the same second aggregation table, the network device has multiple ways to determine the first target aggregation rate.

[0017] Optionally, the network device determines that the highest of the multiple aggregation rates is the first target aggregation rate. That is to say, if the aggregation rates estimated by a second aggregation table with different aggregation tables are all greater than the first threshold, and when a method is selected to merge the second aggregation tables, the highest aggregation rate estimated by different aggregation tables is selected as the first target aggregation rate, then the second aggregation table to be merged is determined to be the second aggregation table corresponding to the first target aggregation rate. For example, the estimated aggregation rate A of aggregation table 1 and aggregation table 2 is the highest among all the aggregation rates corresponding to aggregation table 1, then it can be determined that the first target aggregation rate is aggregation rate A, and the aggregation tables that need to be merged are aggregation table 1 and aggregation table 2.

[0018] Optionally, the network device determines that the first target aggregation rate is the one with the largest number of second aggregation tables among multiple aggregation rates. That is to say, if the estimated aggregation rates of a second aggregation table and different aggregation tables are all greater than the first threshold, and when a method is selected to merge the second aggregation tables, the one with the largest number of corresponding aggregation tables is selected as the first target aggregation rate, then the second aggregation table to be merged is determined to be the second aggregation table corresponding to the first target aggregation rate. For example, if the estimated aggregation rates of aggregation table 1 and aggregation table 2, aggregation table 1 and aggregation table 3, and aggregation tables 1 to 3 are all greater than the first threshold, then it can be determined that the first target aggregation rate is the aggregation rate corresponding to aggregation tables 1 to 3, and the aggregation tables that need to be merged are aggregation table 1, aggregation table 2, and aggregation table 3.

[0019] In the present application, when there are multiple aggregation rates in the estimated aggregation table that are greater than the first threshold and correspond to the same second aggregation table, there are multiple ways to determine the first target aggregation rate corresponding to the N aggregation tables to be merged, which enriches the implementation method and application scenario of the technical solution of the present application and improves the flexibility of the technical solution. In the scheme of selecting the highest of multiple aggregation rates as the first target aggregation rate, according to aggregation rate = (number of base table rows - number of aggregation table rows) / number of base table rows, it can be seen that the higher the aggregation rate, the less data the aggregate table obtained after the merger, further reducing the amount of storage and saving storage resources. In the scheme of selecting the largest number of second aggregation tables corresponding to multiple aggregation rates as the first target aggregation rate, the largest number of second aggregation tables are merged, so that the merged aggregate table can reduce the amount of stored data in addition to including data of more dimensions as much as possible, saving storage resources.

[0020] In some optional implementations of the first aspect, the N aggregate tables to be merged include at least one group of second aggregate tables corresponding to at least one first target aggregation rate. The N aggregate tables to be merged are merged, that is, the keys and values ​​corresponding to each group of second aggregate tables in at least one group of second aggregate tables are merged respectively. It can be understood that merging a group of second aggregate tables can obtain a merged aggregate table. Exemplarily, it is assumed that the N aggregate tables to be merged include 2 groups of second aggregate tables, the first group of second aggregate tables includes aggregate table 1 and aggregate table 2, and the second group of second aggregate tables includes aggregate table 3 to aggregate table 5. Then when the merge is performed, the corresponding keys and values ​​of aggregate table 1 and aggregate table 2 are merged to obtain a merged aggregate table; and aggregate table 3 to aggregate table 5 are merged to obtain a merged aggregate table. When merging, deduplication operations are performed on the same values.

[0021] In this application, not only can aggregate tables corresponding to the same key be merged, but also aggregate tables corresponding to different keys can be merged to reduce the amount of stored data as much as possible. In addition, when merging aggregate tables corresponding to different keys, if the same value exists in different aggregates, deduplication is performed, reducing the number of stored copies of the same value, further reducing the waste of storage resources, and saving users' usage costs.

[0022] In some optional implementations of the first aspect, after merging the N aggregation tables to be merged, that is, after obtaining the merged aggregation table, the network device can also determine whether data broadcasting is required. Specifically, the network device obtains the aggregation rate of all aggregation tables corresponding to the base table. Here, all aggregation tables corresponding to the base table refer to all aggregation tables corresponding to the base table after the N aggregation tables are merged. That is to say, it includes the aggregation tables that have not been merged in the M aggregation tables, and the aggregation table obtained after the N aggregation tables are merged. The network device determines a second target aggregation rate that is greater than the second threshold from the aggregation rates of all aggregation tables, and broadcasts the data included in the aggregation table corresponding to the second target aggregation rate. Among them, the second threshold can be set according to the needs of the actual application and is not limited here.

[0023] In this application, data in the aggregation table with an aggregation rate greater than the second threshold can be broadcast, so that when querying data, users can query at any node without the need for data transmission between nodes, reducing the use of network resources and improving the reliability and convenience of the technical solution. In addition, if the query is directly performed on a node with less load, the efficiency of data query can be improved.

[0024] In some optional implementations of the first aspect, the network device performs aggregate calculations on all aggregate tables corresponding to the base table in parallel. Specifically, the execution plan in the aggregate calculation layer may be modified to execute in parallel using multiple threads, so that when a base table corresponds to multiple aggregate tables, the aggregate calculations on these multiple aggregate tables are performed in parallel.

[0025] In this application, aggregation calculations are performed in parallel, which reduces the impact on the data import process and thus reduces the data processing delay.

[0026] In a second aspect, the present application provides a data processing device, comprising:

[0027] An acquisition unit is used to acquire a base table and M aggregate tables. The base table includes data actually stored in a data processing device. The M aggregate tables are obtained by processing the data included in the base table according to M aggregate definitions. The M aggregate tables correspond one-to-one to the M aggregate definitions, and the M aggregate definitions are different. The aggregate definitions include key-value pairs. The aggregate definitions are used to indicate the method of processing the data included in the base table. M is an integer greater than or equal to 2.

[0028] The processing unit is configured to determine, based on the keys included in the M aggregation definitions, N aggregation tables to be merged from the M aggregation tables, where N is an integer greater than or equal to 2 and less than or equal to M. The processing unit is configured to merge the N aggregation tables to be merged to obtain a merged aggregation table, where the amount of data in the merged aggregation table is less than the sum of the amounts of data in the N aggregation tables to be merged.

[0029] The data processing device is used to implement the method shown in the aforementioned first aspect or any possible implementation of the first aspect.

[0030] In a third aspect, the present application provides a computing device comprising a processor and a memory. The processor of the computing device is configured to execute instructions stored in the memory, so that the computing device implements the method described in the first aspect or any possible implementation of the first aspect.

[0031] In a fourth aspect, the present application provides a computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster implements the method disclosed in the first aspect and any possible implementation of the first aspect.

[0032] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a processor, implement the method shown in the aforementioned first aspect or any possible implementation of the first aspect; or, when the instruction is run by a computer device cluster, enables the computer device cluster to implement the method disclosed in the first aspect and any possible implementation of the first aspect.

[0033] In a sixth aspect, the present application provides a computer-readable storage medium, which stores computer program instructions. When the computer program instructions are executed on a processor, the method shown in the aforementioned first aspect or any possible implementation of the first aspect is implemented; or, when the computer program instructions are executed by a computer device cluster, the computer device cluster implements the method disclosed in the first aspect and any possible implementation of the first aspect.

[0034] The beneficial effects shown in any of the second to sixth aspects are similar to those of the first aspect or any possible implementation method of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] FIG1 is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0036] FIG2 is a schematic diagram of a system architecture provided in an embodiment of the present application;

[0037] FIG3 is a flow chart of a data processing method provided in an embodiment of the present application;

[0038] FIG4 is a schematic diagram provided in an embodiment of the present application;

[0039] FIG5 is another schematic diagram provided in an embodiment of the present application;

[0040] FIG6 is another schematic diagram provided in an embodiment of the present application;

[0041] FIG7 is another flow chart of the data processing method provided in an embodiment of the present application;

[0042] FIG8 is a schematic structural diagram of a communication device provided in an embodiment of the present application;

[0043] FIG9 is another schematic structural diagram of a communication device provided in an embodiment of the present application;

[0044] FIG10 is a schematic diagram of a structure of a computing device cluster provided in an embodiment of the present application;

[0045] FIG11 is another structural diagram of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The embodiment of the present application provides a data processing method for reducing the occupancy of storage resources and saving storage resources.

[0047] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0048] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way are interchangeable when appropriate, and this is merely a way of distinguishing objects of the same attributes when describing the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or device comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or devices. In addition, "at least one" refers to one or more, and "a plurality" refers to two or more. "and / or" describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent: the situation where A exists alone, A and B exist simultaneously, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0049] First, the proper nouns and related concepts that may be involved in the embodiments of the present application are explained.

[0050] 1. Cloud Data Warehouse:

[0051] A cloud data warehouse, also known as a cloud data warehouse, is a data storage and management service based on cloud computing technology. It stores and processes an enterprise's big data on a cloud platform. Compared to traditional data warehouses, cloud data warehouses offer lower costs, greater scalability, and faster response times. They store data in the cloud and, leveraging the elasticity and flexibility of cloud computing, can quickly adjust storage capacity and computing resources as needed. The data processing methods provided in the embodiments of this application can be applied in cloud data warehouse scenarios.

[0052] 2. Base table, aggregate table, and aggregation rate:

[0053] A base table is a table permanently stored in a database and is a logical representation of the actual stored data. For network devices that use a database, the data included in the base table is the data actually stored by the network device.

[0054] Aggregate tables are created by processing data from base tables according to specific rules. The idea behind aggregate tables is that when importing data, pre-created aggregate definitions are used to group data by the defined aggregate key. Aggregate values ​​are then calculated based on the defined aggregate key, resulting in aggregate keys and values. These keys and values ​​are then written to the same table, the aggregate table. This method compresses the dataset. Subsequent queries can scan the aggregate table, improving query performance.

[0055] Aggregation ratio = (number of base table rows - number of aggregate table rows) / number of base table rows, which means that the higher the aggregation ratio, the smaller the amount of data in the aggregated data set and the better the performance.

[0056] 3. Cloud management platform and infrastructure:

[0057] The cloud management platform is used to manage the cloud vendor's infrastructure, which is multiple cloud data centers located in different regions. Each region has at least one cloud data center. The cloud management platform can provide interfaces related to cloud computing services, such as configuration pages (i.e., interfaces) or APIs for tenants to access cloud services. Tenants can log in to the cloud management platform using a pre-registered account and password, and after successful login, they can select and purchase cloud services provided by the cloud data center in the predetermined region, such as object storage services, virtual machine services, container services, or other known cloud services.

[0058] 4. Tenant:

[0059] The top-level object used to manage cloud services and / or cloud resources. Tenants register a tenant account and set a tenant password on the cloud management platform through a local client (such as a browser). The local client remotely logs in to the cloud management platform through the tenant account and set tenant password. The cloud management platform provides a configuration interface or API for tenants to configure and use cloud services. As mentioned above, cloud services are specifically provided by the infrastructure managed by the cloud management platform.

[0060] Below, please refer to Figures 1 and 2, which are both schematic diagrams of the system architecture provided by the embodiments of the present application.

[0061] As shown in Figure 1, a tenant logs into cloud service platform 30 via client 10 over the internet 20, using the account and password registered on cloud service platform 30. Cloud service platform 30 manages infrastructure, which includes multiple data centers located in different regions. For example, Region 1 in Figure 1 includes Cloud Data Center 1 and Cloud Data Center 2, while Region 2 includes Cloud Data Center 3 and Cloud Data Center 4. Each cloud data center is equipped with multiple servers, each of which runs a service instance (including at least one of virtual machines, containers, and dedicated hosts).

[0062] In the embodiment of the present application, a data management service is deployed in the business instance. A tenant purchases cloud services from a cloud service platform 30 through a client. The tenant sends a call request to the cloud service platform 30, requesting cloud services from the cloud service platform 30. The cloud service specifically provides data management services to the tenant, including data storage, data transmission, and data processing, which will be described later.

[0063] As shown in Figure 2, the system includes a coordinator node (CN) and Z computing nodes (DN). Z is an integer greater than 2. CN is used to send execution plans and data to DN; DN is used to perform calculations, transmit data, etc. The data processing method provided in the embodiment of the present application can be used for both CN and DN. The application scenarios include interaction between DN and CN, or interaction between DNs under the same CN, which is not limited here.

[0064] The embodiments of this application do not limit the product form of CN and DN. They can be virtual machines or physical machines on the cloud, etc., and can also be applied in non-cloud scenarios. No specific limitations are made here.

[0065] It should be noted that the embodiment shown in FIG2 is based on an example in which a system includes Z+1 nodes, and does not limit the number of nodes in the system. In actual applications, the system may also include a greater or lesser number of network devices, which is not specifically limited here.

[0066] Next, please refer to FIG3 , which is a flow chart of a data processing method provided in an embodiment of the present application, including:

[0067] 301. The network device obtains a base table and M aggregation tables. The base table includes the data actually stored in the network device. The M aggregation tables are obtained by processing the data included in the base table according to M aggregation definitions. The M aggregation tables correspond to the M aggregation definitions one-to-one, and the M aggregation definitions are different. The aggregation definitions include key-value pairs. The aggregation definitions are used to indicate the method of processing the data included in the base table. M is an integer greater than or equal to 2.

[0068] During the data import process, the network device obtains a base table and M aggregate tables. These aggregate tables are derived by processing the data in the base table according to the aggregation definition. The data in the base table can be part or all of the data in the base table. In other words, the data source of the aggregate table is part or all of the data in the base table.

[0069] An aggregate definition consists of key-value pairs that define the keys and values ​​of an aggregate table. This instructs the network device on how to process the data in the base table to create the aggregate table. Each of the M aggregate tables has a corresponding aggregate definition; that is, each aggregate table is derived by processing the data in the base table according to an aggregate definition.

[0070] The M aggregation tables each correspond to a different aggregation definition. Different aggregation definitions here mean that at least one of the keys and values ​​included in the aggregation definition is different. When an aggregation definition includes multiple keys or multiple values, a difference in any of these keys is also considered a different aggregation definition. In other words, the same aggregation definition means that every key and value included in the aggregation definition is the same.

[0071] For example, it is assumed that the base table is as shown in Table 1 below:

[0072] Table 1

[0073] If the aggregation definition of aggregation table 1 includes a key-value pair of "name-sum(score)", then aggregation table 1 is shown in Table 2 below. In this case, sum(score) represents the sum of the scores.

[0074] Table 2

[0075] If the aggregation definition of aggregation table 2 includes the key-value pair "name-age", then aggregation table 2 is as shown in Table 3 below:

[0076] Table 3

[0077] As shown in Tables 2 and 3, the aggregate table groups the data in the base table according to the keys included in the aggregate definition, achieving data compression. Subsequent data queries can scan and query from the aggregate table, improving query performance.

[0078] It should be noted that the aforementioned Tables 1 to 3 are only examples of base tables and aggregate tables, and do not constitute a limitation on the representation of base tables or aggregate tables. In actual applications, base tables or aggregate tables can also be in other representations, such as tree structures, key-value pairs, etc., which are not specifically limited here.

[0079] 302. The network device determines N aggregation tables to be merged from the M aggregation tables according to the keys included in the M aggregation definitions, where N is an integer greater than or equal to 2 and less than or equal to M.

[0080] Different aggregation definitions may contain the same or different keys. Consequently, different aggregation tables may correspond to the same or different keys. Network devices handle different situations differently, as described below.

[0081] 1. N aggregate tables to be merged, including aggregate tables corresponding to the same key.

[0082] The aggregation definition of each aggregation table includes the key of the aggregation table. When there are multiple first aggregation definitions including the same key among the M aggregation definitions, the network device determines that the N aggregation tables to be merged include the multiple first aggregation tables corresponding to the multiple first aggregation definitions. In addition, the first aggregation table corresponding to the first aggregation definition mentioned here refers to the aggregation definition used by the first aggregation table as the first aggregation definition.

[0083] Because an aggregate definition can include one or more keys, the phrase "multiple first aggregate definitions including the same key" here refers to different aggregate table definitions including the same key. For example, if aggregate definition 1 and aggregate definition 2 include keys A+B, and aggregate definition 3 includes keys A+C, then aggregate definition 1 and aggregate definition 2 are referred to as multiple first aggregate definitions. Aggregate table 1 corresponding to aggregate definition 1 and aggregate table table 2 corresponding to aggregate definition 2 are referred to as multiple first aggregate tables.

[0084] In some optional implementations, there may be multiple groups of first aggregation definitions, that is, the N aggregation tables to be merged include multiple groups of first aggregation tables, wherein the aggregation definitions corresponding to each group of first aggregation tables include the same key.

[0085] For example, assume that aggregation definitions 1 through 3 include the same key, A; and that aggregation definitions 4 and 5 include the same key, B. Then, aggregation table 1 corresponding to aggregation definition 1 through aggregation table 3 corresponding to aggregation definition 3 are multiple first aggregation tables corresponding to the same key, A; and aggregation table 4 corresponding to aggregation definition 4 and aggregation table 5 corresponding to aggregation definition 5 are multiple first aggregation tables corresponding to the same key, B.

[0086] In the present application, multiple first aggregation tables corresponding to the same key in the M aggregation tables can be directly determined as the aggregation tables to be merged. The operation is simple and convenient, which improves the practicality of the technical solution of the present application.

[0087] 2. N aggregate tables to be merged, including aggregate tables corresponding to different keys.

[0088] In summary, corresponding to aggregation tables with different keys, the network device estimates an aggregation ratio generated by merging different aggregation tables, and compares the aggregation ratio with a first threshold, thereby determining whether these different aggregation tables are included in the N aggregation tables to be merged.

[0089] Specifically, when M aggregation definitions include X second aggregation definitions with different corresponding keys, the network device estimates at least one aggregation ratio generated by merging any 2 to X second aggregation tables from the X second aggregation tables corresponding to the X second aggregation definitions. X is an integer greater than or equal to 2 and less than or equal to M. If at least one first target aggregation ratio is greater than a first threshold value among the at least one aggregation ratio, the network device determines that the N aggregation tables to be merged include at least one group of second aggregation tables corresponding to the at least one first target aggregation ratio, and each group of second aggregation tables in the at least one group of second aggregation tables includes multiple second aggregation tables.

[0090] It should be noted that the at least one aggregation rate generated by merging any 2 to X second aggregation tables mentioned above refers to the estimated aggregation rate of each of the at least one new aggregation tables obtained by merging multiple second aggregation tables. For example, assuming that the keys corresponding to aggregation tables 1 to 4 are all different, the network device estimates the aggregation rates generated by combining any 2 to 4 of these 4 aggregation tables, including the aggregation rates of the merged aggregation tables shown in Table 4 below:

[0091] Table 4

[0092] As shown in Table 4, the network device estimates that the aggregation rate of the merged aggregation table obtained by merging aggregation table 1 and aggregation table 2 is aggregation rate A.

[0093] For example, assuming that among the aggregation rates A to K shown in Table 4, the aggregation rates of aggregation rate A, aggregation rate F and aggregation rate I are all greater than the first threshold, then the network device determines the N aggregation tables to be merged, including: a group of aggregation tables corresponding to the aggregation rate A (that is, aggregation table 1 and aggregation table 2), a group of aggregation tables corresponding to the aggregation rate F (that is, aggregation table 3 and aggregation table 4), and a group of aggregation tables corresponding to the aggregation rate I (that is, aggregation table 1, aggregation table 3 and aggregation table 4).

[0094] It should also be noted that the value of the first threshold can be set according to actual application needs and is not specifically limited here. Optionally, the less storage resources (or free storage resources) a device has, the higher the value of the first threshold can be set; optionally, the better the computing performance of the network device, the lower the value of the first threshold can be set.

[0095] In some optional implementations, the network device may store at least one aggregation rate. When obtaining a new aggregation table corresponding to a base table, when determining whether the aggregation table can be merged with the previously obtained aggregation table, there is no need to repeatedly calculate the aggregation table corresponding to the estimated merging scheme, thereby saving computing resources.

[0096] For example, assuming that based on the example of Table 4 above, the network device obtains aggregation table 5, and the keys of aggregation table 5 are different from those of aggregation table 1 and aggregation table 2, the network device only needs to estimate the aggregation rate generated by merging aggregation table 1 and aggregation table 5, merging aggregation table 2 and aggregation table 2, and merging aggregation table 1, aggregation table 2 and aggregation table 5, and does not need to estimate the aggregation rate generated by merging aggregation table 1 and aggregation table 2.

[0097] In this application, by comparing the estimated aggregation rate generated by merging different aggregation tables corresponding to different keys with a first threshold, it is determined whether aggregation tables with different keys can be merged, thereby balancing computing power resources and usage costs. In the scheme where merging is possible, if there are identical values ​​in aggregation tables with different keys, deduplication is performed during the merge, further reducing the occupancy of storage resources and saving storage resources.

[0098] In the above solution, there's no limit on the number of times each aggregation table can be merged; as long as the aggregation rate of the new aggregation table generated by the merger is greater than the first threshold, it can be used. In other words, the same second aggregation table can be included in different groups of second aggregation tables. For example, in Table 4 above, aggregation table 1 is included in both the group of aggregation tables corresponding to aggregation rate A and the group of aggregation tables corresponding to aggregation rate I.

[0099] In some optional embodiments, the multiple second aggregation tables included in each group of second aggregation tables are different from the other aggregation tables in the X second aggregation tables. The other aggregation tables referred to herein are aggregation tables outside the group of second aggregation tables in the X second aggregation tables. In other words, if there is a second aggregation table whose estimated aggregation rate with different aggregation tables is greater than the first threshold, then one of the methods is selected to merge the second aggregation tables.

[0100] For example, assuming that in the example shown in Table 4 above, both aggregation ratio A and aggregation ratio I are greater than the first threshold, the network device selects a second set of aggregation tables as the aggregation tables to be merged from the set of aggregation tables corresponding to aggregation ratio A and the set of aggregation tables corresponding to aggregation ratio I. In other words, the network device determines that the N aggregation tables to be merged include "aggregation table 1 and aggregation table 2"; or, alternatively, include "aggregation table 1, aggregation table 3, and aggregation table 4."

[0101] In the present application, it is defined that the multiple second aggregation tables included in each group of second aggregation tables are different from other aggregation tables in the X second aggregation tables. Therefore, when merging, for the second aggregation tables whose estimated aggregation rates with different aggregation tables are greater than the first threshold, only one method can be selected, and it is not necessary to merge all of them, which simplifies the processing flow and improves processing efficiency.

[0102] When there are multiple aggregation rates in at least one estimated aggregation rate that are greater than the first threshold and correspond to the same second aggregation table, the network device has multiple ways to determine the first target aggregation rate from these multiple aggregation rates, thereby determining the N aggregation tables to be merged, which are explained below.

[0103] In some optional embodiments, when there are multiple aggregation rates greater than a first threshold value and corresponding to the same second aggregation table among at least one estimated aggregation rate, the network device determines that the highest of the multiple aggregation rates is the first target aggregation rate, and thus determines that the aggregation table to be merged includes the aggregation table corresponding to the first target aggregation rate.

[0104] For example, assuming that in the example shown in Table 4 above, aggregation rates B, D, and F are all greater than the first threshold, and that these aggregation rates all correspond to aggregation table 3. The network device then determines the highest aggregation rate among aggregation rates B, D, and F as the first target aggregation rate. Assuming that aggregation rate F is the highest, the network device determines aggregation rate F as the first target aggregation rate, and aggregation tables 3 and 4 are included in the N aggregation tables to be merged.

[0105] In this application, if there are multiple aggregation rates in the estimated aggregation table that are greater than the first threshold and correspond to the same second aggregation table, the highest of the multiple aggregation rates is selected as the first target aggregation rate. According to the formula aggregation rate = (number of base table rows - number of aggregation table rows) / number of base table rows, the higher the aggregation rate, the less data in the aggregated table after merging, further reducing the amount of storage and saving storage resources.

[0106] In some optional embodiments, when there are multiple aggregation rates that are greater than a first threshold and correspond to the same second aggregation table among at least one estimated aggregation rate, the network device can determine that the first target aggregation rate is the one with the largest number of corresponding second aggregation tables among the multiple aggregation rates, and thus determine that the aggregation tables to be merged include the aggregation tables corresponding to the first target aggregation rate.

[0107] For example, assume that in the example shown in Table 4 above, aggregation rates C, F, and I are all greater than the first threshold, and each of these aggregation rates corresponds to aggregation table 1. Since aggregation rates C and F each correspond to two aggregation tables, and aggregation rate I corresponds to three aggregation tables, aggregation rate I corresponds to the largest number of aggregation tables among these aggregation rates, the network device determines aggregation rate I as the first target aggregation rate, and aggregation tables 1, 3, and 4 corresponding to aggregation rate I form a set of second aggregation tables, which are included in the N aggregation tables to be merged.

[0108] In the present application, when there are multiple aggregation rates in the estimated aggregation table that are greater than the first threshold and correspond to the same second aggregation table, the one with the largest number of second aggregation tables corresponding to the multiple aggregation rates is selected as the first target aggregation rate, that is, the second aggregation tables with the largest number of numbers are merged, so that the merged aggregation table can not only include data of more dimensions as much as possible, but also reduce the amount of stored data and save storage resources.

[0109] 3. N aggregate tables to be merged, including aggregate tables corresponding to the same key and aggregate tables corresponding to different keys.

[0110] Combined with the above description, it can be seen that for multiple first aggregation tables corresponding to the same key, the network device can directly determine that these multiple aggregation tables are included in the N aggregation tables to be merged. The network device can also estimate the aggregation rate of merging these multiple first aggregation tables with other aggregation tables with different corresponding keys, thereby determining whether these multiple first aggregation tables can be merged with other aggregation tables. There are many possible ways for the network device to estimate the aggregation table generated by the merger, which are explained below:

[0111] In some optional implementations, multiple first aggregation tables corresponding to the same key may be merged to obtain a merged aggregation table. An aggregation rate of merging the merged aggregation table with other aggregation tables with different keys is estimated, and the aggregation rate is compared with a first threshold to determine whether these aggregation tables can be merged, that is, to determine whether these aggregation tables are included in the N aggregation tables to be merged.

[0112] Exemplarily, it is assumed that the keys of aggregation tables 1 to 3 are all key A, the key of aggregation table 4 is key B, and the key of aggregation table 5 is key C. The network device merges aggregation tables 1 to 3 to obtain the merged aggregation table 6. Then, the four aggregation rates generated by merging aggregation table 4 and aggregation table 6, aggregation table 5 and aggregation table 6, aggregation table 4 and aggregation table 5, aggregation table 4, aggregation table 4 and aggregation table 6 are estimated. Then, based on the relevant instructions in the previous "II. N aggregation tables to be merged, including aggregation tables corresponding to different keys", the first target aggregation table and the multiple second aggregation tables corresponding to the first target aggregation rate are determined from these four aggregation rates. It should be noted that if any of the four aggregation rates corresponding to aggregation table 6 is not greater than the first threshold, the network device will still determine that the N aggregation tables to be merged include aggregation table 1 to aggregation table 3 that are merged to obtain aggregation table 6 and correspond to the same key.

[0113] In some optional implementations, the network device can also directly estimate the aggregation rate generated by merging the aggregation tables with different keys in the obtained aggregation tables, and determine the N aggregation tables to be merged. For example, it is assumed that the keys of aggregation tables 1 to 3 are all key A, the key of aggregation table 4 is key B, and the key of aggregation table 5 is key C. The network device can estimate the aggregation rate generated by merging any multiple aggregation tables with different corresponding keys in these five aggregation tables, and then determine the first target aggregation table and multiple second aggregation tables corresponding to the first target aggregation rate from the estimated aggregation rate based on the relevant instructions in the previous "II. N aggregation tables to be merged, including aggregation tables corresponding to different keys".

[0114] In general, the network device can directly determine whether multiple first aggregation tables corresponding to the same key are included in the N aggregation tables to be merged. For aggregation tables corresponding to different keys, it is also necessary to combine the merger estimation of multiple aggregation tables corresponding to different keys and compare the estimated aggregation rate with the first threshold to determine whether they are included in the N aggregation tables to be merged.

[0115] 303. The network device merges the N aggregation tables to be merged to obtain a merged aggregation table, where the amount of data in the merged aggregation table is less than the sum of the amount of data in the N aggregation tables to be merged.

[0116] After determining the N aggregation tables to be merged, the network device will merge the N aggregation tables to be merged. Since the N aggregation tables to be merged may include multiple aggregation tables, the specific operations of the network device are also different.

[0117] In some optional implementations, the N aggregation tables to be merged include multiple first aggregation tables corresponding to the same key, and the network device directly merges the keys and values ​​corresponding to these multiple first aggregation tables. The following is an illustration with reference to FIG4 , which is a schematic diagram provided in an embodiment of the present application:

[0118] First, let's briefly explain the process of creating an aggregate table. As shown in Figure 4, a network device retrieves multiple values ​​and creates a base table or inserts the retrieved values ​​into an existing base table. Network devices can also perform an aggregation (agg) operation, aggregating the data in a base table according to an aggregation definition to create an aggregate table. Both base and aggregate tables can respond to subquery queries and output query results.

[0119] In the embodiment shown in Figure 4, it is assumed that two aggregate tables (aggtable) are created on the base table (table), where the aggregation definition of aggregate table 1 is "key-value pair: a--sum(b), min(text)"; the aggregation definition of aggregate table 2 is "key-value pair: a--sum(b), avg(age)".

[0120] If the aggregation definitions of these two aggregation tables contain the same key, a, and the same value, sum(b), then the network device directly merges aggregation tables 1 and 2. That is, the network device removes duplicates for the same key, a, and sum(b), resulting in an aggregation definition for the merged aggregation table: "key-value pair: a - sum(b), min(text), avg(age)."

[0121] Comparing the arrows in Figure 4, we can see that after the merge, the aggregation operation sum(b) only requires one aggregation calculation, rather than two before the merge. Furthermore, both the aggregation key a and the aggregate value sum(b) only need to be stored once, rather than twice. This saves computing resources and reduces storage resources.

[0122] The embodiment shown in Figure 4 takes different aggregation tables corresponding to the same key and corresponding to the same value as an example. In this solution, the same values ​​are deduplicated during the merge. In actual applications, different aggregation tables corresponding to the same key may correspond to different values, and deduplication is not required in this case. For example, suppose that the aggregation definition of aggregation table 1 is "key-value pair: a--sum(b)"; the aggregation definition of aggregation table 2 is "key-value pair: a--avg(age)". The network device merges aggregation table 1 and aggregation table 2, and the aggregation definition of the merged aggregation table is "key-value pair: a--sum(b), avg(age)".

[0123] In addition, the embodiment shown in Figure 4 uses a group of first aggregation tables corresponding to the same key as an example. In actual applications, there may be more groups of first aggregation tables, and the network device may merge them separately. For example, the aggregation definitions of aggregation tables 1 to 3 all include key A, the aggregation definitions of aggregation tables 4 to 7 all include key B, and the aggregation definitions of aggregation tables 8 to 10 all include key C. Then the network device merges aggregation tables 1 to 3, aggregation tables 4 to 7, and aggregation tables 8 to 10, respectively, to obtain three merged aggregation tables.

[0124] In addition, the embodiment shown in Figure 4 is based on an example in which different aggregation tables have only one key. In actual applications, aggregation tables can also include a larger number of keys, which is not limited here. For example, assuming that the aggregation definitions of aggregation tables 1 to 3 all include keys A and B, then the keys in the aggregation definition of the merged aggregation table obtained by merging aggregation tables 1 and 2 will still be keys A and B.

[0125] In this application, for multiple first aggregate tables corresponding to the same key in M ​​aggregate tables, multiple first aggregate tables can be directly determined as aggregate tables to be merged, which is simple and convenient to operate, and improves the practicality of the technical solution of this application. At the same time, when merging aggregate tables corresponding to the same key, the same key will be deduplicated, reducing the number of stored copies of the same key, further reducing the waste of storage resources, and saving users' usage costs.

[0126] In some optional implementations, the N aggregate tables to be merged include at least one group of second aggregate tables, each group of second aggregate tables including multiple second aggregate tables corresponding to different keys. In this solution, merging the N aggregate tables to be merged includes: merging the keys and values ​​corresponding to each group of second aggregate tables in the at least one group of second aggregate tables. It will be understood that merging the group of second aggregate tables can produce a merged aggregate table.

[0127] For example, assume that the N aggregation tables to be merged include two groups of second aggregation tables: a first group of second aggregation tables including aggregation table 1 and aggregation table 2, and a second group of second aggregation tables including aggregation tables 3 through 5. The network device merges aggregation table 1 with aggregation table 2 to obtain a merged aggregation table; and merges aggregation tables 3 through 5 to obtain a merged aggregation table.

[0128] For example, suppose a network device determines that it needs to merge two aggregation tables corresponding to different keys. Aggregation table 1 has an aggregation definition of "key-value pair: a -- sum(b), min(text)"; and aggregation table 2 has an aggregation definition of "key-value pair: c -- sum(b), avg(age)." Both aggregation definitions include the same value, sum(b). The network device deduplicates this value, resulting in an aggregation definition for the merged aggregation table of "key-value pair: a, c -- sum(b), min(text), avg(age)."

[0129] Exemplarily, different aggregation tables with different keys and different values ​​may be merged as long as the network device determines in step 302 that these different aggregation tables are included in the N aggregation tables to be merged.

[0130] In the present application, by comparing the estimated aggregation rate generated by merging different aggregation tables corresponding to different keys with a first threshold value, it is determined whether the aggregation tables corresponding to different keys can be merged, thereby balancing computing power resources and usage costs. In the scheme that can be merged, if there are identical values ​​in the aggregation tables of different keys, deduplication is performed during the merger, which further reduces the occupancy of storage resources and saves storage resources. In general, after the network device obtains M aggregation tables corresponding to the same base table, it determines N aggregation tables to be merged from the M aggregation tables according to the aggregation definition of the aggregation table, and merges these N aggregation tables to obtain a merged aggregation table. The amount of data in the merged aggregation table is less than the sum of the data amounts of the N aggregation tables to be merged, that is, the data processing method provided by the present application reduces the number of times the same data in different aggregation tables is stored, thereby reducing the amount of data in the aggregation table and reducing the occupancy of storage resources.

[0131] In some optional implementations, after merging the N aggregation tables to be merged, the network device may further determine whether to change the data distribution method. In summary, the network device may obtain the aggregation ratios of all aggregation tables corresponding to the current moment, determine a second target aggregation ratio greater than a second threshold from the aggregation ratios of all aggregation tables, and broadcast the data included in the aggregation table corresponding to the second target aggregation ratio.

[0132] Among them, all the aggregation tables corresponding to the current moment have multiple possible situations: including the original aggregation table that has not been merged and the merged aggregation table, or including the merged aggregation table, or including the original aggregation table that has not been merged. The following uses the aforementioned M as 5, that is, the network device obtains 5 original aggregation tables, and takes aggregation tables 1 to 5 as an example to illustrate the possible situations respectively:

[0133] For example, if during data processing, the network device merges aggregation tables 1 to 3 to obtain a merged aggregation table 6, and merges aggregation tables 4 and 5 to obtain a merged aggregation table 7, then after the merger, the network device obtains two aggregation tables, namely, aggregation table 6 and aggregation table 7.

[0134] For example, if during data processing, the network device merges Aggregation Table 1 and Aggregation Table 2 to obtain a merged Aggregation Table 8, and determines that Aggregation Tables 3 to 5 cannot be merged, then the network device ultimately obtains four aggregation tables: Aggregation Table 8, Aggregation Table 3, Aggregation Table 4, and Aggregation Table 5.

[0135] For example, if during the data processing process, the network device determines that all aggregation tables cannot be merged, then the aggregation tables finally obtained by the network device are the original five aggregation tables, namely, aggregation table 1 to aggregation table 5.

[0136] The network device broadcasts the data included in the aggregation table corresponding to the second target aggregation rate. Specifically, the data in the aggregation table is distributed in the form of replicas. If the data in the original aggregation table is hash distributed or Rosin-Römmler distributed (RR distribution), this application changes the data distribution form of the aggregation table.

[0137] In general, data broadcasting can be achieved by adding a broadcast operator after the aggregation (agg) operation to distribute the aggregate table data to every node in the system, so that every node has the full data. If the network device plays the role of a CN in the system, then the network device will broadcast the aggregate table data to all DNs that communicate with the CN; if the network device plays the role of a DN in the system, then the network device will broadcast the aggregate table data to all DNs that correspond to the same CN as the DN, as well as the CN corresponding to the DN.

[0138] The following describes the replica distribution with reference to a schematic diagram. Please refer to FIG5 , which is a schematic diagram provided in an embodiment of the present application.

[0139] In the embodiment shown in FIG5 , if the aggregation rates of the aggregation tables stored on DN1 and DN2 are both greater than the second threshold, the data in the aggregation tables stored on each DN will be broadcast. In the "after broadcast" scenario in FIG5 , DN1 and DN2 execute the broadcast operator after the aggregation operation: for DN1, data A stored in itself is broadcast to DN2, where data A refers to the data in the aggregation table stored in DN1 with an aggregation rate greater than the second threshold; for DN1, data B stored in itself is broadcast to DN1, where data B refers to the data in the aggregation table stored in DN2 with an aggregation rate greater than the second threshold.

[0140] Assume that in the "before broadcast" scenario shown in Figure 5, the aggregation tables included in DN1 and DN2 are shown in Table 5 and Table 6 respectively:

[0141] Table 5

[0142] Table 6

[0143] Without data broadcasting, each DN only has a partial data set. Query statements (select statements) require scanning across all DNs and then summarizing data on the CN. With data broadcasting, the newly generated data (30, 100) on DN1 is broadcast to DN2; the newly generated data (40, 200) on DN2 is broadcast to DN1. After the broadcast, both DN1 and DN2 have the full data set.

[0144] After the broadcast, the aggregation tables included in DN1 and DN2 are shown in Table 7 below:

[0145] Table 7

[0146] After broadcasting, each node in the system stores the full amount of aggregate table data. When querying data, you can select any node for data query. Optionally, you can select a node with a smaller load for data query, or the CN can specify a node with a smaller load to complete the data query. The load mentioned here can be understood as the idle resources of the node, which can be determined by the node's central processing unit (CPU) resources, memory resources, number of tasks, etc. For example, the more idle resources the CPU has, or the more idle memory resources, or the fewer tasks, the smaller the load on the node.

[0147] In addition, it should be noted that the value of the second threshold can be set according to the needs of the actual application and is not limited here. Optionally, the second threshold can be greater than the first threshold, or the second threshold can also be less than or equal to the first threshold, which is not specifically limited here.

[0148] In this application, data in the aggregation table with an aggregation rate greater than the second threshold can be broadcast, so that when querying data, users can query at any node without the need for data transmission between nodes, reducing the use of network resources and improving the reliability and convenience of the technical solution. In addition, if the query is directly performed on a node with less load, the efficiency of data query can be improved.

[0149] In some optional implementations, the network device can execute the aggregation calculations for all aggregation tables corresponding to the base table in parallel. Specifically, the execution plan in the aggregation layer can be modified to multi-threaded parallel execution, so that when a base table corresponds to multiple aggregation tables, the aggregation calculations for these multiple aggregation tables can be executed in parallel.

[0150] The following further describes the process with reference to a schematic diagram. Please refer to FIG6 , which is a schematic diagram provided in an embodiment of the present application.

[0151] As shown in Figure 6, assume that the base table corresponds to two aggregation tables, where the aggregation definition of aggregation table 1 is "key-value pair: c--sum(b), min(text)"; the aggregation definition of aggregation table 2 is "key-value pair: a--sum(b), avg(age)".

[0152] If the traditional approach, as shown in the "Before" section of Figure 6, is used, the aggregation calculations for these two aggregate tables are performed serially. Therefore, the aggregation calculation for aggregate table 2 can only be performed after the aggregation calculation for aggregate table 1 is completed, resulting in slow aggregation calculations for the aggregate tables. If there are many aggregate tables, this slow aggregation calculation will lead to a performance bottleneck.

[0153] In the implementation of this application, the parallel method shown in "After" in Figure 6 is used to set the number of parallel executions, and multiple aggregation tables are calculated in parallel in the aggregation calculation layer. For example, parallelism can be achieved through thread pools or multithreading.

[0154] In this application, aggregation calculations are performed in parallel, which reduces the impact on the data import process and thus reduces the data processing delay.

[0155] In general, the data processing method introduced above can be summarized as the process shown in Figure 7. Please refer to Figure 7, which is a flow chart of the data processing method provided in an embodiment of the present application.

[0156] As shown in FIG7 , the data processing method includes the following steps:

[0157] The network device obtains metadata of the aggregate table, which can indicate whether the obtained table is an aggregate table. The network device performs step ①. Determine whether there is more than one "aggregate table" in the base table of the imported data. If there is no "aggregate table" or there is only one "aggregate table", the execution plan is generated normally.

[0158] ②. If multiple aggregate tables exist, when merging them, determine whether they have the same aggregate key. If so, merge them and then generate an aggregate operator based on the keys and values ​​of the new merged aggregate table. The aggregate operator indicates how to merge aggregate tables.

[0159] ③. If there is no identical aggregation key, the aggregation rate is evaluated after the merge. If the aggregation rate is greater than the set merge threshold (merge_threshold), the two "aggregation tables" can be merged, and then the aggregation calculation operator is generated based on the keys and values ​​of the new "aggregation table" after the merge.

[0160] ④. If the estimated aggregation rate is less than the set merge threshold, the merge is not possible and the execution plan is generated normally. The merge threshold is the first threshold mentioned above.

[0161] ⑤. If there are multiple aggregations, obtain the number of aggregation tables and set the corresponding degree of parallelism (set dop to n). When the operator is executed, multiple threads are started and executed in parallel. Here, n is an integer greater than or equal to 2, indicating that there are n threads / processes.

[0162] ⑥ Before generating the aggregation operator, the aggregation rate is evaluated. If the aggregation rate exceeds the set aggregation threshold (agg_threshold), a broadcast operator is generated to broadcast the data to other computing nodes, effectively resetting the data distribution to a replica distribution. The aggregation threshold is the second threshold mentioned above.

[0163] The relevant steps of the embodiment shown in FIG. 7 have been described in detail in the embodiments shown in FIG. 3 to FIG. 5 above, so they will not be repeated here.

[0164] Below, the relevant equipment provided in the embodiments of the present application is described.

[0165] Please refer to Figure 8 , which is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application. As shown in Figure 8 , the data processing device 800 includes an acquisition unit 801 and a processing unit 802 .

[0166] In some optional embodiments, the acquisition unit 801 is used to obtain a base table and M aggregation tables, where the base table includes data actually stored in the data processing device, and the M aggregation tables are obtained by processing the data included in the base table according to M aggregation definitions. The M aggregation tables correspond one-to-one to the M aggregation definitions, and the M aggregation definitions are different. The aggregation definitions include key-value pairs, and the aggregation definitions are used to indicate the method of processing the data included in the base table. M is an integer greater than or equal to 2.

[0167] Processing unit 802 is configured to determine N aggregate tables to be merged from the M aggregate tables based on the keys included in the M aggregate definitions, where N is an integer greater than or equal to 2 and less than or equal to M. The N aggregate tables to be merged are merged to obtain a merged aggregate table, where the amount of data in the merged aggregate table is less than the sum of the amounts of data in the N aggregate tables to be merged.

[0168] In some optional embodiments, the processing unit 802 is specifically used to: if there are multiple first aggregation definitions including the same key in the M aggregation definitions, then the data processing device determines from the M aggregation tables that the N aggregation tables to be merged include multiple first aggregation tables corresponding to the multiple first aggregation definitions; the data processing device merges the N aggregation tables to be merged, including: merging the keys and values ​​corresponding to the multiple first aggregation tables.

[0169] In some optional implementations, the processing unit 802 is specifically configured to: if the M aggregation definitions include X second aggregation definitions corresponding to different keys, then the data processing device estimates at least one aggregation ratio generated by merging any two to X second aggregation tables corresponding to the X second aggregation definitions, where X is an integer greater than or equal to 2 and less than or equal to M;

[0170] The data processing device determines that the N aggregation tables to be merged include at least one set of second aggregation tables corresponding to at least one first target aggregation ratio;

[0171] The first target aggregation rate is an aggregation rate greater than a first threshold among at least one aggregation rate, each group of second aggregation tables in at least one group of second aggregation tables includes multiple second aggregation tables, and the multiple second aggregation tables included in each group of second aggregation tables are different from other aggregation tables in the X second aggregation tables.

[0172] In some optional embodiments, the processing unit 802 is specifically used to: if there are multiple aggregation rates greater than a first threshold and corresponding to the same second aggregation table in at least one aggregation rate, then determine that the highest of the multiple aggregation rates is the first target aggregation rate; or, if there are multiple aggregation rates greater than the first threshold and corresponding to the same second aggregation table in at least one aggregation rate, then determine that the aggregation rate with the largest number of corresponding second aggregation tables in the multiple aggregation rates is the first target aggregation rate.

[0173] In some optional implementations, the processing unit 802 is specifically configured to: merge the keys and values ​​corresponding to each group of second aggregate tables in at least one group of second aggregate tables respectively.

[0174] In some optional implementations, the acquiring unit 801 is further configured to acquire the aggregation rates of all aggregation tables corresponding to the base table.

[0175] The processing unit 802 is further configured to: determine a second target aggregation ratio greater than a second threshold from the respective aggregation ratios of all aggregation tables; and broadcast data included in the aggregation table corresponding to the second target aggregation ratio.

[0176] In some optional implementations, the processing unit 802 is further configured to execute aggregation calculations of all aggregation tables corresponding to the base table in parallel.

[0177] The acquisition unit 801 and the processing unit 802 can be implemented by software or hardware. For example, the implementation of the processing unit 802 will be described below using the processing unit 802 as an example. Similarly, the implementation of the acquisition unit 801 can refer to the implementation of the processing unit 802.

[0178] As an example of a software functional unit, the processing unit 802 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the processing unit 802 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0179] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0180] As an example of a hardware functional unit, the processing unit 802 may include at least one computing device, such as a server. Alternatively, the processing unit 802 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0181] The multiple computing devices included in processing unit 802 can be distributed in the same region or in different regions. The multiple computing devices included in processing unit 802 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in processing unit 802 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0182] It should be noted that the acquisition unit 801 and the processing unit 802 respectively implement different steps in the data processing method to achieve the full functionality of the data processing device 800. The data processing device 800 is used to perform the operations performed by the network device in the embodiments shown in Figures 2 to 7 above to implement the data processing method provided in the embodiments of the present application, and will not be described in detail here.

[0183] Please refer to Figure 9, which is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. The computing device 900 includes a processor 901, a communication interface 902, a bus 903, and a memory 904. The processor 901, the communication interface 902, and the memory 904 communicate via the bus 903. In practical applications, communication can also be achieved through other means such as wireless transmission, which is not limited here.

[0184] The computing device 900 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 900.

[0185] The processor 901 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0186] The communication interface 902 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or a communication network.

[0187] Bus 903 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, for example. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG9 shows a single line, but this does not imply a single bus or type of bus. Bus 903 may include a path for transmitting information between various components of computing device 900 (e.g., memory 904, processor 901, and communication interface 902).

[0188] The memory 904 may include volatile memory, such as random access memory (RAM). The memory 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0189] The memory 904 stores executable program codes, and the processor 901 executes the executable program codes to respectively implement the functions of the aforementioned acquisition unit 801 and processing unit 802, thereby implementing the data processing method. That is, the memory 904 stores instructions for executing the data processing method.

[0190] Alternatively, the memory 904 stores executable code, and the processor 901 executes the executable code to respectively implement the functions of the aforementioned acquisition unit 801 and processing unit 802, thereby implementing the data processing method. That is, the memory 904 stores instructions for executing the data processing method.

[0191] The present application also provides a computing device cluster comprising at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some optional implementations, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0192] Please refer to Figures 10 and 11, which are both structural diagrams of a computing device cluster provided in embodiments of the present application.

[0193] As shown in Figure 10, the computing device cluster includes at least one computing device 900. The memory 904 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing the data processing method provided in the embodiment of the present application.

[0194] In some possible implementations, the memory 904 of one or more computing devices 900 in the computing device cluster may also store partial instructions for executing the data processing method. In other words, the combination of one or more computing devices 904 can jointly execute the instructions for executing the data processing method.

[0195] It should be noted that the memory 904 in different computing devices 900 in the computing device cluster can store different instructions, each for executing a portion of the functions of the data processing apparatus. In other words, the instructions stored in the memory 904 in different computing devices 900 can implement the functions of one or more of the acquisition unit 801 and the processing unit 802.

[0196] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network (WAN) or a local area network (LAN), among others. FIG. 11 illustrates one possible implementation. As shown in FIG. 11 , two computing devices 900A and 900B are connected via a network. Specifically, each computing device is connected to the network via a communication interface within the computing device. In this type of possible implementation, the memory 904 in computing device 900A stores instructions for executing the functions of acquisition unit 801. Simultaneously, the memory 904 in computing device 900B stores instructions for executing the functions of processing unit 802.

[0197] The connection method between the computing device clusters shown in Figure 11 can be based on the consideration that in the data processing method provided in this application, processing operations and operations other than processing operations are performed separately, that is, the function of the acquisition unit 801 is considered to be executed by the computing device 900A, and the function of the processing unit 802 is considered to be executed by the computing device 900B.

[0198] It should be understood that the functionality of the computing device 900A shown in FIG11 may also be implemented by multiple computing devices 900. Similarly, the functionality of the computing device 900B may also be implemented by multiple computing devices 900.

[0199] The present application also provides another computing device cluster, wherein the connection relationship between the computing devices in the computing device cluster can be similar to the connection relationship between the computing device clusters described in FIG10 and FIG11 , and will not be repeated here.

[0200] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored on any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the aforementioned data processing method.

[0201] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned data processing method.

[0202] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0203] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: include: The network device obtains a base table and M aggregation tables, wherein the base table includes data actually stored by the network device, the M aggregation tables are obtained by processing the data included in the base table according to M aggregation definitions, the M aggregation tables correspond to the M aggregation definitions one by one, and the M aggregation definitions are different, the aggregation definitions include key-value pairs, and the aggregation definitions are used to indicate a method for processing the data included in the base table, and M is an integer greater than or equal to 2; The network device determines, according to the keys included in the M aggregation definitions, N aggregation tables to be merged from the M aggregation tables, where N is an integer greater than or equal to 2 and less than or equal to M; The network device merges the N aggregation tables to be merged to obtain a merged aggregation table, and the data volume of the merged aggregation table is less than the sum of the data volumes of the N aggregation tables to be merged.

2. The method according to claim 1, characterized in that The network device determines, according to the keys included in the M aggregation definitions, N aggregation tables to be merged from the M aggregation tables, including: If there are multiple first aggregation definitions including the same key in the M aggregation definitions, the network device determines, from the M aggregation tables, that the N aggregation tables to be merged include multiple first aggregation tables corresponding to the multiple first aggregation definitions; The network device merges the N aggregation tables to be merged, including: merging keys and values ​​corresponding to the multiple first aggregation tables.

3. The method according to claim 1 or 2, characterized in that: The network device determines, according to the keys included in the M aggregation definitions, N aggregation tables to be merged from the M aggregation tables, including: If the M aggregation definitions include X second aggregation definitions with different keys, the network device estimates at least one aggregation rate generated by merging any 2 to X second aggregation tables corresponding to the X second aggregation definitions, where X is an integer greater than or equal to 2 and less than or equal to M; The network device determines that the N aggregation tables to be merged include at least one group of second aggregation tables corresponding to at least one first target aggregation rate; The first target aggregation rate is an aggregation rate among the at least one aggregation rate that is greater than a first threshold value, each group of second aggregation tables in the at least one group of second aggregation tables includes multiple second aggregation tables, and the multiple second aggregation tables included in each group of second aggregation tables are different from other aggregation tables in the X second aggregation tables.

4. The method according to claim 3, characterized in that If, among the at least one aggregation rate, there are multiple aggregation rates that are greater than the first threshold and correspond to the same second aggregation table, the network device determines the highest of the multiple aggregation rates as the first target aggregation rate; or, If there are multiple aggregation rates among the at least one aggregation rate that are greater than the first threshold and correspond to the same second aggregation table, the network device determines that the first target aggregation rate is the one with the largest number of corresponding second aggregation tables among the multiple aggregation rates.

5. The method according to claim 3 or 4, characterized in that: The merging of the N aggregation tables to be merged includes: The network device respectively merges the keys and values ​​corresponding to each group of second aggregation tables in the at least one group of second aggregation tables.

6. The method according to any one of claims 1 to 5, characterized in that After merging the N aggregation tables to be merged, the method further includes: The network device obtains the respective aggregation rates of all the aggregation tables corresponding to the base table; The network device determines a second target aggregation rate greater than a second threshold from the respective aggregation rates of all the aggregation tables; The network device broadcasts the data included in the aggregation table corresponding to the second target aggregation rate.

7. A data processing device, characterized in that: include: an acquisition unit, configured to acquire a base table and M aggregation tables, wherein the base table includes data actually stored by the data processing device, the M aggregation tables are obtained by processing the data included in the base table according to M aggregation definitions, the M aggregation tables correspond to the M aggregation definitions one by one, and the M aggregation definitions are different, the aggregation definitions include key-value pairs, and the aggregation definitions are used to indicate a method for processing the data included in the base table, and M is an integer greater than or equal to 2; a processing unit, configured to determine N aggregation tables to be merged from the M aggregation tables according to the keys included in the M aggregation definitions, where N is an integer greater than or equal to 2 and less than or equal to M; The processing unit is further used to merge the N aggregation tables to be merged to obtain a merged aggregation table. The data volume of the aggregate table is smaller than the sum of the data volumes of the N aggregate tables to be merged.

8. The data processing device according to claim 7, characterized in that: The processing unit is specifically used for: If there are multiple first aggregation definitions including the same key in the M aggregation definitions, the data processing device determines from the M aggregation tables that the N aggregation tables to be merged include multiple first aggregation tables corresponding to the multiple first aggregation definitions; The data processing device merges the N aggregation tables to be merged, including: merging keys and values ​​corresponding to the multiple first aggregation tables.

9. The data processing device according to claim 7 or 8, characterized in that: The processing unit is specifically used for: If the M aggregation definitions include X second aggregation definitions with different corresponding keys, the data processing device estimates at least one aggregation rate generated by merging any 2 to X second aggregation tables corresponding to the X second aggregation definitions, where X is an integer greater than or equal to 2 and less than or equal to M; The data processing device determines that the N aggregation tables to be merged include at least one group of second aggregation tables corresponding to at least one first target aggregation rate; The first target aggregation rate is an aggregation rate among the at least one aggregation rate that is greater than a first threshold value, each group of second aggregation tables in the at least one group of second aggregation tables includes multiple second aggregation tables, and the multiple second aggregation tables included in each group of second aggregation tables are different from other aggregation tables in the X second aggregation tables.

10. The data processing device according to claim 9, characterized in that: The processing unit is specifically used for: If, among the at least one aggregation rate, there are multiple aggregation rates that are greater than the first threshold and correspond to the same second aggregation table, then determining the highest of the multiple aggregation rates as the first target aggregation rate; or, If there are multiple aggregation rates among the at least one aggregation rate that are greater than the first threshold and correspond to the same second aggregation table, it is determined that the first target aggregation rate is the one with the largest number of corresponding second aggregation tables among the multiple aggregation rates.

11. The data processing device according to claim 9 or 10, characterized in that: The processing unit is specifically used for: The keys and values ​​corresponding to each group of second aggregate tables in the at least one group of second aggregate tables are merged respectively.

12. The data processing device according to any one of claims 7 to 11, characterized in that: The acquisition unit is further used to acquire the respective aggregation rates of all the aggregation tables corresponding to the base table; The processing unit is further used for: Determine a second target aggregation rate greater than a second threshold from the respective aggregation rates of all the aggregation tables; The data included in the aggregation table corresponding to the second target aggregation rate is broadcast.

13. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 6.

14. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 6.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer program instructions, which, when executed by a computing device cluster, cause the computing device cluster to perform the method according to any one of claims 1 to 6.

Images