Data processing method, device, equipment and storage medium

Abstract

The present invention provides a data processing method, device, equipment and storage medium, which can be applied to the field of database technology. The method includes: identifying a data deduplication task related to a business general table to obtain an identification result; when the identification result includes a deduplication operator, converting the data deduplication task to generate a data query task that does not include the deduplication operator; determining a balanced partitioning strategy in response to the received data query task; based on the balanced partitioning strategy, partitioning the data in the business general table to obtain multiple business sub-tables, wherein the data volume of each of the multiple business sub-tables is the same, and the multiple business sub-tables are evenly distributed on multiple nodes in a cluster server; executing the data query task for the multiple business sub-tables to obtain a target query result.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] In cluster databases, traditional parallel grouping and aggregation operations rely on data sampling to determine the data segmentation method. For structured query statements containing the deduplication operator (DISTINCT), the system currently only supports data partitioning strategies based on hash algorithms. When the data values ​​between different groups in the business table are very different and the fields bound to the deduplication operator are relatively unique, the data partitioning strategy based on the hash algorithm will lead to uneven distribution of data among cluster nodes. Since data may be concentrated on a few nodes while other nodes are not fully utilized, the increase in the computing load of a few nodes will cause the resources of other nodes to be idle, affecting the parallel processing capability of cluster server queries and the overall throughput of the system. Summary of the invention

[0003] In view of the above problems, the present invention provides a data processing method, apparatus, device and storage medium.

[0004] According to a first aspect of the present invention, there is provided a data processing method, comprising:

[0005] Identify data deduplication tasks related to the business master table and obtain identification results; when the identification results include a deduplication operator, convert the data deduplication task to generate a data query task that does not include a deduplication operator, wherein the deduplication operator in the data deduplication task is bound with reference field information for performing deduplication operations on the reference field information, the grouping operator in the data deduplication task is bound with benchmark field information for performing grouping operations on the benchmark field information, and the grouping operator in the data query task is bound with benchmark field information and reference field information for performing grouping operations on the benchmark field information and reference field information; determine a balanced partitioning strategy in response to the received data query task; based on the balanced partitioning strategy, partition the data in the business master table to obtain multiple business sub-tables, wherein the data volume of each of the multiple business sub-tables is the same, and the multiple business sub-tables are evenly distributed on multiple nodes in the cluster server; execute data query tasks for the multiple business sub-tables to obtain target query results.

[0006] Optionally, the data deduplication task also includes an aggregation operator bound to the deduplication result corresponding to the reference field information and an average operator bound to the reference field information, the aggregation operator is used to perform an aggregation operation on the deduplication result of the reference field information, and the average operator is used to perform an average operation on the reference field information. The data query task includes a data query subtask, a first data aggregation subtask, a second data aggregation subtask, and a data average subtask; wherein, for multiple business subtables, executing the data query task to obtain a target query result includes:

[0007] For multiple business subtables, execute the data query subtask to obtain the initial query results corresponding to each of the multiple business subtables; execute the data first aggregation subtask to aggregate the multiple initial query results to obtain the intermediate aggregation result. Execute the data second aggregation subtask to aggregate the intermediate aggregation results to obtain the target aggregation result, wherein the target aggregation result includes the aggregation result of the deduplication result corresponding to the reference field information; execute the data average subtask to average the target aggregation result to obtain the target query result, wherein the target query result includes the average result of the reference field information.

[0008] Optionally, execute the first data aggregation subtask to aggregate multiple query results to obtain intermediate aggregation results, including:

[0009] Execute the grouping operator bound to the benchmark field information and the reference field information, the aggregation operator bound to the reference field information, and the counting operator bound to the reference field information in the first data aggregation subtask, aggregate multiple query results, and obtain an intermediate aggregation result, wherein the intermediate aggregation result includes the benchmark field grouping sub-result, the reference field grouping sub-result, the reference field grouping aggregation sub-result, and the reference field grouping counting sub-result, and the reference field grouping sub-result represents the deduplication result of the reference field information.

[0010] Optionally, executing the second data aggregation subtask to aggregate the intermediate aggregation results to obtain the target aggregation result includes:

[0011] Execute the grouping operator bound to the reference field information in the second data aggregation subtask, group the intermediate aggregation results, and obtain the second grouping result corresponding to the reference field information; execute the aggregation operator in the second data aggregation subtask, perform aggregation operation on the second grouping result, and obtain the target aggregation result, wherein the target aggregation result includes the aggregation subresult of the reference field grouping subresult, the aggregation subresult of the reference field grouping aggregation subresult, and the aggregation subresult of the reference field grouping counting subresult, and the aggregation subresult of the reference field grouping subresult represents the aggregation result of the deduplication result corresponding to the reference field information.

[0012] Optionally, execute the data averaging subtask to average the target aggregation results to obtain the target query results, including:

[0013] Execute the data averaging subtask, perform an averaging operation on the aggregate subresults of the reference field grouping aggregate subresults and the aggregate subresults of the reference field grouping counting subresults in the target aggregate result, and obtain the reference field average subresult, wherein the reference field average subresult represents the average result of the reference field information; obtain the target query result based on the reference field average subresult and the aggregate subresult of the reference field grouping subresult.

[0014] Optionally, based on a balanced partitioning strategy, the data in the business master table is partitioned to obtain multiple business sub-tables, including:

[0015] Based on the balanced partitioning strategy, the number of partitions and the amount of data in the business table are determined; according to the number of partitions and the amount of data in the business table, the data in the business table is divided to determine the data of each of the multiple business sub-tables.

[0016] Optionally, the balanced partitioning strategy also includes at least one of the following: a round-robin partitioning strategy and a random partitioning strategy.

[0017] A second aspect of the present invention provides a data processing device, comprising:

[0018] The recognition module is used to identify the data deduplication tasks related to the business summary table and obtain the recognition results;

[0019] A conversion module, used for converting the data deduplication task to generate a data query task that does not contain the deduplication operator when the recognition result contains the deduplication operator, wherein the deduplication operator in the data deduplication task is bound with reference field information for performing a deduplication operation on the reference field information, the grouping operator in the data deduplication task is bound with reference field information for performing a grouping operation on the reference field information, and the grouping operator in the data query task is bound with reference field information and reference field information for performing a grouping operation on the reference field information and the reference field information;

[0020] A response module, used to determine a balanced partitioning strategy in response to a received data query task;

[0021] A partitioning module is used to partition the data in the business master table based on a balanced partitioning strategy to obtain multiple business sub-tables, wherein the data volume of each of the multiple business sub-tables is the same, and the multiple business sub-tables are evenly distributed on multiple nodes in the cluster server;

[0022] The execution module is used to execute data query tasks for multiple business sub-tables to obtain target query results.

[0023] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned data processing method.

[0024] The fourth aspect of the present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned data processing method.

[0025] According to the data processing method, device, equipment and storage medium provided by the present invention, a data deduplication task including a deduplication operator is converted to generate a data query task that does not include a deduplication operator, and a balanced partitioning strategy is determined, thereby obtaining multiple business sub-tables; the data query task is executed to obtain the target query result. Since the deduplication operator bound to the reference field information is flexibly replaced by the grouping operator bound to the benchmark field information and the reference field information, it is ensured that the data query task does not include the deduplication operator, and a balanced partitioning strategy for data segmentation in the business master table is determined, so that while obtaining the target query result that is the same as the result of executing the data deduplication task, the data distribution skew problem generated when the deduplication operator is used to deduplicate the business master table with large data volume differences in different groups is solved, and the computing resources of multiple nodes in the cluster server are fully utilized, thereby improving the parallel processing capability of the cluster server and the overall throughput of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0027] Figure 1 An application scenario of the data processing method according to an embodiment of the present invention is shown;

[0028] Figure 2 A flow chart of a data processing method according to an embodiment of the present invention is shown;

[0029] Figure 3 A flow chart of generating a target query result according to an embodiment of the present invention is shown;

[0030] Figure 4 A structural block diagram of a data processing device according to an embodiment of the present invention is shown;

[0031] Figure 5 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0032] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0033] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0034] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0035] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0036] In the process of implementing the present invention, it was found that for structured query statements containing a deduplication operator (DISTINCT), the system currently only supports a data partitioning strategy based on a hash algorithm. When the data value gap between different groups in the business table is large and the field bound to the deduplication operator is relatively unique, the data partitioning strategy based on the hash algorithm will cause uneven distribution of data among cluster nodes. Therefore, there is an urgent need for a data processing method that can achieve data deduplication without executing a deduplication operator and has uniform data partitioning.

[0037] In view of this, an embodiment of the present invention provides a data processing method, device, equipment and storage medium. The method includes: identifying a data deduplication task related to a business general table to obtain an identification result; when the identification result includes a deduplication operator, converting the data deduplication task to generate a data query task that does not include a deduplication operator, wherein the deduplication operator in the data deduplication task is bound with reference field information for performing a deduplication operation on the reference field information, the grouping operator in the data deduplication task is bound with reference field information for performing a grouping operation on the reference field information, and the grouping operator in the data query task is bound with reference field information and reference field information for performing a grouping operation on the reference field information and the reference field information; in response to the received data query task, determining a balanced partitioning strategy; based on the balanced partitioning strategy, partitioning the data in the business general table to obtain multiple business sub-tables, wherein the data volume of each of the multiple business sub-tables is the same, and the multiple business sub-tables are evenly distributed on multiple nodes in the cluster server; executing the data query task for the multiple business sub-tables to obtain the target query result.

[0038] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0039] Figure 1 The application scenario of the data processing method according to the embodiment of the present invention is shown.

[0040] like Figure 1 As shown, the business system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0041] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0042] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0043] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0044] It should be noted that the data processing method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present invention can generally be set in the server 105. The data processing method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data processing device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0045] Alternatively, the data processing method provided in the embodiment of the present invention may also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may also be executed by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the data processing apparatus provided in the embodiment of the present invention may also be arranged in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may be arranged in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0046] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0047] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0048] Figure 2 A flow chart of a data processing method according to an embodiment of the present invention is shown.

[0049] like Figure 2 As shown, the method 200 includes operations S210 to S250.

[0050] In operation S210, a data deduplication task related to the business summary table is identified to obtain an identification result.

[0051] Optionally, the cluster database is deployed on a server, and the business summary table represents a data table related to the system business stored in the cluster database.

[0052] Optionally, the data deduplication task represents the task of deduplicating the data groups obtained from the business summary table. The data deduplication task can be a structured query language (SQL). In response to the received data deduplication task for the business summary table, the server identifies the operator in the data deduplication task and obtains an identification result. The identification result can be the result of the data deduplication task including the deduplication operator and the result of not including the deduplication operator.

[0053] In operation S220, when the recognition result includes a deduplication operator, the data deduplication task is converted to generate a data query task that does not include a deduplication operator.

[0054] Optionally, the deduplication operator in the data deduplication task is bound with reference field information for performing deduplication operations on the reference field information, and the grouping operator in the data deduplication task is bound with reference field information for performing grouping operations on the reference field information. The reference field information represents the field information to be deduplicated in the business master table, and the reference field information represents the field information to be grouped in the business master table.

[0055] Optionally, the deduplication operator can remove duplicate data in the business summary table according to the bound field information, and the grouping operator can group the data in the business summary table according to the bound field information and perform aggregation calculations on each group. For example, the data deduplication task containing the deduplication operator is: select distinct (j) from t group by (i), where distinct is the deduplication operator, j is the reference field information, group by is the grouping operator, i is the reference field information, and t is the business summary table.

[0056] Optionally, when the recognition result includes a deduplication operator, the data deduplication task is converted to generate a data query task that does not include a deduplication operator. For example, the deduplication operator distinct (j) bound to the reference field information is replaced with the grouping operator group by (j) bound to the reference field information, thereby obtaining a grouping operator bound to the base field information and the reference field information, and generating a data query task. For example, the data query task is: select i, j from t groupby (i, j).

[0057] Optionally, the grouping operator in the data query task is bound with the base field information and the reference field information, and is used to perform a grouping operation on the base field information and the reference field information.

[0058] In operation S230 , a balanced partitioning strategy is determined in response to the received data query task.

[0059] Optionally, in response to the data query task, the server identifies other operators in the data query task and evaluates and obtains a balanced partitioning strategy on the premise that the data query task does not include a deduplication operator.

[0060] Optionally, the balanced partitioning strategy represents a strategy for distributing data of the business summary table to cluster servers. When the capacities of multiple nodes in the cluster server remain unchanged, the data of the business summary table is evenly distributed to each node according to the balanced partitioning strategy.

[0061] In operation S240, based on the balanced partitioning strategy, the data in the business master table is partitioned to obtain a plurality of business sub-tables.

[0062] Optionally, the data volumes of the multiple business sub-tables are the same, and the multiple business sub-tables are evenly distributed on multiple nodes in the cluster server.

[0063] For example, the partition number n is reconfigured according to the balanced partitioning strategy, and the data in the business general table is evenly divided according to the partition number to obtain n business sub-tables, and the n business sub-tables are respectively distributed to n nodes in the cluster server.

[0064] In operation S250, a data query task is executed for multiple business sub-tables to obtain a target query result.

[0065] Optionally, multiple nodes perform data query tasks on their respective business sub-tables to obtain target query results, which may include: summarizing the queried data of multiple business sub-tables, and then grouping the summarized data according to the benchmark field information and the reference field information and performing aggregation calculations to obtain the target query results.

[0066] Optionally, the target query result obtained by executing the data query task includes the result obtained by executing the data deduplication task. The target query result can achieve the result of deduplication effect.

[0067] Optionally, by flexibly replacing the deduplication operator bound to the reference field information with the grouping operator bound to the benchmark field information and the reference field information, it is ensured that the data query task does not contain the deduplication operator, and a balanced partitioning strategy for data segmentation in the business master table is determined. This solves the problem of data distribution skew caused by using the deduplication operator to deduplicate the business master table with large differences in data volume in different groups while obtaining the target query result that is the same as the result of executing the data deduplication task. This fully utilizes the computing resources of multiple nodes in the cluster server, thereby improving the parallel processing capability of the cluster server and the overall throughput of the system.

[0068] Figure 3 A flow chart of generating a target query result according to an embodiment of the present invention is shown.

[0069] like Figure 3 As shown, for multiple business sub-tables, executing a data query task to obtain a target query result includes operations S310 to S340.

[0070] In operation S310, a data query subtask is executed for a plurality of business subtables to obtain initial query results corresponding to each of the plurality of business subtables.

[0071] In operation S320, a first data aggregation subtask is executed to aggregate multiple initial query results to obtain an intermediate aggregation result.

[0072] In operation S330, the second data aggregation subtask is executed to aggregate the intermediate aggregation results to obtain the target aggregation result.

[0073] In operation S340, the data averaging subtask is executed to average the target aggregation result to obtain the target query result.

[0074] Optionally, the data deduplication task also includes an aggregation operator bound to the deduplication results corresponding to the reference field information and an average operator bound to the reference field information. The aggregation operator is used to perform aggregation operations on the deduplication results of the reference field information, and the average operator is used to perform average operations on the reference field information.

[0075] For example, the data deduplication task is: select sum(distinct j), avg(j) from t group by (i), where sum is the aggregation operator, distinct j is the deduplication result corresponding to the reference field information, and avg is the average operator.

[0076] Optionally, while replacing the deduplication operator in the data deduplication task, the average operator bound to the reference field information in the data deduplication task is also replaced to obtain a data query task. The converted data query task includes a data query subtask, a data first aggregation subtask, a data second aggregation subtask, and a data average subtask. After executing the four subtasks, a target query result that is the same as the result of executing the data deduplication task is obtained. The target query result includes the aggregation result of the deduplication result corresponding to the reference field information and the average result corresponding to the reference field information.

[0077] Optionally, the data query subtask is used to query the data of the business subtable. The data of the business subtable includes all the base field information and all the reference field information in the business subtable.

[0078] For example, a cluster server has nodes 1, 2, and 3, and a corresponding business sub-table is distributed on each node. Node 1 responds to the data query subtask, performs data query on the business sub-table on node 1, and outputs the initial query result of node 1; node 2 responds to the data query subtask, performs data query on the business sub-table on node 2, and outputs the initial query result of node 2; node 3 responds to the data query subtask, performs data query on the business sub-table on node 3, and outputs the initial query result of node 3.

[0079] Optionally, the initial query result of the business subtable represents all base field information and all reference field information queried from the business subtable.

[0080] Optionally, multiple initial query results are aggregated to obtain an intermediate query result. For example, initial query results of node 1, node 2, and node 3 are aggregated to obtain an intermediate query result, which represents all base field information and all reference field information in the business summary table.

[0081] Table 1 schematically shows an intermediate query result table according to an embodiment of the present invention.

[0082]

[0083] Optionally, the data first aggregation subtask includes a grouping operator and an aggregation operator, and the intermediate query results are grouped according to the base field information and reference field information bound to the grouping operator, and then the first round of aggregation calculation is performed according to the field information bound to the aggregation operator to obtain the intermediate aggregation result. For example, the aggregation operator can be a summation operator. The intermediate aggregation result includes a deduplication result corresponding to the reference field information.

[0084] Optionally, the second data aggregation subtask includes a grouping operator and an aggregation operator, and the intermediate aggregation results are grouped according to the reference field information bound to the grouping operator, and then a second round of aggregation calculation is performed according to the field information bound to the aggregation operator to obtain the target aggregation result. The target aggregation result includes the aggregation result of the deduplication result corresponding to the reference field information.

[0085] Optionally, the data averaging subtask is used to perform an average operation on the target aggregation result to obtain a target query result, where the target query result includes an average result of the reference field information.

[0086] Optionally, execute the first data aggregation subtask to aggregate multiple query results to obtain intermediate aggregation results, including:

[0087] Execute the grouping operator bound to the benchmark field information and the reference field information, the aggregation operator bound to the reference field information, and the counting operator bound to the reference field information in the first data aggregation subtask, aggregate multiple query results, and obtain an intermediate aggregation result, wherein the intermediate aggregation result includes the benchmark field grouping sub-result, the reference field grouping sub-result, the reference field grouping aggregation sub-result, and the reference field grouping counting sub-result, and the reference field grouping sub-result represents the deduplication result of the reference field information.

[0088] Optionally, when performing the first round of aggregation operations on the intermediate query results, the grouping operator bound to the base field information and the reference field information is executed first, and then the aggregation operator bound to the reference field information and the counting operator bound to the reference field information are executed. For example, the data first aggregation subtask may be Select i as i_i, j as j_j, sum(j) assum_j, count(j) as cnt_j from t group by (i, j), where group by (i, j) is used to group the intermediate query results according to the base field information and the reference field information, and return the base field grouping sub-result i_i and the reference field grouping sub-result j_j.

[0089] Optionally, sum(j) and count(j) are respectively an aggregation operator and a counting operator bound with reference field information in the aggregation query subtask.

[0090] Optionally, an aggregation operator bound to the reference field information is executed to obtain the reference field grouping aggregation sub-result sum_j, and a counting operator bound to the reference field information is executed to obtain the reference field grouping counting sub-result count_j. The intermediate aggregation results include the reference field grouping sub-result i_i, the reference field grouping sub-result j_j, the reference field grouping aggregation sub-result sum_j, and the reference field grouping counting sub-result count_j.

[0091] Table 2 schematically shows an intermediate aggregation result table according to an embodiment of the present invention.

[0092]

[0093] Optionally, the intermediate aggregation results generated through the first round of aggregation calculations provide a more balanced data basis for the second round of aggregation calculations, which helps to improve query performance. The multi-stage aggregation process optimizes data distribution and processing flow.

[0094] Optionally, executing a second data aggregation subtask to aggregate the intermediate aggregation results to obtain a target aggregation result includes:

[0095] Execute the grouping operator bound to the reference field information in the second data aggregation subtask, group the intermediate aggregation results, and obtain the second grouping result corresponding to the reference field information; execute the aggregation operator in the second data aggregation subtask, perform aggregation operation on the second grouping result, and obtain the target aggregation result, wherein the target aggregation result includes the aggregation subresult of the reference field grouping subresult, the aggregation subresult of the reference field grouping aggregation subresult, and the aggregation subresult of the reference field grouping counting subresult, and the aggregation subresult of the reference field grouping subresult represents the aggregation result of the deduplication result corresponding to the reference field information.

[0096] Optionally, when performing a second round of aggregation calculation on the intermediate aggregation result, the grouping operator bound to the reference field information is executed first, and then the aggregation operator is executed. For example, the second data aggregation subtask can be: Select sum (j_j), sum (sum_j), sum (cnt_j) from t1 group by (i), where group by (i) is a grouping operator bound to the reference field information. The grouping operator is executed on the intermediate aggregation result to group the intermediate aggregation result according to the reference field information to obtain the second grouping result, and then the second round of aggregation calculation is performed on the second grouping result to obtain the target aggregation result. The target aggregation result includes the aggregation sub-result of the reference field grouping sub-result, the aggregation sub-result of the reference field grouping aggregation sub-result, and the aggregation sub-result of the reference field grouping counting sub-result.

[0097] Optionally, the second data aggregation subtask may include multiple aggregation operators, for example, executing the summation operator Sum(j_j) bound to the reference field grouping subresult j_j to obtain the aggregation subresult s_j_j of the reference field grouping subresult; executing the summation operator Sum(sum_j) bound to the reference field grouping aggregation subresult sum_j to obtain the aggregation subresult s_ sum _j of the reference field grouping aggregation subresult; executing the summation operator Sum(count_j) bound to the reference field grouping count subresult count_j to obtain the aggregation subresult s_coun_j of the reference field grouping count subresult.

[0098] Table 3 schematically shows a target aggregation result table according to an embodiment of the present invention.

[0099]

[0100] Optionally, execute the data averaging subtask to average the target aggregation results to obtain the target query results, including:

[0101] Execute the data averaging subtask, perform an averaging operation on the aggregate subresults of the reference field grouping aggregate subresults and the aggregate subresults of the reference field grouping counting subresults in the target aggregate result, and obtain the reference field average subresult, wherein the reference field average subresult represents the average result of the reference field information; obtain the target query result based on the reference field average subresult and the aggregate subresult of the reference field grouping subresult.

[0102] Optionally, the average operation may be a ratio between the aggregated sub-result s_ sum _j of the aggregated sub-result grouped by the reference field and the aggregated sub-result s_coun_j of the count sub-result grouped by the reference field, to obtain the reference field average sub-result a_j. For example, a_j may be the result of Sum(sum_j) / Sum(count_j). The target query result is obtained based on the reference field average sub-result a_j and the aggregated sub-result s_j_j of the reference field grouped sub-result.

[0103] Table 4 schematically shows a target query result table according to an embodiment of the present invention.

[0104]

[0105] Optionally, the aggregate sub-result s_j_j of the reference field grouping sub-result is the same as the aggregate result of the deduplication result corresponding to the reference field information obtained by sum(distinct j) in executing the data deduplication task, and the reference field average sub-result a_j is the same as the average result corresponding to the reference field information obtained by avg(j) in executing the data deduplication task. Therefore, the target query result is the same as the result of executing the data deduplication task.

[0106] Optionally, in the second round of aggregation calculations, the query execution plan is optimized by removing the deduplication operator and converting it into a form that is easier to process in parallel, thereby simplifying the aggregation operation and reducing the computational complexity; by converting the average operator in the data deduplication task into a combination of a summation operator and a counting operator, and appropriately adjusting these aggregation operators at different aggregation stages, data can be processed more efficiently.

[0107] Optionally, based on a balanced partitioning strategy, the data in the business master table is partitioned to obtain multiple business sub-tables, including:

[0108] Based on the balanced partitioning strategy, the number of partitions and the amount of data in the business table are determined; according to the number of partitions and the amount of data in the business table, the data in the business table is divided to determine the data of each of the multiple business sub-tables.

[0109] Optionally, the number of partitions n is reconfigured according to the balanced partitioning strategy, and the average data volume of the business sub-table is calculated according to the number of partitions and the data volume of the business total table.

[0110] Optionally, the data in the business general table is evenly divided according to the average data volume to obtain n business sub-tables, and the n business sub-tables are respectively distributed to n nodes in the cluster server.

[0111] Optionally, the balanced partitioning strategy also includes at least one of the following: a round-robin partitioning strategy and a random partitioning strategy.

[0112] Optionally, the Round-robin (rr) partitioning strategy is to construct a batch for each data in the business master table. Each batch contains only one data, and the data of each batch is written to each node of the cluster server in turn, thereby ensuring balanced data distribution.

[0113] Optionally, the random partitioning strategy (Randomness) requires first calculating the number of nodes in the cluster server, and then randomly and evenly scattering the data in the business summary table and distributing it to each node.

[0114] Optionally, the balanced partitioning strategy dynamically adjusts the distribution of data on each node according to the actual distribution of data, achieves load balancing, enhances the system's parallel processing capabilities, and improves overall query performance and throughput.

[0115] Based on the above data processing method, the present invention also provides a data processing device. Figure 4 The device is described in detail.

[0116] Figure 4 A structural block diagram of a data processing device according to an embodiment of the present invention is shown.

[0117] like Figure 4 As shown, the data processing device 400 of this embodiment includes an identification module 410 , a conversion module 420 , a response module 430 , a division module 440 and an execution module 450 .

[0118] The identification module 410 is used to determine a balanced partitioning strategy in response to the received data query task. In one embodiment, the identification module 410 can be used to perform the operation S210 described above, which will not be described in detail here.

[0119] The conversion module 420 is used to divide the data in the business master table based on the balanced partitioning strategy to obtain multiple business sub-tables, wherein the data volume of each of the multiple business sub-tables is the same, and the multiple business sub-tables are evenly distributed on multiple nodes in the cluster server. In one embodiment, the conversion module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0120] The response module 430 is used to determine a balanced partitioning strategy in response to the received data query task. In one embodiment, the response module 430 can be used to perform the operation S230 described above, which will not be described in detail here.

[0121] The partitioning module 440 is used to partition the data in the business master table based on a balanced partitioning strategy to obtain multiple business sub-tables, wherein the data volume of each of the multiple business sub-tables is the same, and the multiple business sub-tables are evenly distributed on multiple nodes in the cluster server. In one embodiment, the partitioning module 440 can be used to perform the operation S240 described above, which will not be repeated here.

[0122] The execution module 450 is used to execute the data query task for multiple business sub-tables to obtain the target query result. In one embodiment, the execution module 450 can be used to execute the operation S250 described above, which will not be described in detail here.

[0123] Optionally, the execution module 450 includes a first execution submodule, a second execution submodule, a third execution submodule and a fourth execution submodule.

[0124] The first execution submodule is used to execute a data query subtask for a plurality of business subtables to obtain initial query results corresponding to each of the plurality of business subtables.

[0125] The second execution submodule is used to execute the first data aggregation subtask, aggregate multiple initial query results, and obtain an intermediate aggregation result.

[0126] The third execution submodule is used to execute the second data aggregation subtask, aggregate the intermediate aggregation results, and obtain the target aggregation results, wherein the target aggregation results include the aggregation results of the deduplication results corresponding to the reference field information.

[0127] The fourth execution submodule is used to execute the data averaging subtask, perform average processing on the target aggregation result, and obtain the target query result, wherein the target query result includes the average result of the reference field information.

[0128] Optionally, the second execution submodule includes a first execution unit.

[0129] The first execution unit is used to execute the grouping operator bound to the benchmark field information and the reference field information, the aggregation operator bound to the reference field information, and the counting operator bound to the reference field information in the first data aggregation subtask, aggregate multiple query results, and obtain an intermediate aggregation result, wherein the intermediate aggregation result includes the benchmark field grouping sub-result, the reference field grouping sub-result, the reference field grouping aggregation sub-result, and the reference field grouping counting sub-result, and the reference field grouping sub-result represents the deduplication result of the reference field information.

[0130] Optionally, the third execution submodule includes a second execution unit and a third execution unit.

[0131] The second execution unit is used to execute the grouping operator bound to the reference field information in the second data aggregation subtask, group the intermediate aggregation results, and obtain the second grouping result corresponding to the reference field information.

[0132] The third execution unit is used to execute the aggregation operator in the second data aggregation subtask, perform aggregation operations on the second grouping results, and obtain target aggregation results, wherein the target aggregation results include the aggregation subresults of the reference field grouping subresults, the aggregation subresults of the reference field grouping aggregation subresults, and the aggregation subresults of the reference field grouping counting subresults, and the aggregation subresults of the reference field grouping subresults represent the aggregation results of the deduplication results corresponding to the reference field information.

[0133] Optionally, the fourth execution submodule includes a fourth execution unit and a fifth execution unit.

[0134] The fourth execution unit is used to execute the data averaging subtask, and average the aggregation subresults of the reference field grouping aggregation subresults and the aggregation subresults of the reference field grouping counting subresults in the target aggregation result to obtain the reference field average subresult, wherein the reference field average subresult represents the average result of the reference field information.

[0135] The fifth execution unit is used to obtain the target query result according to the average sub-result of the reference field and the aggregated sub-result of the reference field grouping sub-result.

[0136] Optionally, the division module 440 includes a first division submodule and a second division submodule.

[0137] The first partitioning submodule is used to determine the number of partitions and the data volume of the business total table based on a balanced partitioning strategy.

[0138] The second partitioning submodule is used to partition the data in the business master table according to the number of partitions and the data volume of the business master table, and determine the data of each of the multiple business subtables.

[0139] Optionally, any multiple modules among the identification module 410, the conversion module 420, the response module 430, the partitioning module 440 and the execution module 450 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. Optionally, at least one of the identification module 410, the conversion module 420, the response module 430, the partitioning module 440 and the execution module 450 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of the identification module 410 , the conversion module 420 , the response module 430 , the division module 440 , and the execution module 450 may be at least partially implemented as a computer program module, which may perform a corresponding function when executed.

[0140] Figure 5 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention is shown.

[0141] Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0142] like Figure 5 As shown, the computer electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an on-board memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0143] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in one or more memories.

[0144] Optionally, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0145] Optionally, the method flow according to the embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. Optionally, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0146] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the data processing method according to the embodiment of the present invention is implemented.

[0147] Optionally, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0148] For example, optionally, the computer-readable storage medium may include the ROM 502 and / or the RAM 503 described above and / or one or more memories other than the ROM 502 and the RAM 503 .

[0149] An embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the data processing method provided by the embodiment of the present invention.

[0150] When the computer program is executed by the processor 501, the above functions defined in the system / device of the embodiment of the present invention are performed. Optionally, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0151] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0152] Optionally, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features recorded in the various embodiments of the present invention can be combined and / or combined in various ways. All these combinations and / or combinations fall within the scope of the present invention.

[0154] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Identify the data deduplication tasks related to the business summary table and obtain the identification results; In the case where the recognition result includes a deduplication operator, the data deduplication task is converted to generate a data query task that does not include the deduplication operator, wherein the deduplication operator in the data deduplication task is bound with reference field information for performing a deduplication operation on the reference field information, the grouping operator in the data deduplication task is bound with reference field information for performing a grouping operation on the reference field information, and the grouping operator in the data query task is bound with the reference field information and the reference field information for performing a grouping operation on the reference field information and the reference field information; In response to the received data query task, determining a balanced partitioning strategy; Based on the balanced partitioning strategy, the data in the business master table is partitioned to obtain multiple business sub-tables, wherein the data volumes of the multiple business sub-tables are the same and the multiple business sub-tables are evenly distributed on multiple nodes in the cluster server; The data query task is executed for the multiple business sub-tables to obtain target query results.

2. The method according to claim 1, characterized in that The data deduplication task also includes an aggregation operator bound to the deduplication result corresponding to the reference field information and an average operator bound to the reference field information, the aggregation operator is used to perform an aggregation operation on the deduplication result of the reference field information, and the average operator is used to perform an average operation on the reference field information. The data query task includes a data query subtask, a first data aggregation subtask, a second data aggregation subtask and a data average subtask; The data query task is executed for the multiple business sub-tables to obtain the target query result, including: Execute the data query subtask for the multiple business subtables to obtain initial query results corresponding to the multiple business subtables respectively; Execute the first data aggregation subtask to aggregate the multiple initial query results to obtain an intermediate aggregation result; Execute the second data aggregation subtask to aggregate the intermediate aggregation results to obtain a target aggregation result, wherein the target aggregation result includes an aggregation result of the deduplication result corresponding to the reference field information; The data averaging subtask is executed to average the target aggregation result to obtain the target query result, wherein the target query result includes the average result of the reference field information.

3. The method according to claim 2, characterized in that Executing the first data aggregation subtask to aggregate the multiple query results to obtain an intermediate aggregation result includes: Execute the grouping operator bound to the benchmark field information and the reference field information, the aggregation operator bound to the reference field information, and the counting operator bound to the reference field information in the first data aggregation subtask, aggregate the multiple query results, and obtain the intermediate aggregation result, wherein the intermediate aggregation result includes the benchmark field grouping sub-result, the reference field grouping sub-result, the reference field grouping aggregation sub-result, and the reference field grouping counting sub-result, and the reference field grouping sub-result represents the deduplication result of the reference field information.

4. The method according to claim 3, characterized in that Executing the second data aggregation subtask to aggregate the intermediate aggregation results to obtain a target aggregation result includes: Execute the grouping operator bound to the reference field information in the second data aggregation subtask, group the intermediate aggregation results, and obtain a second grouping result corresponding to the reference field information; Execute the aggregation operator in the second data aggregation subtask, perform aggregation operations on the second grouping results, and obtain the target aggregation result, wherein the target aggregation result includes the aggregation subresult of the reference field grouping subresult, the aggregation subresult of the reference field grouping aggregation subresult, and the aggregation subresult of the reference field grouping counting subresult, and the aggregation subresult of the reference field grouping subresult represents the aggregation result of the deduplication result corresponding to the reference field information.

5. The method according to claim 4, characterized in that Executing the data averaging subtask to average the target aggregation result to obtain the target query result includes: Execute the data averaging subtask, perform an averaging operation on the aggregated subresults of the reference field grouping aggregated subresults and the aggregated subresults of the reference field grouping counting subresults in the target aggregated result, and obtain a reference field averaged subresult, wherein the reference field averaged subresult represents the averaged result of the reference field information; The target query result is obtained according to the reference field average sub-result and the aggregated sub-result of the reference field grouping sub-result.

6. The method according to claim 1, characterized in that Based on the balanced partitioning strategy, the data in the business master table is partitioned to obtain multiple business sub-tables, including: Based on the balanced partitioning strategy, determine the number of partitions and the amount of data in the business summary table; The data in the business general table is divided according to the number of partitions and the amount of data in the business general table, and the data of each of the multiple business sub-tables is determined.

7. The method according to claim 1, characterized in that The balanced partitioning strategy also includes at least one of the following: Polling partition strategy, random partition strategy.

8. A data processing device, characterized in that: The device comprises: The recognition module is used to identify the data deduplication tasks related to the business summary table and obtain the recognition results; A conversion module, used for converting the data deduplication task to generate a data query task that does not include the deduplication operator when the recognition result includes the deduplication operator, wherein the deduplication operator in the data deduplication task is bound with reference field information for performing a deduplication operation on the reference field information, the grouping operator in the data deduplication task is bound with reference field information for performing a grouping operation on the reference field information, and the grouping operator in the data query task is bound with the reference field information and the reference field information for performing a grouping operation on the reference field information and the reference field information; A response module, configured to determine a balanced partitioning strategy in response to the received data query task; A partitioning module, configured to partition the data in the business master table based on the balanced partitioning strategy to obtain a plurality of business sub-tables, wherein the data volumes of the plurality of business sub-tables are the same and the plurality of business sub-tables are evenly distributed on a plurality of nodes in the cluster server; The execution module is used to execute the data query task for the multiple business sub-tables to obtain the target query result.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images