Data processing method and device, equipment and storage medium

Abstract

The invention provides a data processing method and device, equipment and a storage medium, and can be applied to the technical field of databases. The method comprises the following steps: identifying a data deduplication task related to a service general table to obtain an identification result; under the condition that the recognition result contains a deduplication operator, converting the data deduplication task, and generating a data query task which does not contain the deduplication operator; in response to the received data query task, determining a balanced division strategy; based on a balanced division strategy, dividing data in the service general table to obtain a plurality of service sub-tables, the data volumes of the plurality of service sub-tables being the same, and the plurality of service sub-tables being distributed on a plurality of nodes in the cluster server in a balanced manner; and executing the data query task for the plurality of service sub-tables to obtain a target query result.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data processing method, apparatus, device and storage medium. Background Art

[0002] In cluster databases, traditional parallel grouping and aggregation operations rely on data sampling to determine how the data is partitioned. For structured queries containing the DISTINCT operator, the system currently only supports data partitioning strategies based on hashing algorithms. When data values ​​differ significantly between different groups in a business table and the fields bound to the DISTINCT operator are relatively unique, hashing-based data partitioning strategies can lead to uneven data distribution across cluster nodes. Because data may be concentrated on a few nodes, while other nodes are underutilized, the increased computational load on a few nodes can lead to idle resources on other nodes, impacting the parallel processing capabilities of cluster server queries and the overall system throughput. Summary of the Invention

[0003] In view of the above problems, the present invention provides a data processing method, apparatus, device and storage medium.

[0004] According to a first aspect of the present invention, there is provided a data processing method, comprising:

[0005] Identify the data deduplication tasks related to the business summary table and obtain an identification result; when the identification result includes a deduplication operator, convert the data deduplication task to generate a data query task that does not include a deduplication operator, wherein the deduplication operator in the data deduplication task is bound with reference field information for performing a deduplication operation on the reference field information, the grouping operator in the data deduplication task is bound with benchmark field information for performing a grouping operation on the benchmark field information, and the grouping operator in the data query task is bound with benchmark field information and reference field information for performing a grouping operation on the benchmark field information and the reference field information; in response to the received data query task, determine a balanced partitioning strategy; based on the balanced partitioning strategy, partition the data in the business summary table to obtain multiple business sub-tables, wherein the data volume of each of the multiple business sub-tables is the same, and the multiple business sub-tables are evenly distributed on multiple nodes in the cluster server; execute the data query task for the multiple business sub-tables to obtain the target query result.

[0006] Optionally, the data deduplication task also includes an aggregation operator bound to the deduplication results corresponding to the reference field information and an average operator bound to the reference field information. The aggregation operator is used to perform an aggregation operation on the deduplication results of the reference field information, and the average operator is used to perform an average operation on the reference field information. The data query task includes a data query subtask, a first data aggregation subtask, a second data aggregation subtask, and a data averaging subtask. The data query task is executed for multiple business subtables to obtain target query results, including:

[0007] For multiple business subtables, execute the data query subtask to obtain the initial query results corresponding to each of the multiple business subtables. Execute the data first aggregation subtask to aggregate the multiple initial query results to obtain an intermediate aggregation result. Execute the data second aggregation subtask to aggregate the intermediate aggregation results to obtain a target aggregation result, where the target aggregation result includes the aggregation result of the deduplication results corresponding to the reference field information. Execute the data averaging subtask to average the target aggregation results to obtain the target query result, where the target query result includes the average result of the reference field information.

[0008] Optionally, execute the first data aggregation subtask to aggregate multiple query results to obtain intermediate aggregation results, including:

[0009] Execute the grouping operator bound to the benchmark field information and the reference field information, the aggregation operator bound to the reference field information, and the counting operator bound to the reference field information in the first data aggregation subtask, aggregate multiple query results, and obtain an intermediate aggregation result, wherein the intermediate aggregation result includes the benchmark field grouping sub-result, the reference field grouping sub-result, the reference field grouping aggregation sub-result, and the reference field grouping counting sub-result, and the reference field grouping sub-result represents the deduplication result of the reference field information.

[0010] Optionally, execute the second data aggregation subtask to aggregate the intermediate aggregation results to obtain the target aggregation result, including:

[0011] Execute the grouping operator bound to the reference field information in the second data aggregation subtask, group the intermediate aggregation results, and obtain a second grouping result corresponding to the reference field information; execute the aggregation operator in the second data aggregation subtask, perform aggregation operations on the second grouping results, and obtain a target aggregation result, wherein the target aggregation result includes the aggregation sub-result of the reference field grouping sub-result, the aggregation sub-result of the reference field grouping aggregation sub-result, and the aggregation sub-result of the reference field grouping counting sub-result, and the aggregation sub-result of the reference field grouping sub-result represents the aggregation result of the deduplication result corresponding to the reference field information.

[0012] Optionally, execute the data averaging subtask to average the target aggregation results to obtain the target query results, including:

[0013] Execute the data averaging subtask to average the aggregation sub-results of the reference field grouping aggregation sub-results and the aggregation sub-results of the reference field grouping counting sub-results in the target aggregation result to obtain the reference field average sub-result, where the reference field average sub-result represents the average result of the reference field information; obtain the target query result based on the reference field average sub-result and the aggregation sub-result of the reference field grouping sub-result.

[0014] Optionally, based on a balanced partitioning strategy, the data in the business master table is partitioned to obtain multiple business sub-tables, including:

[0015] Based on the balanced partitioning strategy, the number of partitions and the amount of data in the business table are determined; based on the number of partitions and the amount of data in the business table, the data in the business table is divided to determine the data of each of the multiple business sub-tables.

[0016] Optionally, the balanced partitioning strategy also includes at least one of the following: a round-robin partitioning strategy and a random partitioning strategy.

[0017] A second aspect of the present invention provides a data processing device, comprising:

[0018] The recognition module is used to identify the data deduplication tasks related to the business summary table and obtain the recognition results;

[0019] a conversion module for converting the data deduplication task to generate a data query task that does not include the deduplication operator when the recognition result includes the deduplication operator, wherein the deduplication operator in the data deduplication task is bound to reference field information and is used to perform a deduplication operation on the reference field information, the grouping operator in the data deduplication task is bound to base field information and is used to perform a grouping operation on the base field information, and the grouping operator in the data query task is bound to base field information and reference field information and is used to perform a grouping operation on the base field information and the reference field information;

[0020] A response module, configured to determine a balanced partitioning strategy in response to a received data query task;

[0021] A partitioning module is used to partition the data in the business master table based on a balanced partitioning strategy to obtain multiple business sub-tables, wherein the data volume of each of the multiple business sub-tables is the same and the multiple business sub-tables are evenly distributed on multiple nodes in the cluster server;

[0022] The execution module is used to execute data query tasks for multiple business sub-tables and obtain target query results.

[0023] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned data processing method.

[0024] A fourth aspect of the present invention further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned data processing method.

[0025] According to the data processing method, apparatus, device and storage medium provided by the present invention, a data deduplication task containing a deduplication operator is converted to generate a data query task that does not contain a deduplication operator, and a balanced partitioning strategy is determined, thereby obtaining multiple business sub-tables; the data query task is executed to obtain a target query result. Since the deduplication operator bound to the reference field information is flexibly replaced by the grouping operator bound to the benchmark field information and the reference field information, it is ensured that the data query task does not contain a deduplication operator, and a balanced partitioning strategy for data segmentation in the business master table is determined. Thus, while obtaining the target query result that is the same as the result of executing the data deduplication task, the data distribution skew problem caused by deduplication of a business master table with large data volume differences in different groups using a deduplication operator is solved, and the computing resources of multiple nodes in the cluster server are fully utilized, thereby improving the parallel processing capability of the cluster server and the overall throughput of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0027] Figure 1 An application scenario of the data processing method according to an embodiment of the present invention is shown;

[0028] Figure 2 A flow chart of a data processing method according to an embodiment of the present invention is shown;

[0029] Figure 3 A flow chart of generating target query results according to an embodiment of the present invention is shown;

[0030] Figure 4 Shown is a structural block diagram of a data processing device according to an embodiment of the present invention;

[0031] Figure 5 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0032] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0033] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise," "include," etc. used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0034] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0035] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0036] During the implementation of this invention, it was discovered that for structured queries containing the DISTINCT operator, the system currently only supports data partitioning strategies based on hashing algorithms. When data values ​​differ significantly between different groups in a business table and the fields bound to the DISTINCT operator are relatively unique, this hashing-based data partitioning strategy can lead to uneven data distribution across cluster nodes. Therefore, a data processing method is urgently needed that can achieve data deduplication without executing a DISTINCT operator and achieve uniform data partitioning.

[0037] In view of this, an embodiment of the present invention provides a data processing method, apparatus, device and storage medium. The method includes: identifying a data deduplication task related to a business general table to obtain an identification result; when the identification result includes a deduplication operator, converting the data deduplication task to generate a data query task that does not include the deduplication operator, wherein the deduplication operator in the data deduplication task is bound to reference field information for performing a deduplication operation on the reference field information, the grouping operator in the data deduplication task is bound to benchmark field information for performing a grouping operation on the benchmark field information, and the grouping operator in the data query task is bound to benchmark field information and reference field information for performing a grouping operation on the benchmark field information and the reference field information; in response to a received data query task, determining a balanced partitioning strategy; based on the balanced partitioning strategy, partitioning the data in the business general table to obtain multiple business sub-tables, wherein the data volume of each of the multiple business sub-tables is the same and the multiple business sub-tables are evenly distributed on multiple nodes in a cluster server; executing the data query task for the multiple business sub-tables to obtain a target query result.

[0038] In the technical solution of the present invention, the user information involved (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0039] Figure 1 The application scenario of the data processing method according to the embodiment of the present invention is shown.

[0040] like Figure 1 As shown, the business system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0041] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0042] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0043] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0044] It should be noted that the data processing method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present invention can generally be set in the server 105. The data processing method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data processing device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0045] Alternatively, the data processing method provided in the embodiment of the present invention may also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may also be executed by another terminal device different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the data processing apparatus provided in the embodiment of the present invention may also be provided in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may be provided in another terminal device different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0046] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0047] It should be noted that the sequence numbers of the operations in the following method are only used to indicate the operation for the purpose of description, and should not be regarded as indicating the order in which the operations should be performed. Unless explicitly stated, the method does not need to be performed in the order shown.

[0048] Figure 2 A flow chart of a data processing method according to an embodiment of the present invention is shown.

[0049] like Figure 2 As shown, the method 200 includes operations S210 to S250.

[0050] In operation S210 , data deduplication tasks related to the business summary table are identified to obtain an identification result.

[0051] Optionally, the cluster database is deployed on a server, and the business summary table represents a data table related to the system business stored in the cluster database.

[0052] Optionally, a data deduplication task represents a task for deduplicating data groups obtained from a business summary table. The data deduplication task can be a structured query language (SQL). In response to receiving a data deduplication task for a business summary table, the server identifies the operator in the data deduplication task and obtains an identification result. The identification result can be a result of the data deduplication task including the deduplication operator or a result of the data deduplication task excluding the deduplication operator.

[0053] In operation S220 , when the recognition result includes a deduplication operator, the data deduplication task is converted to generate a data query task that does not include the deduplication operator.

[0054] Optionally, the deduplication operator in a data deduplication task is bound to reference field information for deduplication operations on the reference field information, and the grouping operator in a data deduplication task is bound to base field information for grouping operations on the base field information. The reference field information represents the field information to be deduplicated in the business summary table, and the base field information represents the field information to be grouped in the business summary table.

[0055] Optionally, a deduplication operator can remove duplicate data from the business summary table based on bound field information. A grouping operator can group data in the business summary table based on bound field information and perform aggregation calculations on each group. For example, a data deduplication task that includes a deduplication operator is: select distinct(j) from t group by (i), where distinct is the deduplication operator, j is the reference field information, group by is the grouping operator, i is the reference field information, and t is the business summary table.

[0056] Optionally, if the identification result includes a deduplication operator, the data deduplication task is converted to generate a data query task without the deduplication operator. For example, the deduplication operator distinct(j), which is bound to the reference field, is replaced with the grouping operator group by(j), which is also bound to the reference field. This results in a grouping operator bound to both the base field and the reference field, and generates a data query task. For example, the data query task is: select i, j from t groupby(i, j).

[0057] Optionally, the grouping operator in the data query task is bound with the base field information and the reference field information, and is used to perform a grouping operation on the base field information and the reference field information.

[0058] In operation S230 , a balanced partitioning strategy is determined in response to the received data query task.

[0059] Optionally, in response to the data query task, the server identifies other operators in the data query task and evaluates and obtains a balanced partitioning strategy on the premise that the data query task does not include a deduplication operator.

[0060] Optionally, the balanced partitioning strategy represents a strategy for distributing the data of the business summary table to the cluster servers. When the capacity of multiple nodes in the cluster servers remains unchanged, the data of the business summary table is evenly distributed to each node according to the balanced partitioning strategy.

[0061] In operation S240 , the data in the business master table is divided based on a balanced division strategy to obtain a plurality of business sub-tables.

[0062] Optionally, the data volumes of the multiple business sub-tables are the same, and the multiple business sub-tables are evenly distributed on multiple nodes in the cluster server.

[0063] For example, the number of partitions n is reconfigured according to the balanced partitioning strategy, and the data in the business table is evenly divided according to the number of partitions to obtain n business sub-tables, which are then distributed to n nodes in the cluster server.

[0064] In operation S250 , a data query task is executed for the multiple business sub-tables to obtain a target query result.

[0065] Optionally, multiple nodes perform data query tasks on their respective business sub-tables, and obtaining the target query results may include: summarizing the data of multiple business sub-tables queried, and then grouping the summarized data according to the benchmark field information and reference field information and performing aggregation calculations to obtain the target query results.

[0066] Optionally, the target query result obtained by executing the data query task includes the result obtained by executing the data deduplication task. The target query result can achieve the deduplication effect.

[0067] Optionally, by flexibly replacing the deduplication operator bound to the reference field information with the grouping operator bound to the benchmark field information and the reference field information, it is ensured that the data query task does not contain the deduplication operator, and a balanced partitioning strategy for data segmentation in the business summary table is determined. Thus, while obtaining the same target query result as the result of executing the data deduplication task, it solves the data distribution skew problem caused by using the deduplication operator to deduplicate the business summary table with large differences in data volume in different groups, fully utilizes the computing resources of multiple nodes in the cluster server, and improves the parallel processing capability of the cluster server and the overall throughput of the system.

[0068] Figure 3 A flowchart of generating a target query result according to an embodiment of the present invention is shown.

[0069] like Figure 3 As shown, executing a data query task for multiple business sub-tables to obtain target query results includes operations S310 to S340.

[0070] In operation S310 , a data query subtask is executed for the multiple business subtables to obtain initial query results corresponding to the multiple business subtables.

[0071] In operation S320 , a first data aggregation subtask is executed to aggregate the multiple initial query results to obtain an intermediate aggregation result.

[0072] In operation S330 , the second data aggregation subtask is executed to aggregate the intermediate aggregation results to obtain a target aggregation result.

[0073] In operation S340 , the data averaging subtask is executed to average the target aggregation result to obtain the target query result.

[0074] Optionally, the data deduplication task also includes an aggregation operator bound to the deduplication results corresponding to the reference field information and an average operator bound to the reference field information. The aggregation operator is used to perform aggregation operations on the deduplication results of the reference field information, and the average operator is used to perform average operations on the reference field information.

[0075] For example, the data deduplication task is: select sum(distinct j), avg(j) from t group by (i), where sum is the aggregation operator, distinct j is the deduplication result corresponding to the reference field information, and avg is the average operator.

[0076] Optionally, while replacing the deduplication operator in the data deduplication task, the averaging operator bound to the reference field information in the data deduplication task is also replaced to obtain a data query task. The converted data query task includes a data query subtask, a data first aggregation subtask, a data second aggregation subtask, and a data averaging subtask. After executing the four subtasks, a target query result identical to the result of executing the data deduplication task is obtained. The target query result includes the aggregation result of the deduplication result corresponding to the reference field information and the average result corresponding to the reference field information.

[0077] Optionally, the data query subtask is used to query the data of the business subtable. The data of the business subtable includes all the base field information and all the reference field information in the business subtable.

[0078] For example, a cluster server has nodes 1, 2, and 3, and a corresponding business sub-table is distributed on each node. Node 1 responds to the data query sub-task, performs data query on the business sub-table on node 1, and outputs the initial query result of node 1; node 2 responds to the data query sub-task, performs data query on the business sub-table on node 2, and outputs the initial query result of node 2; node 3 responds to the data query sub-task, performs data query on the business sub-table on node 3, and outputs the initial query result of node 3.

[0079] Optionally, the initial query result of the business subtable represents all base field information and all reference field information queried from the business subtable.

[0080] Optionally, multiple initial query results are aggregated to obtain an intermediate query result. For example, the initial query results of node 1, node 2, and node 3 are aggregated to obtain an intermediate query result. The intermediate query result represents all base field information and all reference field information in the business summary table.

[0081] Table 1 schematically shows an intermediate query result table according to an embodiment of the present invention.

[0082]

[0083] Optionally, the first data aggregation subtask includes a grouping operator and an aggregation operator. The intermediate query results are grouped based on the base field information and reference field information bound to the grouping operator. A first round of aggregation calculations is then performed based on the field information bound to the aggregation operator to obtain an intermediate aggregation result. For example, the aggregation operator can be the sum operator. The intermediate aggregation result includes a deduplication result corresponding to the reference field information.

[0084] Optionally, the second data aggregation subtask includes a grouping operator and an aggregation operator. The intermediate aggregation results are grouped according to the reference field information bound to the grouping operator. A second round of aggregation calculations is then performed based on the field information bound to the aggregation operator to obtain the target aggregation result. The target aggregation result includes the aggregation result of the deduplication results corresponding to the reference field information.

[0085] Optionally, the data averaging subtask is used to perform an average operation on the target aggregation result to obtain a target query result, which includes an average result of the reference field information.

[0086] Optionally, execute the first data aggregation subtask to aggregate multiple query results to obtain intermediate aggregation results, including:

[0087] Execute the grouping operator bound to the benchmark field information and the reference field information, the aggregation operator bound to the reference field information, and the counting operator bound to the reference field information in the first data aggregation subtask, aggregate multiple query results, and obtain an intermediate aggregation result, wherein the intermediate aggregation result includes the benchmark field grouping sub-result, the reference field grouping sub-result, the reference field grouping aggregation sub-result, and the reference field grouping counting sub-result, and the reference field grouping sub-result represents the deduplication result of the reference field information.

[0088] Optionally, during the first round of aggregation operations on the intermediate query results, the grouping operator bound to the base field information and the reference field information is executed first, followed by the aggregation operator and the counting operator bound to the reference field information. For example, the first data aggregation subtask can be Select i as i_i, j as j_j, sum(j) assum_j, count(j) as cnt_j from t group by (i, j), where group by (i, j) is used to group the intermediate query results based on the base field information and the reference field information, and return the base field grouped sub-result i_i and the reference field grouped sub-result j_j.

[0089] Optionally, sum(j) and count(j) are respectively the aggregation operator and the counting operator bound with reference field information in the aggregation query subtask.

[0090] Optionally, an aggregation operator bound to the reference field information is executed to obtain the reference field grouping aggregation sub-result sum_j, and a count operator bound to the reference field information is executed to obtain the reference field grouping count sub-result count_j. The intermediate aggregation results include the reference field grouping sub-result i_i, the reference field grouping sub-result j_j, the reference field grouping aggregation sub-result sum_j, and the reference field grouping count sub-result count_j.

[0091] Table 2 schematically shows an intermediate aggregation result table according to an embodiment of the present invention.

[0092]

[0093] Optionally, generating intermediate aggregation results through the first round of aggregation calculations provides a more balanced data foundation for the second round of aggregation calculations, which helps improve query performance. The multi-stage aggregation process optimizes data distribution and processing flow.

[0094] Optionally, executing a second data aggregation subtask to aggregate the intermediate aggregation results to obtain a target aggregation result includes:

[0095] Execute the grouping operator bound to the reference field information in the second data aggregation subtask, group the intermediate aggregation results, and obtain a second grouping result corresponding to the reference field information; execute the aggregation operator in the second data aggregation subtask, perform aggregation operations on the second grouping results, and obtain a target aggregation result, wherein the target aggregation result includes the aggregation sub-result of the reference field grouping sub-result, the aggregation sub-result of the reference field grouping aggregation sub-result, and the aggregation sub-result of the reference field grouping counting sub-result, and the aggregation sub-result of the reference field grouping sub-result represents the aggregation result of the deduplication result corresponding to the reference field information.

[0096] Optionally, when performing a second round of aggregation on the intermediate aggregation results, first execute the grouping operator bound to the reference field information, then execute the aggregation operator. For example, the second data aggregation subtask can be: Select sum(j_j), sum(sum_j), sum(cnt_j) from t1 group by (i), where group by (i) is a grouping operator bound to the reference field information. The grouping operator is executed on the intermediate aggregation results to group them according to the reference field information, resulting in a second grouping result. A second round of aggregation is then performed on the second grouping result to obtain the target aggregation result. The target aggregation result includes the aggregation sub-result of the reference field grouping sub-result, the aggregation sub-result of the reference field grouping aggregation sub-result, and the aggregation sub-result of the reference field grouping count sub-result.

[0097] Optionally, the second data aggregation subtask may include multiple aggregation operators, for example, executing the summation operator Sum(j_j) bound to the reference field grouping sub-result j_j to obtain the aggregation sub-result s_j_j of the reference field grouping sub-result; executing the summation operator Sum(sum_j) bound to the reference field grouping aggregation sub-result sum_j to obtain the aggregation sub-result s_ sum _j of the reference field grouping aggregation sub-result; executing the summation operator Sum(count_j) bound to the reference field grouping count sub-result count_j to obtain the aggregation sub-result s_coun_j of the reference field grouping count sub-result.

[0098] Table 3 schematically shows a target aggregation result table according to an embodiment of the present invention.

[0099]

[0100] Optionally, execute the data averaging subtask to average the target aggregation results to obtain the target query results, including:

[0101] Execute the data averaging subtask to average the aggregation sub-results of the reference field grouping aggregation sub-results and the aggregation sub-results of the reference field grouping counting sub-results in the target aggregation result to obtain the reference field average sub-result, where the reference field average sub-result represents the average result of the reference field information; obtain the target query result based on the reference field average sub-result and the aggregation sub-result of the reference field grouping sub-result.

[0102] Optionally, the averaging operation can be performed by calculating the ratio between the aggregated sub-result s_sum_j of the reference field grouped aggregated sub-result and the aggregated sub-result s_coun_j of the reference field grouped count sub-result, yielding the reference field average sub-result a_j. For example, a_j can be the result of Sum(sum_j) / Sum(count_j). The target query result is obtained by combining the reference field average sub-result a_j with the aggregated sub-result s_j_j of the reference field grouped sub-result.

[0103] Table 4 schematically shows a target query result table according to an embodiment of the present invention.

[0104]

[0105] Optionally, the aggregate sub-result s_j_j of the reference field grouping sub-result is the same as the aggregate result of the deduplication result corresponding to the reference field information obtained by sum(distinct j) in executing the data deduplication task, and the reference field average sub-result a_j is the same as the average result corresponding to the reference field information obtained by avg(j) in executing the data deduplication task. Therefore, the target query result is the same as the result of executing the data deduplication task.

[0106] Optionally, in the second round of aggregation calculations, the query execution plan is optimized by removing the deduplication operator and converting it into a form that is easier to process in parallel, thereby simplifying the aggregation operation and reducing the computational complexity; by converting the average operator in the data deduplication task into a combination of a sum operator and a count operator, and by appropriately adjusting these aggregation operators at different aggregation stages, data can be processed more efficiently.

[0107] Optionally, based on a balanced partitioning strategy, the data in the business master table is partitioned to obtain multiple business sub-tables, including:

[0108] Based on the balanced partitioning strategy, the number of partitions and the amount of data in the business table are determined; based on the number of partitions and the amount of data in the business table, the data in the business table is divided to determine the data of each of the multiple business sub-tables.

[0109] Optionally, the number of partitions n is reconfigured according to the balanced partitioning strategy, and the average data volume of the business sub-table is calculated based on the number of partitions and the data volume of the business total table.

[0110] Optionally, the data in the business general table is evenly divided according to the average data volume to obtain n business sub-tables, and the n business sub-tables are respectively distributed to n nodes in the cluster server.

[0111] Optionally, the balanced partitioning strategy also includes at least one of the following: a round-robin partitioning strategy and a random partitioning strategy.

[0112] Alternatively, the Round-robin (rr) partitioning strategy constructs a batch for each piece of data in the business summary table. Each batch contains only one piece of data, and the data of each batch is written to each node of the cluster server in turn, thereby ensuring balanced data distribution.

[0113] Optionally, the random partitioning strategy (Randomness) requires first calculating the number of nodes in the cluster server, and then randomly and evenly scattering the data in the business summary table and distributing it to each node.

[0114] Optionally, the balanced partitioning strategy dynamically adjusts the distribution of data on each node according to the actual distribution of data, achieving load balancing, enhancing the system's parallel processing capabilities, and improving overall query performance and throughput.

[0115] Based on the above data processing method, the present invention also provides a data processing device. Figure 4 The device is described in detail.

[0116] Figure 4 A structural block diagram of a data processing device according to an embodiment of the present invention is shown.

[0117] like Figure 4 As shown, the data processing device 400 of this embodiment includes an identification module 410 , a conversion module 420 , a response module 430 , a division module 440 and an execution module 450 .

[0118] The identification module 410 is configured to determine a balanced partitioning strategy in response to a received data query task. In one embodiment, the identification module 410 may be configured to execute the operation S210 described above, which will not be described in detail herein.

[0119] Conversion module 420 is configured to partition the data in the business master table based on a balanced partitioning strategy to obtain multiple business sub-tables, wherein each of the multiple business sub-tables has the same amount of data and is evenly distributed across multiple nodes in the cluster server. In one embodiment, conversion module 420 can be configured to perform operation S220 described above and will not be further described here.

[0120] The response module 430 is configured to determine a balanced partitioning strategy in response to the received data query task. In one embodiment, the response module 430 may be configured to execute the operation S230 described above, which will not be described in detail herein.

[0121] Partitioning module 440 is configured to partition the data in the business master table based on a balanced partitioning strategy to obtain multiple business sub-tables, wherein each of the multiple business sub-tables has the same amount of data and is evenly distributed across multiple nodes in the cluster server. In one embodiment, partitioning module 440 may be configured to perform operation S240 described above and will not be further described herein.

[0122] The execution module 450 is used to execute the data query task for multiple business sub-tables to obtain the target query result. In one embodiment, the execution module 450 can be used to execute the operation S250 described above, which will not be repeated here.

[0123] Optionally, the execution module 450 includes a first execution submodule, a second execution submodule, a third execution submodule and a fourth execution submodule.

[0124] The first execution submodule is configured to execute a data query subtask for the multiple business subtables to obtain initial query results corresponding to the multiple business subtables.

[0125] The second execution submodule is used to execute the first data aggregation subtask, aggregate multiple initial query results, and obtain intermediate aggregation results.

[0126] The third execution submodule is used to execute the second data aggregation subtask, aggregate the intermediate aggregation results, and obtain the target aggregation result, wherein the target aggregation result includes the aggregation result of the deduplication result corresponding to the reference field information.

[0127] The fourth execution submodule is used to execute the data averaging subtask, perform average processing on the target aggregation result, and obtain the target query result, wherein the target query result includes the average result of the reference field information.

[0128] Optionally, the second execution submodule includes a first execution unit.

[0129] The first execution unit is used to execute the grouping operator bound to the benchmark field information and the reference field information, the aggregation operator bound to the reference field information, and the counting operator bound to the reference field information in the first data aggregation subtask, aggregate multiple query results, and obtain an intermediate aggregation result, wherein the intermediate aggregation result includes the benchmark field grouping sub-result, the reference field grouping sub-result, the reference field grouping aggregation sub-result, and the reference field grouping counting sub-result, and the reference field grouping sub-result represents the deduplication result of the reference field information.

[0130] Optionally, the third execution submodule includes a second execution unit and a third execution unit.

[0131] The second execution unit is used to execute the grouping operator bound to the reference field information in the second data aggregation subtask, group the intermediate aggregation results, and obtain a second grouping result corresponding to the reference field information.

[0132] The third execution unit is used to execute the aggregation operator in the second aggregation subtask of the data, perform aggregation operations on the second grouping results, and obtain the target aggregation results, wherein the target aggregation results include the aggregation subresults of the reference field grouping subresults, the aggregation subresults of the reference field grouping aggregation subresults, and the aggregation subresults of the reference field grouping counting subresults, and the aggregation subresults of the reference field grouping subresults represent the aggregation results of the deduplication results corresponding to the reference field information.

[0133] Optionally, the fourth execution submodule includes a fourth execution unit and a fifth execution unit.

[0134] The fourth execution unit is used to execute the data averaging subtask, and perform an averaging operation on the aggregation subresults of the reference field grouping aggregation subresults and the aggregation subresults of the reference field grouping counting subresults in the target aggregation result to obtain the reference field average subresult, wherein the reference field average subresult represents the average result of the reference field information.

[0135] The fifth execution unit is configured to obtain a target query result according to the average sub-result of the reference field and the aggregated sub-result of the reference field grouping sub-result.

[0136] Optionally, the division module 440 includes a first division submodule and a second division submodule.

[0137] The first partitioning submodule is used to determine the number of partitions and the data volume of the business total table based on a balanced partitioning strategy.

[0138] The second partitioning submodule is used to partition the data in the business table according to the number of partitions and the data volume of the business table, and determine the data of each of the multiple business subtables.

[0139] Optionally, any multiple modules among identification module 410, conversion module 420, response module 430, partitioning module 440, and execution module 450 may be combined into a single module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in a single module. Optionally, at least one of identification module 410, conversion module 420, response module 430, partitioning module 440, and execution module 450 may be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or may be implemented in hardware or firmware through any other reasonable means of circuit integration or packaging, or may be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, at least one of the identification module 410 , the conversion module 420 , the response module 430 , the division module 440 and the execution module 450 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0140] Figure 5 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention is shown.

[0141] Figure 5 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0142] like Figure 5 As shown, a computer electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0143] Various programs and data required for the operation of the electronic device 500 are stored in the RAM 503. The processor 501, ROM 502, and RAM 503 are connected to each other via a bus 504. The processor 501 executes the programs in the ROM 502 and / or RAM 503 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than the ROM 502 and RAM 503. The processor 501 may also execute the programs stored in one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0144] Optionally, electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to bus 504. Electronic device 500 may also include one or more of the following components connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN card or modem. Communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 510 as needed, so that computer programs read from the removable media can be installed into storage section 508 as needed.

[0145] Optionally, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above-mentioned functions defined in the system of the embodiment of the present invention are performed. Optionally, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0146] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the data processing method according to the embodiments of the present invention.

[0147] Optionally, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0148] For example, optionally, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than the ROM 502 and RAM 503 .

[0149] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the data processing method provided by the embodiment of the present invention.

[0150] When the computer program is executed by the processor 501, the above functions defined in the system / device of the embodiment of the present invention are performed. Optionally, the above-described systems, devices, modules, units, etc. can be implemented by computer program modules.

[0151] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 509, and / or installed from a removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0152] Optionally, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, Python, "C" language, or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0154] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Identify the data deduplication tasks related to the business summary table and obtain the identification results; In the case where the recognition result includes a deduplication operator, the data deduplication task is converted to generate a data query task that does not include the deduplication operator, wherein the deduplication operator in the data deduplication task is bound with reference field information for performing a deduplication operation on the reference field information, the grouping operator in the data deduplication task is bound with reference field information for performing a grouping operation on the reference field information, and the grouping operator in the data query task is bound with the reference field information and the reference field information for performing a grouping operation on the reference field information and the reference field information; In response to the received data query task, determining a balanced partitioning strategy; Based on the balanced partitioning strategy, the data in the business master table is partitioned to obtain multiple business sub-tables, wherein the data volumes of the multiple business sub-tables are the same and the multiple business sub-tables are evenly distributed on multiple nodes in the cluster server; The data query task is executed for the multiple business sub-tables to obtain target query results.

2. The method according to claim 1, characterized in that The data deduplication task also includes an aggregation operator bound to the deduplication result corresponding to the reference field information and an average operator bound to the reference field information, the aggregation operator is used to perform an aggregation operation on the deduplication result of the reference field information, and the average operator is used to perform an average operation on the reference field information. The data query task includes a data query subtask, a first data aggregation subtask, a second data aggregation subtask and a data average subtask; The data query task is executed for the multiple business sub-tables to obtain the target query result, including: Execute the data query subtask for the multiple business subtables to obtain initial query results corresponding to the multiple business subtables respectively; Execute the first data aggregation subtask to aggregate the multiple initial query results to obtain an intermediate aggregation result; Execute the second data aggregation subtask to aggregate the intermediate aggregation results to obtain a target aggregation result, wherein the target aggregation result includes an aggregation result of the deduplication result corresponding to the reference field information; The data averaging subtask is executed to average the target aggregation result to obtain the target query result, wherein the target query result includes the average result of the reference field information.

3. The method according to claim 2, characterized in that Executing the first data aggregation subtask to aggregate the multiple query results to obtain an intermediate aggregation result includes: Execute the grouping operator bound to the benchmark field information and the reference field information, the aggregation operator bound to the reference field information, and the counting operator bound to the reference field information in the first data aggregation subtask, aggregate the multiple query results, and obtain the intermediate aggregation result, wherein the intermediate aggregation result includes the benchmark field grouping sub-result, the reference field grouping sub-result, the reference field grouping aggregation sub-result, and the reference field grouping counting sub-result, and the reference field grouping sub-result represents the deduplication result of the reference field information.

4. The method according to claim 3, characterized in that Executing the second data aggregation subtask to aggregate the intermediate aggregation results to obtain a target aggregation result includes: Execute the grouping operator bound to the reference field information in the second data aggregation subtask, group the intermediate aggregation results, and obtain a second grouping result corresponding to the reference field information; Execute the aggregation operator in the second data aggregation subtask, perform aggregation operations on the second grouping results, and obtain the target aggregation result, wherein the target aggregation result includes the aggregation subresult of the reference field grouping subresult, the aggregation subresult of the reference field grouping aggregation subresult, and the aggregation subresult of the reference field grouping counting subresult, and the aggregation subresult of the reference field grouping subresult represents the aggregation result of the deduplication result corresponding to the reference field information.

5. The method according to claim 4, characterized in that Executing the data averaging subtask to average the target aggregation result to obtain the target query result includes: Execute the data averaging subtask, perform an averaging operation on the aggregated subresults of the reference field grouping aggregated subresults and the aggregated subresults of the reference field grouping counting subresults in the target aggregated result, and obtain a reference field averaged subresult, wherein the reference field averaged subresult represents the averaged result of the reference field information; The target query result is obtained according to the reference field average sub-result and the aggregated sub-result of the reference field grouping sub-result.

6. The method according to claim 1, characterized in that Based on the balanced partitioning strategy, the data in the business master table is partitioned to obtain multiple business sub-tables, including: Based on the balanced partitioning strategy, determine the number of partitions and the amount of data in the business summary table; The data in the business general table is divided according to the number of partitions and the amount of data in the business general table, and the data of each of the multiple business sub-tables is determined.

7. The method according to claim 1, characterized in that The balanced partitioning strategy also includes at least one of the following: Polling partition strategy, random partition strategy.

8. A data processing device, characterized in that: The device comprises: The recognition module is used to identify the data deduplication tasks related to the business summary table and obtain the recognition results; A conversion module, used for converting the data deduplication task to generate a data query task that does not include the deduplication operator when the recognition result includes the deduplication operator, wherein the deduplication operator in the data deduplication task is bound with reference field information for performing a deduplication operation on the reference field information, the grouping operator in the data deduplication task is bound with reference field information for performing a grouping operation on the reference field information, and the grouping operator in the data query task is bound with the reference field information and the reference field information for performing a grouping operation on the reference field information and the reference field information; A response module, configured to determine a balanced partitioning strategy in response to the received data query task; A partitioning module, configured to partition the data in the business master table based on the balanced partitioning strategy to obtain a plurality of business sub-tables, wherein the data volumes of the plurality of business sub-tables are the same and the plurality of business sub-tables are evenly distributed on a plurality of nodes in the cluster server; The execution module is used to execute the data query task for the multiple business sub-tables to obtain the target query result.

9. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images