Data processing method, device, electronic device, storage medium and program product

By splitting the database query connection into multiple subtasks and assigning them to multiple computing nodes, the problem of query performance degradation caused by data skew is solved, and query efficiency and performance are improved.

CN118689881BActive Publication Date: 2024-11-22PINGCAP XINGCHEN (BEIJING) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411189496.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-22
Estimated Expiration
2044-08-27

Smart Images

  • Figure CN118689881B_ABST
    Figure CN118689881B_ABST
Patent Text Reader

Abstract

The disclosed embodiments provide a data processing method, device, electronic device, storage medium and program product, which relate to the field of databases, and can be used in the field of distributed databases, for example. The method includes: obtaining query information; determining at least two ranges to be connected based on each first data range in the routing information; the range to be connected includes at least one first data range; allocating at least two ranges to be connected to at least two computing nodes, so that each computing node determines the received range to be connected, and based on the routing information, connects at least two data in each data table in the range to be connected to obtain a connection result; based on the connection results corresponding to each computing node, the query result for the query information is determined. The disclosed embodiments improve the utilization rate of computing node cluster resources, and improve the efficiency and performance of query connections.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of database technology, and in particular to a data processing method, device, electronic device, storage medium and program product. Background Art

[0002] In the database field, query operations usually involve connecting multiple data tables with related relationships. When performing the connection operation, the uneven distribution of the data involved in the connection will cause some nodes to process more data than other nodes, which is the problem of data skew, resulting in a decrease in query performance. Summary of the invention

[0003] The embodiments of the present disclosure provide a data processing method, device, electronic device, storage medium and program product, which can solve the problem of data skew causing query performance degradation in the prior art. The technical solutions provided by the present disclosure are as follows:

[0004] According to one aspect of an embodiment of the present disclosure, a data processing method is provided, the method comprising:

[0005] Acquire query information; the query information includes at least two data tables to be queried; the at least two data tables include corresponding routing information; the routing information includes at least two first data ranges and a storage path corresponding to each first data range; the amount of data included in each first data range is not less than a first preset value and not greater than a second preset value; the storage path is used to store the address of the sorting result of the data included in the first data range;

[0006] Based on each first data range in the routing information, determining at least two ranges to be connected; the range to be connected includes at least one first data range;

[0007] Allocating the at least two ranges to be connected to at least two computing nodes, so that each computing node determines the received range to be connected, and based on the routing information, connects at least two data in each data table in the range to be connected to obtain a connection result;

[0008] Based on the connection results respectively corresponding to the computing nodes, a query result for the query information is determined.

[0009] Optionally, the connecting at least two data in the to-be-connected range of each data table based on the routing information to obtain a connection result includes:

[0010] For each data table, reading storage paths corresponding to at least two data in the to-be-connected range of the data table from the routing information;

[0011] Based on the storage path, obtaining the sorting result of at least two data in the data table in the to-be-connected range;

[0012] At least two sorting results corresponding to each data table in the range to be connected are connected to obtain a connection result corresponding to the range to be connected.

[0013] Optionally, the routing information is determined based on the following method:

[0014] Acquire a data set to be processed; based on the data set to be processed, generate a preset number of subtasks; assign the preset number of subtasks to at least one computing node; the data set to be processed includes any data table;

[0015] Determine, by each computing node, a corresponding data set to be sorted based on at least one received subtask, and sort the at least two data in the data set to be sorted based on the keys respectively corresponding to the at least two data, to obtain a corresponding first sorting result; based on the first sorting result, update storage paths respectively corresponding to at least two initial data ranges in the routing information;

[0016] Acquire the data amounts corresponding to the respective initial data ranges in the routing information after the storage path is updated, and perform at least one adjustment operation on at least one initial data range in the routing information according to the data amounts corresponding to the respective initial data ranges, so as to obtain at least two target data ranges that meet preset conditions; the preset conditions include that the data amount of the data included in each target data range is not less than a first preset value and not greater than a second preset value;

[0017] The data corresponding to each target data range are sorted by at least one computing node to obtain a second sorting result corresponding to each target data range, and the routing information is updated based on at least two second sorting results to obtain updated routing information.

[0018] Optionally, the method further comprises:

[0019] Based on the updated routing information, a global data sorting result for the data set to be processed is obtained.

[0020] Optionally, the method further comprises:

[0021] Determine at least one to-be-sorted range based on at least one target data range, and assign at least one to-be-sorted range to at least one computing node; the to-be-sorted range includes at least one target data range;

[0022] The step of sorting the data corresponding to each target data range by at least one computing node to obtain a second sorting result corresponding to each target data range includes:

[0023] For each computing node, sort at least two data in the received to-be-sorted range to obtain a corresponding third sorting result;

[0024] Based on the third sorting results respectively corresponding to the computing nodes, the second sorting results respectively corresponding to each second data range in the routing information are determined.

[0025] Optionally, the adjustment operation includes:

[0026] Determine at least one range to be adjusted from the routing information corresponding to the current adjustment operation; for each range to be adjusted, if the amount of data contained in the range to be adjusted is greater than a third preset value, divide the range to be adjusted into at least two sub-ranges;

[0027] For each range to be adjusted, if the amount of data included in the range to be adjusted is less than a fourth preset value, determining at least one adjacent data range adjacent to the range to be adjusted, and merging the range to be adjusted with the at least one adjacent data range corresponding to the range to obtain a corresponding merged range;

[0028] Based on at least one of the at least two sub-ranges and the merged range, routing information corresponding to a next adjustment operation is determined.

[0029] Optionally, the updating, based on the first sorting result, storage paths respectively corresponding to at least two initial data ranges in the routing information includes:

[0030] For each computing node, determining a result range corresponding to the first sorting result;

[0031] If the result range is within any initial data range, adding the storage location corresponding to the first sorting result to the storage path corresponding to the initial data range to which the result range belongs;

[0032] Otherwise, the first sorting result is divided based on at least one initial data range in the routing information to obtain at least two sub-results; for each sub-result, the storage location corresponding to the sub-result is added to the storage path corresponding to the initial data range to which the sub-result belongs.

[0033] Optionally, the acquiring of the data volumes corresponding to the respective initial data ranges in the routing information includes:

[0034] For each first data range, reading a storage path corresponding to the initial data range;

[0035] Based on the storage path, obtaining at least two data included in the initial data range;

[0036] Based on at least two data included in the initial data range, the amount of data corresponding to the initial data range is determined.

[0037] According to another aspect of an embodiment of the present disclosure, there is provided a data processing device, the device comprising:

[0038] A data acquisition module, used to acquire query information; the query information includes at least two data tables to be queried; the at least two data tables include corresponding routing information; the routing information includes at least two first data ranges and a storage path corresponding to each first data range; the amount of data included in each first data range is not less than a first preset value and not greater than a second preset value; the storage path is used to store the address of the sorting result of the data included in the first data range;

[0039] A to-be-connected range determination module, configured to determine at least two to-be-connected ranges based on each first data range in the routing information; the to-be-connected ranges include at least one first data range;

[0040] an allocation module, configured to allocate the at least two to-be-connected ranges to at least two computing nodes, so that each computing node determines the received to-be-connected range, and connects at least two data in each data table in the to-be-connected range based on the routing information to obtain a connection result;

[0041] The query result determination module is used to determine the query result for the query information based on the connection results corresponding to each computing node.

[0042] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of any one of the above-mentioned data processing methods when executing the program.

[0043] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data processing methods are implemented.

[0044] According to one aspect of an embodiment of the present disclosure, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned data processing methods are implemented.

[0045] The technical solution provided by the embodiments of the present disclosure has the following beneficial effects:

[0046] By determining multiple ranges to be connected based on multiple first data ranges in routing information corresponding to at least two data tables, and allocating the multiple ranges to be connected to multiple computing nodes, the query connection between at least two data tables is split into multiple subtasks, which is more conducive to distributed implementation of the query connection between at least two data tables, avoids the problem of single-point processing bottleneck caused by data skew, improves the utilization rate of computing node cluster resources, and improves the efficiency of query connection and query performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for describing the embodiments of the present disclosure are briefly introduced below.

[0048] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure;

[0049] Figure 2 A flowchart of another data processing method provided by an embodiment of the present disclosure;

[0050] Figure 3 A schematic diagram of initial data range division provided in an embodiment of the present disclosure;

[0051] Figure 4 A schematic diagram of a routing table provided in an embodiment of the present disclosure;

[0052] Figure 5 A flowchart of a data sorting method provided by an embodiment of the present disclosure;

[0053] Figure 6 A schematic diagram of a routing table adjustment process provided by an embodiment of the present disclosure;

[0054] Figure 7 A schematic diagram of the structure of a data processing device provided in an embodiment of the present disclosure;

[0055] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] The embodiments of the present disclosure are described below in conjunction with the drawings in the present disclosure. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions of the embodiments of the present disclosure.

[0057] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present disclosure refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" or "A, B" indicates that it is implemented as "A", or implemented as "B", or implemented as "A and B".

[0058] In order to make the objectives, technical solutions and advantages of the present disclosure more clear, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings.

[0059] In the specific implementation of the present disclosure, any data related to an object is involved. When the embodiments of the present disclosure are applied to specific products or technologies, the permission or consent of the object needs to be obtained, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions. In other words, if any of the above data related to an object is involved in the embodiments of the present disclosure, these data need to be obtained with the authorization and consent of the object and in compliance with the relevant laws, regulations and standards of the country and region.

[0060] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present disclosure and the technical effects produced by the technical solutions of the present disclosure. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0061] Figure 1 A flowchart of a data processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the method includes:

[0062] Step S110, obtaining query information; the query information includes at least two data tables to be queried; the at least two data tables include corresponding routing information;

[0063] The routing information includes at least two first data ranges and a storage path corresponding to each first data range; the amount of data contained in each first data range is not less than a first preset value and not greater than a second preset value; the storage path is used to store the address of the sorting result of the data contained in the first data range.

[0064] Specifically, the execution subject of the data processing method provided in the embodiment of the present disclosure may be a scheduling node;

[0065] When performing a data query, query information may be obtained, and the query information may be expressed as a SQL (Structured Query Language) statement. The query information may include at least two data tables to be queried and query conditions. For example, the query information may be:

[0066] SELECT orders.order_id, customers.customer_name

[0067] FROM orders

[0068] INNER JOIN customers ON orders.customer_id = customers.customer_id;

[0069] Among them, orders and customers represent two data tables storing order information and customer information respectively, and orders and customers can be two data tables to be queried.

[0070] The above query information will connect the orders table and the customers table, and return the query results containing order_id (order number) and customer_name (customer name) based on the match between the customer_id field in the orders table and the customer_id field in the customers table.

[0071] It should be noted that the above examples are only for better illustrating the data processing method provided by the embodiment of the present disclosure, and do not constitute a limitation on the data processing method (such as query conditions and connection types, etc.) provided by the embodiment of the present disclosure.

[0072] There is an association relationship between the at least two data tables to be queried, that is, the at least two data tables include a same connection field (such as customer_id in the above example), and the at least two data tables can be connected through this connection field.

[0073] At least two data tables may correspond to one piece of routing information, wherein the routing information may include at least two first data ranges, and the first data range in the routing information may be a range formed by connection fields corresponding to the at least two data tables.

[0074] The amount of data contained in each first data range in the routing information is not less than the first preset value and not greater than the second preset value, wherein the first preset value is not greater than the second preset value. When the difference between the first preset value and the second preset value is small, the amount of data contained in each first data range fluctuates within a smaller range, that is, the amount of data contained in each first data range is slightly different, and the data contained in each first data range is evenly distributed. Among them, the first data range may include a key range, and the amount of data corresponding to the first data range may be expressed as the number of keys included in the first data range, or the storage size of multiple data with keys distributed within the first data range.

[0075] Optionally, when the first preset value is equal to the second preset value, the data volume of each first data range in the routing information is the first preset value or the second preset value, that is, the data volume of each first data range is of a fixed size. For example, when the data volume is expressed as storage size, the first preset value and the second preset value can both be set to 1G.

[0076] The routing information may also include a storage path corresponding to each first data range, wherein the storage path is used to store the address of the sorting result of the data included in the first data range corresponding to the storage path.

[0077] Optionally, the routing information may be represented as a routing table, which may include multiple rows, each row including a corresponding first data range and its corresponding storage path.

[0078] The routing information corresponding to at least two data tables can be understood as, for each data table, using the connection field as a key, arranging multiple data in the data table in order based on the key, and dividing the multiple data in the table into multiple first data ranges with evenly distributed data volume based on the corresponding key, so as to facilitate subsequent connection operations based on the routing information.

[0079] In addition, when the scheduling node obtains at least two data tables to be queried, it can construct corresponding routing information in the process of scanning each data table. The process of constructing the routing information will be described in detail below.

[0080] Step S120: determining at least two ranges to be connected based on the first data ranges in the routing information; the ranges to be connected include at least one first data range.

[0081] Specifically, after obtaining routing information corresponding to at least two data tables, at least two to-be-connected ranges may be determined based on each first data range in the routing information.

[0082] For example, each first data range may be regarded as a range to be connected; or at least one first data range may be regarded as a range to be connected. When the range to be connected includes at least two first data ranges, the at least two first data ranges are arranged consecutively.

[0083] Step S130, allocating at least two ranges to be connected to at least two computing nodes, so that each computing node determines the received range to be connected, and based on the routing information, connects at least two data in each data table in the range to be connected to obtain a connection result.

[0084] Specifically, after determining at least two ranges to be connected, the at least two ranges to be connected may be allocated to at least two computing nodes.

[0085] Optionally, the scheduling node can obtain the computing resources corresponding to each computing node, and allocate at least two to-be-connected ranges to at least two computing nodes in a balanced manner according to the computing resources corresponding to each computing node. For example, the amount of data contained in each first data range is balanced. The more first data ranges contained in the to-be-connected range, the larger the amount of data contained in the to-be-connected range. Therefore, according to the amount of data contained in each to-be-connected range and the computing resources corresponding to each computing node, the to-be-connected range can be allocated to a matching computing node to achieve balanced load on at least two computing nodes, thereby improving resource utilization and facilitating improving data processing efficiency.

[0086] Optionally, when each first data range is taken as a range to be connected, the data connection for a range to be connected can be taken as a subtask, and then the connection between at least two data tables can be split into multiple subtasks, and the data volume corresponding to each subtask is balanced.

[0087] The scheduling node can evenly distribute multiple subtasks to each computing node. Each computing node can execute subtasks according to its own computing resources. The more computing resources a computing node has, the more subtasks it can execute; the fewer computing resources a computing node has, the fewer subtasks it can execute. After executing the above steps, it is also possible to detect whether there are unfinished subtasks. If unfinished subtasks are detected, the unfinished subtasks can be reallocated in the above manner until all subtasks are completed.

[0088] The above allocation method can achieve balanced load on multiple computing nodes, and the scheduling node does not need to count and collect the real-time computing resources of each computing node, nor does it need to adaptively allocate the computing resources corresponding to each computing node. This simplifies the process of data range allocation and improves the efficiency of data processing.

[0089] The amount of data contained in each first data range is balanced, and the amount of data contained in each first data range is within the controllable range of each computing node (that is, a single computing node can quickly complete the data processing of a single first data range). The data connection of each range to be connected (that is, the first data range) is used as a processing unit, which is more conducive to the distributed implementation of query connections between at least two data tables, improves the utilization rate of computing node cluster resources, and avoids the problem of single-point processing bottlenecks caused by data skew.

[0090] For each computing node, the computing node can receive at least one range to be connected, and based on the routing information, obtain multiple data of each data table in the at least one received range to be connected, connect the multiple data corresponding to each data table, and obtain the connection results of at least two data tables for the range to be connected.

[0091] Step S140: determining a query result for the query information based on the connection results corresponding to each computing node.

[0092] Specifically, each range to be connected includes at least one first data range, and the order between the first data ranges is determined, so the order between the ranges to be connected is also determined. After obtaining the connection results generated by each computing node, multiple connection results can be spliced ​​based on the order between the ranges to be connected corresponding to each computing node, to obtain a connection table after at least two data tables are connected, and the corresponding query result is determined from the connection table based on the query condition in the query information.

[0093] The data processing method provided by the embodiment of the present disclosure determines multiple ranges to be connected based on multiple first data ranges in the routing information corresponding to at least two data tables, and allocates the multiple ranges to be connected to multiple computing nodes, thereby splitting the query connection between at least two data tables into multiple subtasks, which is more conducive to distributed implementation of the query connection between at least two data tables, avoids the problem of single-point processing bottleneck caused by data skew, improves the utilization rate of computing node cluster resources, and improves the efficiency of query connection and query performance.

[0094] As an optional embodiment, based on the routing information, at least two data in the to-be-connected range of each data table are connected to obtain a connection result, including:

[0095] For each data table, reading storage paths corresponding to at least two data in the data table in the to-be-connected range from routing information;

[0096] Based on the storage path, obtaining the sorting result of at least two data in the data table in the to-be-connected range;

[0097] At least two sorting results corresponding to each data table in the range to be connected are connected to obtain a connection result corresponding to the range to be connected.

[0098] Specifically, for each computing node, the computing node can receive the range to be connected assigned by the scheduling node. For each data table, the computing node can obtain the storage path for the data table corresponding to each first data range from the routing information based on at least one first data range in the range to be connected, and based on the storage path corresponding to each first data range, obtain the sorting result of the data contained in each first data range to obtain the sorting result of multiple data of the data table in the range to be connected.

[0099] The sorting result corresponding to the range to be connected not only includes the multiple data in the range to be connected in the data table, but also in the sorting result, the multiple data in the range to be connected are a sequence composed of orderly arrangement according to the keys of each data.

[0100] When at least two data tables share a routing information, the data in each data table uses the connection field as the corresponding key, that is, the key of the data in each data table is the same. After obtaining multiple ordered data in each data table in the range to be connected, the computing node can connect multiple data with the same key in each data table based on the key of the data, and then obtain the connection result for the range to be connected.

[0101] For example, at least two data tables include a first data table and a second data table. For a computing node, after determining the range to be connected, the computing node can obtain multiple first data in the first data table in the range to be connected and multiple second data in the second data table in the range to be connected based on the routing information. Since the first data table and the second data table share the same routing information, that is, the key of the first data in the first data table is the same as the key of the second data in the second data table, the computing node can connect the first data with the second data with the same key, and then obtain the connection result corresponding to the range to be connected.

[0102] Optionally, the connection types between at least two data tables may include nested loop join, merge sort join, hash join, etc. When Nest loop join is used, the inner table and the outer table correspond to the same routing information. Based on the routing table, the data in the inner table can be divided into multiple data blocks, and each data in the outer table is processed according to the routing table sub-item (i.e., the first data range) to reduce the number of scans for each data; the outer table is split into multiple data blocks based on the routing table, and a connection operation is performed for each first data range in the routing information. When Merge sort join is used, the same routing information is applied to the two connected tables respectively, and the data in the two tables are divided into different items based on the routing information, and the connection is performed in parallel for each first data range in the routing information. When Hash join is used, the routing table can be constructed as a hash table. The outer table is divided into multiple tasks, and a connection operation is performed with the corresponding items of the hash table.

[0103] In the disclosed embodiment, the data range of each data table is evenly divided through routing information, so that multiple computing nodes can be reasonably allocated to achieve balanced loads on multiple computing nodes, avoiding the problem of single-point processing bottlenecks caused by data skew;

[0104] Furthermore, by using routing information to split the query connection between at least two data tables into data connections between multiple data ranges to be connected, it is more conducive to distributed implementation of query connection and fully utilizes the computing node cluster resources. When multiple computing nodes execute the corresponding data connections in the range to be connected in parallel, the efficiency of the query connection is greatly improved, thereby improving the query performance.

[0105] In addition, the routing information also includes the storage address of the sorting result corresponding to each first data range. That is to say, the routing information has pre-sorted the data in each data table, reducing the sorting requirements in the connection operation and the number of times the data is scanned in the connection operation, so that the computing nodes can concentrate the available resources to perform the connection operation, further improving the efficiency of the connection operation, thereby improving the efficiency and performance of the query.

[0106] As an optional embodiment, the routing information is determined based on the following method:

[0107] Acquire a data set to be processed; generate a preset number of subtasks based on the data set to be processed; assign the preset number of subtasks to at least one computing node; the data set to be processed includes any data table;

[0108] Determine, by each computing node, a corresponding data set to be sorted based on at least one received subtask, and sort at least two data in the data set to be sorted based on keys corresponding to at least two data in the data set to be sorted, to obtain a corresponding first sorting result; based on the first sorting result, update storage paths corresponding to at least two initial data ranges in the routing information;

[0109] Acquire the data amounts corresponding to the respective initial data ranges in the routing information after the storage path is updated, and perform at least one adjustment operation on at least one initial data range in the routing information according to the data amounts corresponding to the respective initial data ranges, so as to obtain at least two target data ranges that meet preset conditions; the preset conditions include that the data amount of the data included in each target data range is not less than a first preset value and not greater than a second preset value;

[0110] The data corresponding to each target data range are sorted by at least one computing node to obtain a second sorting result corresponding to each target data range, and the routing information is updated based on at least two second sorting results to obtain updated routing information.

[0111] Specifically, the process of constructing the routing information may be performed by the scheduling node. To construct the routing information, a data set to be processed may be first obtained, wherein in the application scenario of data query, the data set to be processed may include any data table.

[0112] Before constructing the routing information, an initial routing information may be pre-set, and the routing information may include multiple initial data ranges and storage paths corresponding to each initial data range, wherein the initial data ranges are arranged in order and there is no overlap between the initial data ranges. For example, the data range where the key of all data in the data set to be processed is located may be obtained, and the data range may be divided into multiple initial data ranges. The storage path corresponding to each initial data range in the initial routing information may be set to empty.

[0113] After obtaining the data set to be processed, a preset number of subtasks may be generated based on the data set to be processed, wherein the data contained in each subtask does not overlap. For example, the data set to be processed may be segmented, and the segmented sub-datasets are used as a subtask.

[0114] Optionally, in other application scenarios, the data set to be processed may also include multiple data obtained from different data sources, for example, the data set to be processed may include multiple data files. When the data set to be processed includes multiple data files, one data file may be segmented to obtain multiple subtasks, one data file may be used as a subtask, or one data file and at least part or all of another data file may be used as a subtask. The embodiment of the present application does not limit the method for determining the subtasks.

[0115] After obtaining a preset number of subtasks, multiple subtasks may be assigned to at least one computing node, wherein the multiple subtasks may be evenly assigned or randomly assigned, which is not limited in the embodiments of the present disclosure.

[0116] For each computing node, the computing node receives at least one subtask, and determines the corresponding data set to be sorted based on the received at least one subtask, wherein the data set to be sorted may include all data included in each subtask received by the computing node. The computing node may sort the multiple data in the data set to be sorted based on the keys (i.e., keys) corresponding to the multiple data in the data set to be sorted, and obtain the corresponding first sorting result, that is, the first sorting result may include the multiple ordered data in the data set to be sorted.

[0117] After the computing node obtains the first sorting result, multiple data in the first sorting result are arranged in order, and the minimum and maximum values ​​of the first sorting result can be determined, and each initial data range is also determined. The computing node can divide the first sorting result into at least one initial data range in the routing information, and for each initial data range, the storage address of the intermediate sorting result of multiple data in the initial data range can be updated to the storage path corresponding to the initial data range.

[0118] After performing the above operations, each computing node can divide multiple data in the data set to be processed into multiple initial data ranges in the routing information. For each initial data range in the updated routing information, the scheduling node can obtain the data volume of the data contained in the initial data range based on the storage path corresponding to the initial data range. The scheduling node can perform at least one adjustment operation on at least one initial data range according to the data volume corresponding to each initial data range, until multiple data ranges of the routing information meet the preset conditions, and use the multiple data ranges in the routing information that meet the preset conditions as multiple target data ranges. The specific process of the adjustment operation will be described in detail below.

[0119] The preset condition may include that the amount of data included in each target data range is not less than a first preset value and not greater than a second preset value, wherein the first preset value is not greater than the second preset value. When the difference between the first preset value and the second preset value is small, the amount of data included in each target data range fluctuates within a small range, that is, the amount of data included in each target data range is slightly different, and the data included in each target data range is evenly distributed.

[0120] Optionally, when the first preset value is equal to the second preset value, the amount of data contained in each first data range in the routing information is the first preset value or the second preset value, that is, the amount of data contained in each target data range is the same.

[0121] The scheduling node performs at least one adjustment operation on the routing information so that the data amounts of the data included in the multiple target data ranges in the routing information are uniform.

[0122] At this time, the storage path corresponding to each target data range may include storage addresses corresponding to multiple sub-results in the target data range, and the multiple data in each sub-result are arranged in order, but the order between the sub-results is uncertain. Therefore, the scheduling node can use the sorting of the data included in each target data range as a new sub-task, and continue to assign the new sub-task to multiple computing nodes, so that the multiple computing nodes can sort the data included in each target data range, obtain the second sorting result corresponding to each target data range, and update the second sorting result to the storage path corresponding to the corresponding target data range, so as to obtain the final updated routing information.

[0123] At this time, the routing information may include multiple target data ranges and a storage path corresponding to each target data range, where the storage path is used to store the storage address of the sorting result of the multiple data included in the corresponding target data range.

[0124] So far, in the process of constructing routing information, the data set to be processed is divided into multiple target data ranges with evenly distributed data volumes, and the sorting of multiple data in each target data range is completed.

[0125] When it is necessary to obtain the global sorting result of the data set to be processed, the sorting results corresponding to each target data range can be spliced ​​based on the sorting results corresponding to each target data range in the updated routing information and the order between the target data ranges, so as to obtain the global data sorting result for the data set to be processed.

[0126] Compared with the existing data sorting method which first needs to preprocess the data to obtain statistical information of data distribution and then sort the data, which increases the time of data sorting and the consumption of computing resources, the data processing method provided by the embodiment of the present disclosure can automatically sense the data distribution situation for a large amount of data with unknown data distribution, without the need to obtain data distribution information through preprocessing, and can complete global data sorting in one data processing, thereby saving computing resources and improving data sorting efficiency.

[0127] Furthermore, in the embodiment of the present disclosure, data is evenly divided into multiple data ranges, and multiple computing nodes are reasonably allocated to achieve balanced loads on the multiple computing nodes, thereby improving the utilization rate of computing node cluster resources.

[0128] In addition, by allocating multiple data ranges to multiple computing nodes and performing global sorting of data in a distributed manner, processing power can be improved by vertical expansion of computing nodes and / or by horizontal expansion of computing nodes, thereby further improving the efficiency of data sorting.

[0129] As an optional embodiment, based on the first sorting result, the storage paths corresponding to at least two initial data ranges in the routing information are updated, including:

[0130] For each computing node, determine a result range corresponding to the first sorting result;

[0131] If the result range is within any of the initial data ranges, then the storage address corresponding to the first sort result is added to the storage path corresponding to the initial data range to which the result range belongs;

[0132] Otherwise, the first sorting result is divided based on at least one initial data range in the routing information to obtain at least two sub-results; for each sub-result, a storage address corresponding to the sub-result is added to a storage path corresponding to the initial data range to which the sub-result belongs.

[0133] Specifically, for each computing node, after generating the first sorting result, the computing node may determine a result range corresponding to the first sorting result, and the result range may be a range where the keys of multiple data in the first sorting result are located. If the result range is located in any initial data range in the routing information, the storage address corresponding to the first sorting result may be updated to the storage path corresponding to the initial data range.

[0134] Otherwise, the first sorting result is divided based on at least one initial data range to obtain at least two sub-results, wherein the sub-result range of each sub-result is within an initial data range. For each sub-result, the storage address corresponding to the sub-result is updated to the storage path corresponding to the initial data range to which the sub-result belongs.

[0135] In the routing information after the storage path is updated, the storage path corresponding to each initial data range includes a storage address corresponding to at least one intermediate sorting result.

[0136] As an optional embodiment, the method further includes:

[0137] Determine at least one to-be-sorted range based on at least one target data range, and assign at least one to-be-sorted range to at least one computing node; the to-be-sorted range includes at least one target data range;

[0138] Sorting the data corresponding to each target data range by at least one computing node to obtain a second sorting result corresponding to each target data range, including:

[0139] For each computing node, sort at least two data in the received to-be-sorted range to obtain a corresponding third sorting result;

[0140] Based on the third sorting results respectively corresponding to the computing nodes, the second sorting results respectively corresponding to each second data range in the routing information are determined.

[0141] Specifically, after obtaining multiple target ranges, at least one range to be sorted can be determined based on at least one target data range. The range to be sorted can include at least one target data range. For example, each target data range can be used as a range to be sorted; or at least one target data range can be used as a range to be sorted. When the range to be sorted includes at least two target data ranges, the at least two target data ranges are arranged continuously.

[0142] Optionally, if the storage path corresponding to the target data range stores an intermediate sorting result, and multiple data in a single sorting result are already sorted, then the multiple data in the target data range do not need to be sorted again, and the target data range may not be further allocated to the computing node for sorting.

[0143] The scheduling node allocates at least one to-be-sorted range to at least one computing node. The allocation method of the to-be-sorted range can refer to the description corresponding to the to-be-connected range above, which will not be repeated here.

[0144] For each computing node, the computing node may determine the received range to be sorted, and sort the multiple data in the range to be sorted to obtain a corresponding third sorting result.

[0145] Optionally, if the storage path corresponding to the target data range stores multiple intermediate sorting results, and there is no overlap between the multiple intermediate sorting results, the target data range is used as the first target data range. If the range to be connected includes the first target data range, and the ranges of multiple intermediate sorting results in the first target data range do not overlap, the order between the multiple intermediate sorting results can be determined, and each intermediate result is already sorted, then the computing node can splice the multiple intermediate sorting results in the first target data range to obtain the sorting result for the first target data range.

[0146] Optionally, if the storage path corresponding to the target data range stores multiple intermediate sorting results, and the ranges of at least two intermediate sorting results overlap, the target data range is used as the second target data range. If the range to be connected includes the second target data range, for multiple intermediate sorting results with overlapping ranges, the computing node can determine the key corresponding to each data in the multiple intermediate sorting results, and re-sort the multiple data based on the keys corresponding to the respective data, to generate a global sorting result for the multiple intermediate sorting results.

[0147] It should be noted that, when the range to be connected includes the first target data range and the second target data range, the above two methods can be combined to generate a corresponding sorting result.

[0148] In the disclosed embodiment, when a computing node sorts multiple data included in a target data range, the computing node can use previously sorted intermediate sorting results to directly merge a portion of the intermediate sorting results, thereby simplifying the sorting operation and improving the efficiency of data sorting.

[0149] The third sorting result is for the range to be connected. When the range to be connected includes a target data range, the storage address corresponding to the third sorting result can be updated to the storage path of the corresponding target data range; when the range to be connected includes multiple target data ranges, the third sorting result can be divided to obtain sub-results corresponding to each target data range, and the storage address of the sub-result corresponding to each target data range can be updated to the storage path of the target data range.

[0150] After the routing information is updated, for each target data range, the second sorting result corresponding to the target data range may be the sorting result stored in the storage path of the target data range.

[0151] As an optional embodiment, obtaining the data amounts corresponding to each initial data range in the routing information includes:

[0152] For each initial data range, read the storage path corresponding to the initial data range;

[0153] Based on the storage path, obtaining at least two data included in the initial data range;

[0154] Based on at least two data included in the initial data range, the amount of data corresponding to the initial data range is determined.

[0155] Specifically, for each initial data range, the updated storage path of the initial data range may be read, and based on the storage path, a plurality of data included in the initial data range may be acquired, thereby obtaining the data volume of the data included in the initial data range.

[0156] Optionally, the routing information may be represented as a routing table, which may include multiple rows. The amount of data included in the data range of each row may be used as a field, and the obtained amount of data may be stored in a field corresponding to the amount of data in each row.

[0157] As an optional embodiment, the adjustment operation includes:

[0158] Determine at least one range to be adjusted from the routing information corresponding to the current adjustment operation;

[0159] For each range to be adjusted, if the amount of data included in the range to be adjusted is greater than a third preset value, dividing the range to be adjusted into at least two sub-ranges;

[0160] For each range to be adjusted, if the amount of data included in the range to be adjusted is less than a fourth preset value, determining at least one adjacent data range adjacent to the range to be adjusted, merging the range to be adjusted with the at least one adjacent data range corresponding to the range to obtain a corresponding merged range;

[0161] Based on at least one of the at least two sub-ranges and the merged range, routing information corresponding to a next adjustment operation is determined.

[0162] Specifically, the routing information may include multiple initial data ranges. In order to achieve uniform division of the multiple data ranges in the routing information, for the current adjustment operation, the initial data range with a data amount greater than a preset maximum value or less than a preset minimum value can be used as a range to be adjusted according to the data amount corresponding to each initial data range, to obtain at least one range to be adjusted.

[0163] For each range to be adjusted, if the amount of data corresponding to the range to be adjusted is greater than a third preset value, the range to be adjusted may be divided into at least two sub-ranges.

[0164] Optionally, the range to be adjusted can be divided into multiple sub-ranges so that the data volume corresponding to each sub-range is between the first preset value and the second preset value; the range to be adjusted can also be divided into two equal parts once, and then the divided sub-ranges can be further divided into two equal parts until the data volume corresponding to each sub-range is between the first preset value and the second preset value.

[0165] For each range to be adjusted, if the data amount corresponding to the range to be adjusted is less than the fourth preset value, at least one adjacent data range adjacent to the range to be adjusted can be determined, and the range to be adjusted and its corresponding at least one adjacent data range can be merged to obtain a corresponding merged range.

[0166] Optionally, at least one adjacent data range adjacent to the range to be adjusted may include at least one of an initial data range in the routing information and a sub-range obtained by splitting the initial data range, and the data volume corresponding to each adjacent data range is smaller than the first preset value.

[0167] The above steps are performed for each range to be adjusted, so that a data range with a larger data volume is split into multiple sub-ranges, and multiple continuous data ranges with smaller data volumes are merged into a merged range. The obtained multiple sub-ranges and / or merged ranges are used as new routing information, and the next adjustment operation is performed based on the new routing information.

[0168] The above adjustment operation is repeatedly performed until the data volume corresponding to each data range in the routing information satisfies the preset condition of being not less than the first preset value and not greater than the second preset value.

[0169] In addition, the scheduling node can also update the data volume corresponding to each initial data range in the process of adjusting multiple initial data ranges in the routing information. For example, when the range to be adjusted is divided into at least two sub-ranges, the data volume corresponding to each sub-range can be updated; when the range to be adjusted is merged with at least one adjacent data range corresponding to it, the data volume corresponding to the merged range can be updated.

[0170] Optionally, each computing node may periodically access the routing information. If the computing node detects that the routing information has been updated (i.e., the data range in the routing information has been adjusted), the computing node may divide the generated sorting result into the updated data range in the routing information to update the data volume corresponding to each data range. In this case, the adjustment of the data range of the routing information by the scheduling node and the adjustment of the data volume corresponding to the data range by multiple computing nodes may be performed asynchronously.

[0171] Optionally, the scheduling node may update the data volume corresponding to multiple data ranges in the routing information as a new subtask, and assign the subtask to multiple computing nodes so that each computing node updates the data volume corresponding to multiple data ranges in the routing information based on the received subtask.

[0172] The data processing method provided by the embodiment of the present disclosure is described below with reference to an example. Figure 2 A flowchart of a data processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the method includes:

[0173] First, you can obtain the data set to be processed. The methods for obtaining the data set to be processed include inputting from an external common format prepared by the user, such as preparing the data by the user and storing it in files in formats such as CSV (Comma-Separated Values), parquet (a columnar storage format), etc.; reading data from other databases; scanning data from data interfaces of other big data platforms; reading data from different data sources, etc.

[0174] Figure 2 The data set to be processed may include multiple CSV files and data corresponding to multiple regions (data shards) in the database. Figure 2 The multiple computing nodes participating in the task execution include computing node 1, computing node 2 and computing node 3. It should be noted that Figure 2 This is only an example provided in the embodiment of the present disclosure and does not constitute a limitation on the data processing method provided in the embodiment of the present disclosure.

[0175] In this example, the routing information is represented as a routing table. Figure 3 A schematic diagram of an initial data range division provided in an embodiment of the present disclosure, such as Figure 3 As shown, the routing table includes multiple initial data ranges range1, range2, range3 and range4, where range1 corresponds to [0,5), range2 corresponds to [5,10), range3 corresponds to [10,15), and range4 corresponds to [15,20). Multiple ranges are arranged consecutively, and the end key (key end) of the previous range is the start key (key start) of the next range.

[0176] Figure 4 A schematic diagram of a routing table provided in an embodiment of the present disclosure, such as Figure 4 As shown, the routing table includes multiple rows, each row includes a data range (ie, key range) and its corresponding storage path and data volume.

[0177] The scheduling node generates multiple subtasks based on the data set to be processed, and assigns the multiple subtasks to multiple computing nodes. Each computing node can determine the data contained in at least one received subtask. Computing node 1 corresponds to range1, range2, and range3, computing node 2 corresponds to range1, range3, and range4, and computing node 3 corresponds to range2 and range4.

[0178] It should be noted that for each range, taking range1 as an example, multiple range1s represent multiple ranges within the initial data range1, and multiple range1s may be different. For example, the initial data range1 represents a continuous range of [0,5), the range1 corresponding to computing node 1 may include [1,3], and the range1 corresponding to computing node 2 may include [0,2,4].

[0179] For each computing node, the computing node determines a data set to be sorted based on at least one received subtask, and sorts the multiple data based on the keys corresponding to the multiple data in the data set to be sorted, obtains a sorting result file, and divides the sorting result file based on the multiple initial data ranges in the routing information, and updates the storage address of at least one sub-result belonging to each initial data range to the storage path corresponding to the initial data range. For example, the data set to be sorted corresponding to computing node 1 includes (13,3,7,9,1,5,11), where the multiple data in the data set to be sorted are not necessarily in order, and the sorting result file obtained after sorting may include a sequence of [1,3,5,7,9,11,13]. The sorting result is divided based on each initial data range, and [1,3] is divided as a sub-result to range1, that is, the file storage address of this sub-result is added to the storage path corresponding to range1 in the routing table. Similarly, [5,7,9] is divided into range2, and [11,13] is divided into range3, so as to update the routing table.

[0180] The scheduling node adjusts at least one initial data range based on the data volume corresponding to each initial data range in the updated routing table. For example, range3 is split into two sub-ranges [10,12) and [12,15), and [10,12) is merged with the initial range2, and [12,15) is merged with the initial range4 to obtain new data ranges [5,12) and [12,20). In other words, the final target data ranges include [0,5), [5,12) and [12,20), and the data volume corresponding to each target data range is uniform.

[0181] The scheduling node can determine multiple ranges to be sorted based on each target data range. For example, a target data range is used as a range to be sorted, and multiple ranges to be sorted are reasonably allocated to multiple computing nodes, so as to achieve balanced loads of multiple computing nodes and improve the resource utilization of the computing node cluster. At the same time, for each computing node, the data range to be sorted received by the computing node is sorted once. If the range to be sorted includes a result file, and the single result file is already sorted, the computing node does not need to sort; if the range to be sorted includes multiple result files, and the ranges corresponding to these multiple result files do not overlap, such as [5,10) and [10,12) in computing node 2, then the computing node only needs to splice the multiple result files in order without re-sorting, which simplifies the sorting operation and improves data processing efficiency.

[0182] Figure 5 A flowchart of a data sorting method provided in an embodiment of the present disclosure is provided. Figure 5 shows the interaction between a single compute node and the scheduling node, Figure 5 The given example does not constitute a limitation on the number of computing nodes. When there are multiple computing nodes, the interaction between each computing node and the scheduling node may be the same. Figure 5 The computing node can be TiDB (a type of database) in the example, and in other application scenarios, the computing node can be other types of databases.

[0183] like Figure 5 As shown, the method includes:

[0184] 1. The scheduling node generates an execution plan based on the data set to be processed;

[0185] a. Evenly divide the data set to be processed into multiple subtasks. Each subtask contains a part of the data to be processed, and there is no overlap between the data processed by each subtask.

[0186] b. Construct the initial routing table: i. Initialize the routing table by dividing the data set to be processed into multiple initial data ranges as the starting state of the routing table; ii. Persist the initial routing table;

[0187] c. Allocate subtasks to participating computing nodes for execution;

[0188] d. The scheduling node periodically monitors and collects information about the routing table, adjusts the data range in the routing table, and updates the routing table;

[0189] 2. For each computing node, obtain and execute the subtask assigned by the scheduling node; (the subtask of step 2 is to sort the multiple data in the received range to be sorted)

[0190] a. For the current subtask received by each computing node, read the specified part of the data in the current subtask in a corresponding manner according to the data processing scope and data file type described in the current subtask;

[0191] b. Encode the data in the current subtask and sort them according to the encoded key values ​​to obtain the intermediate sorting results;

[0192] c. Split the intermediate sorting results according to multiple initial data ranges in the routing table, and upload the segmented sub-results to the path storage specified by the corresponding item in the routing table. This intermediate medium (i.e. Figure 5 Temporary storage of sorting results in ) can be considered as shared storage of unlimited size, such as S3 (Simple Storage Service, a cloud storage service);

[0193] d. The computing node can periodically update the routing table: i. Periodically access the routing table to obtain the latest routing table information; ii. The computing node can split and adjust the previously processed data according to the latest routing table information. At this time, the update of the routing table information can be used as a new subtask of the computing node;

[0194] e. Check whether there are any assigned and unprocessed subtasks. If yes, get the next subtask to continue processing; otherwise, go to step 4;

[0195] 3. The scheduling node periodically monitors and collects information about the routing table, performs multiple adjustment operations on the routing table, periodically updates the scheduling routing table, and persists it;

[0196] 4. The scheduling node can determine that all subtasks of the current step are completed by receiving feedback from each participating computing node. It should be noted that the task of step 2.d.ii can be put into this step for execution; the scheduling node can also make step 2.d.ii into a subtask in this step and assign it to the computing nodes for distributed execution to improve data processing efficiency;

[0197] 5. The scheduling node traverses the routing table and divides the subtasks according to the total amount of data stored in the routing table items. For example, the size of the data included in each subtask is 1G (that is, the range to be sorted is determined);

[0198] 6. For each computing node, sort the data contained in the received subtasks, because the amount of data in each subtask is controlled here. At the same time, after the processing of the previous steps, the data of each subtask will not overlap and has been divided according to the key value range. For the sorting results generated for each computing node, the generated sorting results can be directly consumed by the subsequent business end; the sorting results can also be spliced ​​according to the range and output as the global sorting results; the sorted data can also be written back to S3 in some way.

[0199] 7. Complete the global sorting task of the data set to be processed.

[0200] Figure 6 A schematic diagram of a routing table adjustment process provided by an embodiment of the present disclosure, such as Figure 6 As shown, the initialized routing table (i.e. Figure 6 The first level of division in the above example may include multiple initial data ranges, for example, 1000. Figure 6 In the data range A, data range B, data range C and data range D, each key range (i.e. data range) corresponds to a statistical information upload file on S3. The statistical information (i.e. the amount of data contained in the data range) includes: the number of keys in this key range and / or the storage size.

[0201] On this basis, if the amount of data corresponding to a certain data range is greater than a preset value, the data range can be divided, for example Figure 6 The data range B is divided into B1, B2, B3, and B4. This can be done directly or in equal parts, or in equal parts according to the actual data, or in N equal parts after exceeding a certain preset value, until a set amount of data is obtained, such as 1G.

[0202] At the same time, the statistics of each new data range are generated and placed in a separate statistics file at the first level (for example, a file with the prefix statistic). The file naming rules include: statistic_key_size.

[0203] The scheduling node scans the files prefixed with statistic in the first layer (sorted in alphabetical order by file name), and generates distributed sorting subtasks based on the file name. If a data range contains multiple files, it must be divided into a subtask, because all files in this data range need to be sorted to ensure that all data in this data range are sorted in one node.

[0204] Since there may be a delay in updating the routing table for the participating computing nodes, after updating the new routing table, it is necessary to check whether the previously processed data is within the updated routing table range, and the statistical information of that data needs to be split accordingly and put into the corresponding item of the routing table. This operation can be placed above Figure 5 After step 4, the scheduling node schedules all computing nodes to process the data that was previously placed in the discarded item because the routing table was not updated in time.

[0205] Figure 7 A structural diagram of a data processing device provided in an embodiment of the present disclosure is shown in FIG. Figure 7 As shown, the device of this embodiment may include:

[0206] The data acquisition module 210 is used to acquire query information; the query information includes at least two data tables to be queried; the at least two data tables include corresponding routing information; the routing information includes at least two first data ranges and a storage path corresponding to each first data range; the amount of data included in each first data range is not less than a first preset value and not greater than a second preset value; the storage path is used to store the address of the sorting result of the data included in the first data range;

[0207] The to-be-connected range determining module 220 is configured to determine at least two to-be-connected ranges based on the first data ranges in the routing information; the to-be-connected ranges include at least one first data range;

[0208] The allocation module 230 is used to allocate the at least two to-be-connected ranges to at least two computing nodes, so that each computing node determines the received to-be-connected range, and connects at least two data in each data table in the to-be-connected range based on the routing information to obtain a connection result;

[0209] The query result determination module 240 is used to determine the query result for the query information based on the connection results corresponding to each computing node.

[0210] As an optional embodiment, when the computing node connects at least two data in the to-be-connected range of each data table based on the routing information to obtain a connection result, it is specifically used to:

[0211] For each data table, reading storage paths corresponding to at least two data in the to-be-connected range of the data table from the routing information;

[0212] Based on the storage path, obtaining the sorting result of at least two data in the data table in the to-be-connected range;

[0213] At least two sorting results corresponding to each data table in the range to be connected are connected to obtain a connection result corresponding to the range to be connected.

[0214] As an optional embodiment, the device further includes a routing information building module, which is used to:

[0215] Acquire a data set to be processed; based on the data set to be processed, generate a preset number of subtasks; assign the preset number of subtasks to at least one computing node; the data set to be processed includes any data table;

[0216] Determine, by each computing node, a corresponding data set to be sorted based on at least one received subtask, and sort the at least two data in the data set to be sorted based on the keys respectively corresponding to the at least two data, to obtain a corresponding first sorting result; based on the first sorting result, update storage paths respectively corresponding to at least two initial data ranges in the routing information;

[0217] Acquire the data amounts corresponding to the respective initial data ranges in the routing information after the storage path is updated, and perform at least one adjustment operation on at least one initial data range in the routing information according to the data amounts corresponding to the respective initial data ranges, so as to obtain at least two target data ranges that meet preset conditions; the preset conditions include that the data amount of the data included in each target data range is not less than a first preset value and not greater than a second preset value;

[0218] The data corresponding to each target data range are sorted by at least one computing node to obtain a second sorting result corresponding to each target data range, and the routing information is updated based on at least two second sorting results to obtain updated routing information.

[0219] As an optional embodiment, the device further includes a first sorting module, configured to:

[0220] Based on the updated routing information, a global data sorting result for the data set to be processed is obtained.

[0221] As an optional embodiment, the device further includes a second sorting module, configured to:

[0222] Determine at least one to-be-sorted range based on at least one target data range, and assign at least one to-be-sorted range to at least one computing node; the to-be-sorted range includes at least one target data range;

[0223] The step of sorting the data corresponding to each target data range by at least one computing node to obtain a second sorting result corresponding to each target data range includes:

[0224] For each computing node, sort at least two data in the received to-be-sorted range to obtain a corresponding third sorting result;

[0225] Based on the third sorting results respectively corresponding to the computing nodes, the second sorting results respectively corresponding to each second data range in the routing information are determined.

[0226] As an optional embodiment, the adjustment operation includes:

[0227] Determine at least one range to be adjusted from the routing information corresponding to the current adjustment operation; for each range to be adjusted, if the amount of data contained in the range to be adjusted is greater than a third preset value, divide the range to be adjusted into at least two sub-ranges;

[0228] For each range to be adjusted, if the amount of data included in the range to be adjusted is less than a fourth preset value, determining at least one adjacent data range adjacent to the range to be adjusted, and merging the range to be adjusted with the at least one adjacent data range corresponding to the range to obtain a corresponding merged range;

[0229] Based on at least one of the at least two sub-ranges and the merged range, routing information corresponding to a next adjustment operation is determined.

[0230] As an optional embodiment, when the routing information construction module updates the storage paths corresponding to at least two initial data ranges in the routing information based on the first sorting result, it is specifically configured to:

[0231] For each computing node, determining a result range corresponding to the first sorting result;

[0232] If the result range is within any initial data range, adding the storage location corresponding to the first sorting result to the storage path corresponding to the initial data range to which the result range belongs;

[0233] Otherwise, the first sorting result is divided based on at least one initial data range in the routing information to obtain at least two sub-results; for each sub-result, the storage location corresponding to the sub-result is added to the storage path corresponding to the initial data range to which the sub-result belongs.

[0234] As an optional embodiment, when the routing information construction module obtains the data volume corresponding to each initial data range in the routing information, it is specifically used to:

[0235] For each first data range, reading a storage path corresponding to the initial data range;

[0236] Based on the storage path, obtaining at least two data included in the initial data range;

[0237] Based on at least two data included in the initial data range, the amount of data corresponding to the initial data range is determined.

[0238] The device of the embodiment of the present disclosure can execute the method provided by the embodiment of the present disclosure, and its implementation principle is similar and has corresponding technical effects. The actions performed by each module in the device of each embodiment of the present disclosure correspond to the steps in the method of each embodiment of the present disclosure. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0239] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0240] In an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, and the processor executes the above computer program to implement the steps of the method provided by any optional embodiment of the present disclosure. Compared with the prior art, it can be achieved that: by determining multiple ranges to be connected based on multiple first data ranges in the routing information corresponding to at least two data tables, and allocating the multiple ranges to be connected to multiple computing nodes, the query connection between at least two data tables is split into multiple subtasks, which is more conducive to distributed implementation of the query connection between at least two data tables, avoiding the problem of single-point processing bottleneck caused by data skew, improving the utilization rate of computing node cluster resources, and improving the efficiency of query connection and query performance.

[0241] In an alternative embodiment, an electronic device is provided, such as Figure 8 As shown, Figure 8 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present disclosure.

[0242] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0243] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0244] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0245] The memory 4003 is used to store the computer program for executing the embodiment of the present disclosure, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0246] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc., and fixed terminals such as digital TVs, desktop computers, etc.

[0247] An embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0248] The embodiments of the present disclosure also provide a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiments when executed by a processor.

[0249] It should be understood that, although the flowchart of the embodiment of the present disclosure indicates each operation step by arrows, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present disclosure, the implementation steps in each flowchart can be executed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times. In scenarios with different execution times, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present disclosure does not limit this.

[0250] The above is only an optional implementation method for some implementation scenarios of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the scheme of the present disclosure, other similar implementation methods based on the technical ideas of the present disclosure are also within the protection scope of the embodiments of the present disclosure.

Claims

1. A data processing method, It is characterized in that include: Get query information; The query information includes at least two data tables to be queried; the at least two data tables include corresponding routing information; The routing information includes at least two first data ranges and a storage path corresponding to each first data range; the amount of data contained in each first data range is not less than a first preset value and not greater than a second preset value; The storage path is used to store the address of the sorting result of the data included in the first data range; Based on each first data range in the routing information, determining at least two ranges to be connected; the range to be connected includes at least one first data range; Allocating the at least two ranges to be connected to at least two computing nodes, so that each computing node determines the received range to be connected, and based on the routing information, connects at least two data in each data table in the range to be connected to obtain a connection result; Determine a query result for the query information based on the connection results corresponding to each computing node; The step of connecting at least two data in the to-be-connected range in each data table based on the routing information to obtain a connection result includes: For each data table, reading storage paths corresponding to at least two data in the to-be-connected range of the data table from the routing information; Based on the storage path, obtaining the sorting result of at least two data in the data table in the to-be-connected range; Connecting at least two sorting results corresponding to each data table in the to-be-connected range, to obtain a connection result corresponding to the to-be-connected range; The routing information is determined based on the following method: Acquire a data set to be processed; based on the data set to be processed, generate a preset number of subtasks; assign the preset number of subtasks to at least one computing node; the data set to be processed includes any data table; Determine, by each computing node, a corresponding data set to be sorted based on at least one received subtask, and sort the at least two data in the data set to be sorted based on the keys respectively corresponding to the at least two data, to obtain a corresponding first sorting result; based on the first sorting result, update storage paths respectively corresponding to at least two initial data ranges in the routing information; Acquire the data amounts corresponding to the respective initial data ranges in the routing information after the storage path is updated, and perform at least one adjustment operation on at least one initial data range in the routing information according to the data amounts corresponding to the respective initial data ranges, so as to obtain at least two target data ranges that meet preset conditions; the preset conditions include that the data amount of the data included in each target data range is not less than a first preset value and not greater than a second preset value; The data corresponding to each target data range are sorted by at least one computing node to obtain a second sorting result corresponding to each target data range, and the routing information is updated based on at least two second sorting results to obtain updated routing information.

2. The data processing method according to claim 1, It is characterized in that The method further comprises: Based on the updated routing information, a global data sorting result for the data set to be processed is obtained.

3. The data processing method according to claim 1, It is characterized in that The method further comprises: Determine at least one to-be-sorted range based on at least one target data range, and assign at least one to-be-sorted range to at least one computing node; the to-be-sorted range includes at least one target data range; The step of sorting the data corresponding to each target data range by at least one computing node to obtain a second sorting result corresponding to each target data range includes: For each computing node, sort at least two data in the received to-be-sorted range to obtain a corresponding third sorting result; Based on the third sorting results respectively corresponding to the computing nodes, the second sorting results respectively corresponding to each second data range in the routing information are determined.

4. The data processing method according to claim 1, It is characterized in that The adjustment operation includes: Determine at least one range to be adjusted from the routing information corresponding to the current adjustment operation; For each range to be adjusted, if the amount of data contained in the range to be adjusted is greater than a third preset value, dividing the range to be adjusted into at least two sub-ranges; For each range to be adjusted, if the amount of data included in the range to be adjusted is less than a fourth preset value, determining at least one adjacent data range adjacent to the range to be adjusted, and merging the range to be adjusted with the at least one adjacent data range corresponding to the range to obtain a corresponding merged range; Based on at least one of the at least two sub-ranges and the merged range, routing information corresponding to a next adjustment operation is determined.

5. The data processing method according to claim 1, It is characterized in that The updating of storage paths corresponding to at least two initial data ranges in the routing information based on the first sorting result includes: For each computing node, determining a result range corresponding to the first sorting result; If the result range is within any initial data range, adding the storage location corresponding to the first sorting result to the storage path corresponding to the initial data range to which the result range belongs; Otherwise, the first sorting result is divided based on at least one initial data range in the routing information to obtain at least two sub-results; for each sub-result, the storage location corresponding to the sub-result is added to the storage path corresponding to the initial data range to which the sub-result belongs.

6. The data processing method according to claim 1, It is characterized in that The data amounts corresponding to the respective initial data ranges in the routing information after the storage path is updated are obtained, including: For each initial data range, read the storage path corresponding to the initial data range; Based on the storage path, obtaining at least two data included in the initial data range; Based on at least two data included in the initial data range, the amount of data corresponding to the initial data range is determined.

7. A data processing device, It is characterized in that include: A data acquisition module, used to obtain query information; The query information includes at least two data tables to be queried; the at least two data tables include corresponding routing information; The routing information includes at least two first data ranges and a storage path corresponding to each first data range; the amount of data contained in each first data range is not less than a first preset value and not greater than a second preset value; The storage path is used to store the address of the sorting result of the data included in the first data range; A to-be-connected range determination module, configured to determine at least two to-be-connected ranges based on each first data range in the routing information; the to-be-connected ranges include at least one first data range; an allocation module, configured to allocate the at least two to-be-connected ranges to at least two computing nodes, so that each computing node determines the received to-be-connected range, and connects at least two data in each data table in the to-be-connected range based on the routing information to obtain a connection result; A query result determination module, used to determine the query result for the query information based on the connection results corresponding to each computing node; A routing information building module is used to obtain the data set to be processed; Based on the data set to be processed, generating a preset number of subtasks; Allocating the preset number of subtasks to at least one computing node; the data set to be processed includes any data table; Determine, by each computing node, a corresponding data set to be sorted based on at least one received subtask, and sort the at least two data in the data set to be sorted based on keys corresponding to the at least two data respectively, to obtain a corresponding first sorting result; Based on the first sorting result, updating storage paths respectively corresponding to at least two initial data ranges in the routing information; Acquire the data amounts corresponding to the respective initial data ranges in the routing information after the storage path is updated, and perform at least one adjustment operation on at least one initial data range in the routing information according to the data amounts corresponding to the respective initial data ranges, so as to obtain at least two target data ranges that meet preset conditions; the preset conditions include that the data amount of the data included in each target data range is not less than a first preset value and not greater than a second preset value; Sorting the data corresponding to each target data range respectively through at least one computing node to obtain a second sorting result corresponding to each target data range respectively, and updating the routing information based on at least two second sorting results to obtain updated routing information; Wherein, when the computing node connects at least two data in the to-be-connected range of each data table based on the routing information to obtain a connection result, it is specifically used to: For each data table, reading storage paths corresponding to at least two data in the to-be-connected range of the data table from the routing information; Based on the storage path, obtaining the sorting result of at least two data in the data table in the to-be-connected range; At least two sorting results corresponding to each data table in the range to be connected are connected to obtain a connection result corresponding to the range to be connected.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, It is characterized in that The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.