Data sorting method and device, electronic equipment and storage medium

Abstract

The application discloses a data sorting method and device, electronic equipment and storage medium. The method comprises the following steps: when the current environment is a distributed environment and there is a sending plan node under a sorting plan node in a plan tree, determining whether the sending mode is column sending and the column of sending and sorting is the same; if yes, merging a sending operator and a sorting operator, and setting an identification bit of the sending operator to a set value; in the execution stage, if the identification bit of the sending operator is the set value, allocating sorting space required by N-way sorting, and putting each row of data into the sorting space of the corresponding way; after the data acquisition is completed, sorting the data of the N-way sorting space and sending the sorted N-way data. When the sending mode is column sending and the column of sending and sorting is the same, the sending operator and the sorting operator are merged, and the data is sent after the N-way sorting space completes the sorting, so that the rationality of resource allocation during data sorting processing is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular to a data sorting method, device, electronic device, and storage medium. Background Art

[0002] In a distributed system, large-scale data can be distributed or sorted. For example, data can be stored in several storage nodes according to distribution rules. When querying data, the corresponding data can be queried from each storage node, the queried data can be sorted, and the sorted data can be sent to the computing node.

[0003] When sorting data in existing distributed database systems, each storage node may complete the table data fetching and sorting operations at similar times. This causes each storage node to send data to the same compute node, overloading the network for that compute node and leaving other compute nodes idle. This undermines the powerful computing power of the distributed cluster. Therefore, improving resource allocation for data sorting in a distributed environment is a pressing technical challenge. Summary of the Invention

[0004] The present invention provides a data sorting method, device, electronic device and storage medium to solve the problem of unreasonable resource allocation during data sorting in existing distributed systems.

[0005] In a first aspect, an embodiment of the present invention provides a data sorting method, comprising:

[0006] When the current environment is a distributed environment and there is a sort plan node below the send plan node in the plan tree, determine whether the sending mode is column-by-column, and the sent and sorted columns are the same, or the sent column is the leading column of the sorted column;

[0007] If so, merge the send operator and the sort operator, and set the flag of the send operator to the set value;

[0008] In the execution phase, if the flag of the sending operator is the set value, the sorting space required for N-way sorting is allocated, and each row of data is placed in the sorting space of the corresponding way;

[0009] After the data acquisition is completed, the data in the N-way sorting space is sorted and the sorted N-way data is sent.

[0010] In a second aspect, an embodiment of the present invention provides a data sorting device, comprising:

[0011] A determination module is used to determine whether the sending mode is column-based sending, and the sent and sorted columns are the same, or the sent column is the leading column of the sorted column, when the current environment is a distributed environment and there is a sorting plan node below the sending plan node in the plan tree;

[0012] a merging module, configured to merge a sending operator and a sorting operator if the sending mode is column-by-column sending and the sent and sorted columns are the same, or the sent column is a leading column of the sorted column, and set a flag of the sending operator to a set value;

[0013] The allocation module is used to allocate the sorting space required for N-way sorting and place each row of data into the sorting space of the corresponding way if the flag bit of the sending operator is a set value during the execution phase;

[0014] The sending module is used to sort the data in the N-way sorting space and send the sorted N-way data after the data acquisition is completed.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the method according to the first aspect.

[0019] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0020] The technical solution of the embodiment of the present invention combines the sending operator and the sorting operator when the sending method is column-by-column and the sent and sorted columns are the same, or the sent column is the leading column of the sorted column, and sends the data after the sorting is completed in the N-way sorting space, thereby improving the rationality of resource allocation during data sorting processing in a distributed environment.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 This is a flow chart of a data sorting method provided according to the first embodiment of the present invention;

[0024] Figure 2 This is a flow chart of a data sorting method provided according to the second embodiment of the present invention;

[0025] Figure 3 This is a structural diagram of a data sorting device provided according to a third embodiment of the present invention;

[0026] Figure 4 It is a structural diagram of an electronic device for implementing the data sorting method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first," "second," and the like in the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0029] It is understandable that before using the technical solutions disclosed in the embodiments of the present invention, the type, scope of use, and usage scenarios of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0030] The node types in the distributed database system are described as follows:

[0031] Compute nodes: These nodes are responsible for processing user requests, generating execution plans, and controlling the distribution and execution of these plans. For a given user, only one of the many compute nodes is the master compute node.

[0032] Storage nodes: Data distribution is stored in several storage nodes according to distribution rules, and the sub-plan of data access of the computing nodes is executed.

[0033] Metadata node: stores all metadata information in the entire distributed database system.

[0034] A distributed system can have multiple compute nodes, storage nodes, and metadata nodes. These nodes can be different electronic devices that can perform the same functions as compute nodes, storage nodes, and metadata nodes. These electronic devices include, but are not limited to, computers, laptops, and servers.

[0035] In a distributed system, the following explanation is given using the example of inserting into a single-machine database:

[0036] Create tables T1 and T2. In table T1, create an integer column C1 and perform hash partitioning on the data in column C1. The number of partitions can be 3. In table T2, create two integer columns D1 and D2 and perform hash partitioning on the data in column D1. The number of partitions can be 4. Group the data in table T2 by column D2 and sort by column D1. Query table T2 based on the row number of the sorted result within the group. Insert the query results into table T1. The above steps can be implemented using the following Structured Query Language (SQL):

[0037] CREATE TABLE T1(C1 INT)PARTITION BY HASH(C1)PARTITIONS 3;

[0038] CREATE TABLE T2(D1 INT,D2 INT)PARTITION BY HASH(D1)PARTITIONS 4;

[0039] EXPLAIN INSERT INTO T1 SELECT ROW_NUMBER()OVER(PARTITION BY D2 ORDERBY D1)FROM T2.

[0040] During the execution of the insert plan, multiple operators are used, as follows:

[0041] Insert: INSERT, can be used to insert data;

[0042] Send: SEND, can be used to send data;

[0043] Sorting: SORT, which can sort the received data according to the specified key value;

[0044] Analysis: AFUN, which can be used to analyze the query items of the test case;

[0045] Receive: RECV, can be used for data reception;

[0046] Projection: PRJT;

[0047] Get table data: CSCN;

[0048] Select PARALLEL for the partition table subtable.

[0049] Since AFUN cannot perform sorting internally, a new SORT operator is needed to sort the data to be processed by AFUN according to the D2 and D1 columns. This is then passed to the upper-level AFUN for execution to facilitate the calculation of the analysis function value for each group.

[0050] To fully utilize the machine performance of the distributed cluster, SORT / CSCN is executed on the storage nodes. The SEND operator then sends the data to different compute nodes based on the hash values ​​of D2 and D1. The RECV operator is executed by each compute node. Since the data comes from different storage nodes, the order cannot be guaranteed. Therefore, RECV itself needs to perform a merge sort to ensure the order before passing it to the upper layer for processing. SEND / RECV can also be called ESEND / ERECV, etc., for sending and receiving data, respectively. The names are not limited here. AFUN calculations are also performed on the compute nodes. Since the D2 and D1 values ​​of each compute node are different, the results of each compute node do not affect each other, so they can be calculated concurrently. After the calculation is completed, the SEND operator sends the AFUN results to the target node.

[0051] Because the original data is evenly distributed according to column D1, the data volume and data characteristics of each data storage node are basically the same, and the time it takes for each data storage node to complete CSCN and SORT is also roughly the same. At this time, if data is distributed according to the values ​​of columns D2 and D1, each data storage node may send data to the same compute node, causing conflicts and placing an excessive network burden on the compute node. At this time, other compute nodes are basically idle, and the powerful computing power of the distributed cluster is not fully utilized.

[0052] To solve the above problem, the SEND and SORT operations can be combined. The system allocates N SORTs according to the total number N of computing nodes to be sent. When the underlying CSCN receives data, it calculates which computing node it will be sent to and which SORT it will be placed in according to the values ​​of the D2 and D1 columns. After all the data is obtained, all the SORTs are sorted and processed in turn. After the sorting is completed, SEND is started. The data of each SORT is sent in turn. Before sending the data, a check message is sent first. The check message is used to verify whether the receiving end can process the current data. If the receiving end cannot process more data because other storage nodes have sent too much data and are processing it, the next SORT is sent. In this way, ideally, each computing node is receiving and processing data concurrently, which can improve performance.

[0053] Example 1

[0054] Figure 1 This is a flowchart of a data sorting method provided according to the first embodiment of the present invention. This embodiment is applicable to the situation of sorting data in a distributed environment. The method can be executed by a data sorting device, which can be implemented in the form of software and / or hardware and integrated into an electronic device. Furthermore, the electronic device includes but is not limited to: computers, laptops, servers, etc. The electronic device can be a storage node. If the total number of computing nodes to which a storage node is to send is N, the sorting operation in the storage node can be divided into N paths, and the N-path sorting operations correspond to N computing nodes respectively.

[0055] like Figure 1 As shown, the method includes:

[0056] S110. When the current environment is a distributed environment and there is a sorting plan node below the sending plan node in the plan tree, determine whether the sending mode is column-by-column sending, and the sent and sorted columns are the same, or the sent column is a leading column of the sorted columns.

[0057] A distributed environment refers to industry-standard software technologies used to establish and manage computing and data exchange within distributed systems. It can be implemented as an abstraction layer between existing hardware, operating systems, networks, and distributed applications. In a distributed environment, multiple electronic devices can be interconnected through a network and communicate and collaborate via messages.

[0058] When a user enters a query statement in the database to access data, the coordination server in the database can parse the query statement and generate a corresponding plan tree, which can contain multiple plan subtrees.

[0059] There is no limitation on the method for determining whether the current environment is a distributed environment. For example, it can be determined by configuration parameters in a database configuration file.

[0060] There is no limitation on the method for determining whether there is a sorting plan node below the sending plan node in the plan tree. For example, the nodes in the plan tree can be traversed from the root node to the child nodes. If the traversal result shows that the sending plan node appears first and then the sorting plan node appears, it indicates that there is a sorting plan node below the sending plan node in the plan tree.

[0061] There is no limitation on determining whether the sending method is to send by column. For example, it can be determined whether the sending method is to send by key value, where each key value can identify a column of data in the table. When the key value is determined, it can be determined which column of data needs to be sent. If the sending method is to send by key value, it can be determined that the sending method is to send by column.

[0062] There is no limitation on the method for determining that the columns to be sent and sorted are the same. For example, it can be determined whether the key values ​​of the sending operator and the sorting operator are consistent. If so, it indicates that the columns to be sent and sorted are the same.

[0063] The method of determining whether the sent column is the leading column of the sorted columns is not limited. For example, if the sorted columns are C1, C2, and C3, and the sent column is C1, then the sent column is determined to be the leading column of the sorted columns.

[0064] It should be noted that sorting can be performed on the storage node. After the sorting is completed, the sorted data is sent to the computing node, which performs subsequent operations.

[0065] If the current environment is a distributed environment and there is a sort plan node below the send plan node in the plan tree, if the sending method is column-by-column and the sent and sorted columns are the same, or the sent column is the leading column of the sorted column, it indicates that the current send operator and sort operator can process the same data. The two operators can be merged without affecting the data processing process.

[0066] S120: If yes, merge the send operator and the sort operator, and set the flag of the send operator to a set value.

[0067] When the send method is column-by-column and the send and sort columns are the same, or the send column is the leading column of the sort column, there are no restrictions on how to combine the send operator and the sort operator. As long as the send operator and the sort operator can be combined, the method can be used. For example, in the plan tree, remove the sort operator below the send operator and implement the sorting function that the sort operator can perform within the send operator.

[0068] The flag can identify whether the send operator and the sort operator need to be merged, and the flag can be set according to actual needs. When the send operator and the sort operator need to be merged, the flag of the send operator can be set to a set value, wherein the set value can be a value set according to actual needs. The present invention does not specifically limit the flag and set value of the send operator, as long as it can be determined whether the send operator and the sort operator need to be merged through the flag and set value of the send operator. For example, the flag of the send operator is set to f1 and the value is set to 1. When f1=1, it indicates that the send operator and the sort operator need to be merged; otherwise, the send operator does not need to be merged with the sort operator.

[0069] In one embodiment, the flag of the send operator can be set to a set value, and then the send operator and the sort operator can be merged based on the set value. For example, assuming the flag of the send operator is f1 and the set value is 1, f1 is first set to the set value, and then the value of f1 is used to determine whether the send operator and the sort operator need to be merged. If f1 = 1, it indicates that the send operator and the sort operator need to be merged, and the send operator and the sort operator can be merged. Otherwise, the send operator does not need to be merged with the sort operator.

[0070] The present invention does not impose any specific restrictions on the order of merging the send operator and the sort operator and setting the flag of the send operator to the set value. The send operator and the sort operator may be merged first, and then the flag of the send operator is set to the set value; the flag of the send operator may be set to the set value first, and then the send operator and the sort operator are merged; or both may be performed simultaneously.

[0071] S130. In the execution phase, if the flag of the sending operator is a set value, the sorting space required for N-way sorting is allocated, and each row of data is placed in the sorting space of the corresponding way.

[0072] During the execution phase, if the flag of the send operator is the set value, it indicates that the send operator and the sort operator process the same data, and the send operator and the sort operator have been merged, and the sorting function of the sort operator can be implemented inside the send operator.

[0073] In an embodiment of the present invention, when the sending operator and the sorting operation are combined, each storage node in the distributed environment can divide the sorting operation into multiple paths according to the number of computing nodes to be sent. If the total number of computing nodes to be sent by a storage node is N, the sorting operation in the storage node can be divided into N paths, and then the corresponding N-path sorting operations are allocated with N-path sorting spaces, so that each path sorting space has a path sorting operation. After the sorting operation of each path is completed, the sorted data is sent to the computing node corresponding to the path. Each path sorting operation corresponds to a unique sorting space, and the computing node sent after the sorting operation is completed is uniquely corresponding to the path sorting operation. The value of N can be determined according to the actual application needs.

[0074] There is no limitation on the method of allocating the sorting space required for N-way sorting. For example, the size of the sorting space can be set according to actual application needs, and the same size of sorting space can be set for each sorting path, so that when the sorting operation is allocated to multiple paths, the maximum number of sorting operations that can be carried by each sorting space is the same.

[0075] There is no limitation on how each row of data is placed in the sorting space of the corresponding path. For example, each row in the column to be sent can be numbered, and the data in that column can be sent to both compute node 1 and compute node 2. Compute node 1 corresponds to a unique sorting operation, and after the sorting operation is completed, the sorted data is sent to compute node 1. This sorting operation has a corresponding unique sorting space, that is, compute node 1 has a corresponding unique sorting space, and similarly, compute node 2 has a corresponding unique sorting space. If odd-numbered rows are set to be sent to compute node 1, then the data corresponding to the odd-numbered rows can be sent to the sorting space of the corresponding path of compute node 1. If even-numbered rows are set to be sent to compute node 2, then the data corresponding to the even-numbered rows can be sent to the sorting space of the corresponding path of compute node 2.

[0076] After each row of data that needs to be sorted and sent is placed in the sorting space of the corresponding way, the sorting operation can be performed in the sorting space of the corresponding way.

[0077] S140 , after the data acquisition is completed, sort the data in the N-way sorting space and send the sorted N-way data.

[0078] After data acquisition is completed, the data in the N-way sorting space is sorted and the sorted N-way data is sent. It can be understood that when all the N-way sorting spaces have received the data to be sorted, the data to be sorted is sorted in each sorting space, and the sorted data is sent to the computing node corresponding to the path.

[0079] There is no limitation on the method of determining whether data acquisition is complete. For example, all acquired data can be queried in the N-way sorting space and compared with all data to be sorted. If they are consistent, it indicates that data acquisition is complete.

[0080] There is no limitation on the method of sorting the data in the N-way sorting space. For example, according to actual needs, the ways corresponding to the N-way sorting space can be arranged in sequence from the 1st way to the Nth way, and the data in the sorting space corresponding to the way can be sorted in sequence starting from the 1st way until all the data in the N-way sorting space is sorted.

[0081] There is no limitation on the method of sending the sorted N-way data. For example, each way of sorting can be sent in sequence to the computing node corresponding to the sorting space of the way, as follows:

[0082] Before sending each data path, you can first determine whether the data path has been sent. If not, you can first send a request message to the corresponding computing node. The request message is used to verify whether the computing node can receive and process the data that needs to be sent.

[0083] If the computing node responds to the request message and indicates that it can receive the data that needs to be sent, the data path is sent to the corresponding computing node. If the data path is sent, a sorting space is selected from each sorting space, and the process returns to determine whether the data in the selected sorting space has been sent, and then continues to perform subsequent operations.

[0084] If the computing node indicates that it is currently processing other data and cannot process more data, it selects the next sorting space from each sorting space, returns to determine whether the data in the selected next sorting space has been sent, and continues to perform subsequent operations;

[0085] Repeat the above steps until all N channels of data are sent, and then end the operation.

[0086] The technical solution of an embodiment of the present invention combines the sending operator and the sorting operator when the sending and sorting columns are the same, or the sending column is the leading column of the sorted column. After the sorting is completed in the N-way sorting space, the sorted N-way data is sent. Sorting the N-way data within the N-way sorting space and sending the sorted N-way data to the corresponding computing nodes allows all computing nodes in the distributed system to receive and process the data, improving the rationality of resource allocation during data sorting and processing in a distributed environment.

[0087] Example 2

[0088] Figure 2 This is a flow chart of a data sorting method provided according to the second embodiment of the present invention. This embodiment is a further refinement based on the above-mentioned first embodiment.

[0089] In an embodiment of the present invention, determining whether the sending mode is column-based sending and the columns to be sent and sorted are the same includes:

[0090] Determine whether the sending method is to send by key value, if so, determine whether the sending method is to send by column;

[0091] Determine whether the key value of the sending operator is consistent with the key value of the sorting operator. If so, determine that the columns sent and sorted are the same.

[0092] In an embodiment of the present invention, the sending operator and the sorting operator are combined, including:

[0093] Remove the sort operator below the send operator in the plan subtree;

[0094] Place the functionality of the sort operator into the send operator to complete the sorting function within the send operator.

[0095] In an embodiment of the present invention, placing each row of data into the sorting space of the corresponding path includes:

[0096] According to the key value corresponding to the send operator and N, each row of data is placed in the sort space of the corresponding path.

[0097] In an embodiment of the present invention, sending sorted N paths of data includes:

[0098] Select a sorting space from each sorting space and determine whether the data in the sorting space has been sent;

[0099] If not, send a request message to the computing node corresponding to the sorting space;

[0100] receiving response information from the computing node based on the request information, the response information including first indication information of receiving data or second indication information of not receiving data;

[0101] After obtaining the response information including the first indication information, the data is obtained from the sorting space and sent to the computing node, and the process returns to continue determining whether the data in the sorting space has been sent.

[0102] like Figure 2 As shown, the method includes:

[0103] S111. When the current environment is a distributed environment and there is a sorting plan node below the sending plan node in the plan tree, determine whether the sending mode is sending by key value; if so, determine whether the sending mode is sending by column.

[0104] Each key value can identify a column of data in the table. Once the key value is determined, the data column to be sent can be determined. There is no limitation on whether the sending method is to send by key value. For example, it can be determined by determining whether the key value is stored in the storage space of the distributed system. If so, the sending method is determined to be key value sending, and then the sending method is determined to be column-based sending.

[0105] S112: Determine whether the key value of the sending operator is consistent with the key value of the sorting operator. If so, determine that the columns to be sent and sorted are the same.

[0106] There is no limitation on the method for determining whether the key value of the sending operator is consistent with the key value of the sorting operator. For example, it can be determined whether the columns corresponding to the key values ​​of the sending operator and the sorting operator are the same. If so, the key value of the sending operator is consistent with the key value of the sorting operator, and then it is determined that the columns to be sent and sorted are the same.

[0107] S121. Remove the sort operator below the send operator in the plan subtree.

[0108] S122: Put the function of the sort operator into the send operator to complete the sort function in the send operator.

[0109] Steps S121 and S122 can be understood as modifying the sorting of the plan subtree in the plan tree and removing the sorting operator below the send operator, that is, removing the node representing the sorting operator after the send operator in the plan tree, and at the same time implementing the function of the sorting operator within the send operator.

[0110] S123: Set the flag of the sending operator to a set value.

[0111] S131. During the execution phase, if the flag of the send operator is a set value, the sorting space required for N-way sorting is allocated, and each row of data is placed in the sorting space of the corresponding way according to the key value corresponding to the send operator and N.

[0112] The key value corresponding to the send operator can identify the column that the send operator needs to send. When placing each row of data in the column into the sorting space, you can customize which row of data will be sent to which computing node, and the customization method is not limited; and the computing node uniquely corresponds to one sorting space; therefore, for a row of data in the column, if it is determined to which computing node the row of data will be sent, the sorting space of the path corresponding to the computing node can be determined, and the row of data can be placed into the sorting space of the path corresponding to the computing node, that is, each row of data is placed into the sorting space of the corresponding path according to the key value corresponding to the send operator and N.

[0113] S141. After data acquisition is completed, the data in the N-way sorting space is sorted.

[0114] S142: Select a sorting space from each sorting space and determine whether the data in the sorting space has been sent.

[0115] There is no limitation on the method of selecting a sorting space from each sorting space. For example, each sorting space is numbered in order from 1 to N. When selecting a sorting space, the sorting spaces corresponding to 1 to N can be selected in sequence.

[0116] There is no limitation on the method of determining whether the data in the sorting space has been sent. For example, it can be determined by querying whether there is any unsent data in the sorting space to be determined. If so, it indicates that the data in the sorting space has not been sent. Otherwise, the data in the sorting space has been sent. For example, the Q[P] array is used to indicate whether the data in each sorting space has been sent. If Q[P] = 1, the data in the sorting space has been sent. If Q[P] = 0, the data in the sorting space has not been sent. The value of Q[P] and its meaning can be set according to actual needs and are not limited by the present invention. P represents the number of sorting data to be processed. The value of P can be [1, N], indicating the 1st to Nth sorting data.

[0117] S143: If not, send a request message to the computing node corresponding to the sorting space.

[0118] The request information may be information that requests the computing node to send data. The form of the request information is not limited, as long as it can confirm to the computing node whether it can receive the data sent by the sorting space.

[0119] If not, a request message is sent to the computing node corresponding to the sorting space. It can be understood that the data in the sorting space has not been sent completely and the data in the sorting space needs to be sent to the computing node. Then a request message can be sent to the computing node to confirm whether the computing node can receive the data sent by the sorting space.

[0120] S144: Receive response information from the computing node based on the request information, where the response information includes first indication information indicating that data can be received or second indication information indicating that data cannot be received.

[0121] After sending the request information to the computing node corresponding to the sorting space, the computing node responds to the request information and returns the response information.

[0122] The response information may refer to information returned by the computing node indicating whether the computing node can receive data. The form of the response information is not limited, as long as the response node can return information indicating whether the computing node can receive data.

[0123] The response information includes first indication information indicating that the data can be received or second indication information indicating that the data cannot be received. The first indication information indicates that the computing node can receive data and the data in the sorting space corresponding to the computing node can be sent to the computing node; the second indication information indicates that the computing node cannot receive data, that is, the computing node is currently processing other data and cannot process more data. There is no limitation on the specific form of the first indication information and the second indication information. For example, the first indication information and the second indication information can be represented by the identification bit of the indication information. For example, setting f2=1 is the first indication information and f2=0 is the second indication information.

[0124] The manner of receiving the response information of the computing node based on the request information response is not limited. For example, the response information of the computing node based on the request information response can be received by wirelessly communicating with the computing node.

[0125] S145. After obtaining the response information including the first indication information, obtain the data from the sorting space and send it to the computing node, and return to continue to determine whether the data in the sorting space has been sent.

[0126] Obtaining response information including the first indication information, obtaining data from the sorting space and sending it to the computing node, it can be understood that if the computing node can receive data, the data in the sorting space is sent to the computing node so that the computing node can process the sent data.

[0127] After sending the data in the sorting space to the computing node, it is necessary to return to determine whether the data in the sorting space has been sent to ensure that all the data in the sorting space can be sent to the computing node.

[0128] Furthermore, the data sorting method also includes:

[0129] If all the data in the sorting space has been sent, the process returns to select the next sorting space from each sorting space.

[0130] If the number of paths corresponding to the selected sorting space is less than or equal to N, then continue to perform subsequent operations after selecting the sorting space;

[0131] If the number of paths corresponding to the selected sorting space is greater than N, it is determined whether the data of the N-path sorting spaces have been sent. If not, a sorting space is selected from each sorting space and subsequent operations are performed.

[0132] If the data in the sorting space has been sent, the process returns to select the next sorting space from each sorting space. This means that the data in the currently selected sorting space has been sent, and the data in the next sorting space can be processed. Therefore, the process returns to select the next sorting space from each sorting space.

[0133] There is no limitation on the method of selecting the next sorting space from each sorting space. For example, when the N-way sorting spaces are numbered from 1 to N, the number corresponding to the currently selected sorting space can be increased by 1 to obtain the next sorting space.

[0134] If the number of paths corresponding to the selected sorting space is less than or equal to N, then the subsequent operations after selecting the sorting space are continued to be executed, which can be understood as determining whether the data in the next selected sorting space has been sent, and continuing to execute steps S143 to S145.

[0135] If the number of paths corresponding to the selected sorting space is greater than N, it is determined whether the data in the N-way sorting space has been sent. It can be understood that all N-way sorting spaces have been selected, and then it is determined whether the data in the N-way sorting space has been sent. If not, continue to select a sorting space from each sorting space and perform subsequent operations to ensure that all data in the N-way sorting space is sent to the computing node.

[0136] Furthermore, the data sorting method also includes:

[0137] After obtaining the response information including the second indication information, returning to continue selecting the next sorting space from each sorting space;

[0138] If the number of paths corresponding to the selected sorting space is less than or equal to N, then continue to perform subsequent operations after selecting the sorting space;

[0139] If the number of paths corresponding to the selected sorting space is greater than N, it is determined whether the data of the N-path sorting spaces have been sent. If not, a sorting space is selected from each sorting space and subsequent operations are performed.

[0140] After obtaining the response information including the second indication information, returning to continue to select the next sorting space from each sorting space, it can be understood that the computing node corresponding to the currently selected sorting space cannot receive data, and it is necessary to continue to select the next sorting space from each sorting space so that the next sorting space can continue to execute subsequent operations.

[0141] After the next sorting space is selected, subsequent operations for selecting the sorting space are performed until all sorting spaces are selected and the data in each sorting space is sent to the computing node, and the operation ends.

[0142] The technical solution of this embodiment of the present invention combines the sending operator and the sorting operator when sending by column and the columns being sent and sorted are the same. After sorting is completed in the N-way sort space, the sorted N-way data is sent. This allows all data in the N-way sort space to be sent to the compute nodes, allowing each compute node to participate in data processing, improving resource allocation during data sorting in a distributed environment.

[0143] In one embodiment, the following is another data sorting method provided by Embodiment 2 of the present invention, including:

[0144] 1) Determine whether the current environment is a distributed environment. If so, continue with step 2), otherwise exit the optimization.

[0145] 2) Determine whether there is a plan subtree with SORT below SEND in the plan tree (that is, there is a sort plan node below the send plan node in the plan tree). If so, continue with step 3), otherwise exit the optimization.

[0146] 3) Determine whether the sending method of SEND is to send according to the KEY value (i.e., key value) (i.e., step S111), and the KEY is consistent with the KEY of the SORT below (i.e., step S112). If so, set the optimization mark (i.e., identification bit) f1 of the SEND operator to 1 (i.e., step S123), otherwise it is 0. The consistency of the KEY value indicates that the data processed by SEND and SORT are the same, and the two operators can be merged. When f1 is 1, remove the SORT below SEND in the plan tree (i.e., step S121), and implement the SORT function in SEND (i.e., step S122), and continue with step 4), otherwise exit the optimization.

[0147] 4) Execution Phase: If f1 of SEND is 1, the memory structure and space required for the N-way sorting of the SEND node to be sent are first allocated at the storage node. When the lower layer returns data, each row of data is placed into the corresponding sorting space according to the KEY value and N (i.e., step S131). After all the lower layer data has been retrieved, the data in the N-way sorting space is sorted in sequence (i.e., step S141). After the sorting is completed, the data is sent in sequence according to step 5).

[0148] 5) Sending data, including steps a) to e), specifically as follows:

[0149] a) Initialize variables, P represents the number of sorted data to be processed, and takes values: [1, N], with an initial value of 1;

[0150] The Q[P] array indicates whether each path of sorted data has been sent, and its value range is: 0 (initial value, not sent yet), 1 (sent yet) (determining the value of Q[P] is step S142);

[0151] Go to step b).

[0152] b) If Q[P] is 1, the data on this path has been sent. Go to step c);

[0153] If Q[P] is 0, it means that the data on this path has not been sent yet and needs to be processed further, then a request is sent to inquire whether the remote site has the resources to process it, that is, a request message is sent to the target computing node corresponding to the P-th path sorted data (that is, step S143). If the target computing node is capable of receiving data, it returns a received data message (that is, the first indication message), marked with f2=1; otherwise, it returns a non-receivable data message (that is, the second indication message), marked with f2=0, and executes step d).

[0154] c) Set P = P + 1. If P is less than or equal to N (i.e., the number of paths corresponding to the selected sorting space is less than or equal to N), execute step b); otherwise, execute step e).

[0155] d) After the storage node receives the return message (ie, the response information), if the flag f2 = 1, it sends a batch of data to the computing node (ie, step S145), and then executes step b); if f2 = 0, it executes step c).

[0156] e) Check the Q[P] array. If all N data have been sent, the process ends. Otherwise, wait for a specified interval and start a new cycle. Set P=1 and execute step b).

[0157] This data sorting method can send the data of each SORT in turn, and send the request information first before sending the data to verify whether the receiving end (i.e., the computing node) can receive and process the current data. If the receiving end cannot process more data because other storage nodes have sent too much data and are processing it, the next SORT will be sent instead, so that each computing node can receive and process the data, thereby improving the performance of the distributed system.

[0158] Example 3

[0159] Figure 3 is a structural diagram of a data sorting device according to the third embodiment of the present invention. This embodiment is applicable to the case of sorting data in a distributed environment, such as Figure 3 As shown, the specific structure of the device includes:

[0160] Determining module 21, for determining whether the sending mode is column-based sending, and whether the sent and sorted columns are the same, or whether the sent column is a leading column of the sorted columns, when the current environment is a distributed environment and there is a sorting plan node below the sending plan node in the plan tree;

[0161] A merging module 22 is configured to merge the sending operator and the sorting operator if the sending mode is column-by-column sending and the sent and sorted columns are the same, or the sent column is the leading column of the sorted column, and set the flag of the sending operator to a set value;

[0162] The allocation module 23 is used to allocate the sorting space required for N-way sorting and place each row of data into the sorting space of the corresponding way if the flag of the sending operator is a set value during the execution phase;

[0163] The sending module 24 is used to sort the data in the N-way sorting space and send the sorted N-way data after the data acquisition is completed.

[0164] The data sorting device provided in this embodiment first determines whether the sending mode is column-by-column sending, and the sent and sorted columns are the same, or the sent column is the leading column of the sorted column, through the determination module 21 when the current environment is a distributed environment and there is a sorting plan node below the sending plan node in the plan tree; secondly, when the sending mode is column-by-column sending, and the sent and sorted columns are the same, or the sent column is the leading column of the sorted column, through the merging module 22, the sending operator and the sorting operator are merged, and the identification bit of the sending operator is set to a set value; then, in the execution stage, if the identification bit of the sending operator is the set value, the sorting space required for N-way sorting is allocated by the allocation module 23, and each row of data is placed in the sorting space of the corresponding way; finally, after the data acquisition is completed, the sending module 24 sorts the data in the N-way sorting space and sends the sorted N-way data.

[0165] Furthermore, the determination module 21 is specifically configured to:

[0166] Determine whether the sending method is to send by key value, if so, determine whether the sending method is to send by column;

[0167] Determine whether the key value of the sending operator is consistent with the key value of the sorting operator. If so, determine that the columns sent and sorted are the same.

[0168] Furthermore, the merging module 22 is specifically configured to:

[0169] Remove the sort operator below the send operator in the plan subtree;

[0170] Place the functionality of the sort operator into the send operator to complete the sorting function within the send operator.

[0171] Furthermore, the allocation module 23 is specifically configured to:

[0172] According to the key value corresponding to the send operator and N, each row of data is placed in the sort space of the corresponding path.

[0173] Furthermore, the sending module 24 is specifically configured to:

[0174] Select a sorting space from each sorting space and determine whether the data in the sorting space has been sent;

[0175] If not, send a request message to the computing node corresponding to the sorting space;

[0176] receiving response information from the computing node based on the request information, the response information including first indication information of receiving data or second indication information of not receiving data;

[0177] After obtaining the response information including the first indication information, the data is obtained from the sorting space and sent to the computing node, and the process returns to continue determining whether the data in the sorting space has been sent.

[0178] Furthermore, the sending module 24 further includes:

[0179] A first returning unit is configured to return to select the next sorting space from the sorting spaces if all data in the sorting space have been sent;

[0180] A first determining unit is configured to continue executing subsequent operations after selecting the sorting space if the number of paths corresponding to the selected sorting space is less than or equal to N;

[0181] The second determining unit is configured to determine whether the data of the N-way sorting spaces have been sent if the number of ways corresponding to the selected sorting space is greater than N. If not, continue to select a sorting space from each sorting space and perform subsequent operations.

[0182] Furthermore, the sending module 24 further includes:

[0183] A second returning unit is configured to return to continue selecting a next sorting space from each sorting space after obtaining the response information including the second indication information;

[0184] A third determining unit is configured to continue executing subsequent operations after selecting the sorting space if the number of paths corresponding to the selected sorting space is less than or equal to N;

[0185] The fourth determining unit is configured to determine whether the data of the N-way sorting spaces have been sent if the number of ways corresponding to the selected sorting space is greater than N. If not, continue to select a sorting space from each sorting space and perform subsequent operations.

[0186] The data sorting device provided in the embodiment of the present invention can execute the data sorting method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0187] Example 4

[0188] Figure 4A schematic diagram of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0189] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0190] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0191] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors for running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the data sorting method.

[0192] In some embodiments, the data sorting method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data sorting method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data sorting method in any other suitable manner (e.g., by means of firmware).

[0193] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0194] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0195] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0196] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0197] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0198] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0199] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0200] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A data sorting method, characterized in that: include: When the current environment is a distributed environment and there is a sort plan node below the send plan node in the plan tree, determine whether the sending mode is column-by-column, and the sent and sorted columns are the same, or the sent column is the leading column of the sorted column; If so, merge the send operator and the sort operator, and set the flag of the send operator to the set value; In the execution phase, if the flag of the sending operator is the set value, the sorting space required for N-way sorting is allocated, and each row of data is placed in the sorting space of the corresponding way; After the data acquisition is completed, the data in the N-way sorting space is sorted and the sorted N-way data is sent; The step of determining whether the sending mode is column-based sending and the columns to be sent and sorted are the same includes: Determine whether the sending method is to send by key value, if so, determine whether the sending method is to send by column; Determine whether the key value of the sending operator is consistent with the key value of the sorting operator. If so, determine that the columns sent and sorted are the same; The merge send operator and the sort operator include: removing the sort operator below the send operator in the plan tree; Put the functionality of the sort operator into the send operator to complete the sort function within the send operator; Among them, the method for determining whether there is a sorting plan node below the sending plan node in the plan tree includes traversing the nodes in the plan tree from the root node to the child nodes. If the traversal result shows that the sending plan node appears first and then the sorting plan node appears, it is determined that there is a sorting plan node below the sending plan node in the plan tree.

2. The method according to claim 1, characterized in that The step of placing each row of data into the sorting space of the corresponding path includes: According to the key value corresponding to the send operator and N, each row of data is placed in the sort space of the corresponding path.

3. The method according to claim 1, characterized in that The sending of the sorted N-channel data includes: Selecting a sorting space from each sorting space and determining whether the data in the sorting space has been sent; If not, send a request message to the computing node corresponding to the sorting space; receiving response information from the computing node based on the request information, the response information including first indication information of receiving data or second indication information of not receiving data; After obtaining the response information including the first indication information, data is obtained from the sorting space and sent to the computing node, and the process returns to continue determining whether the data in the sorting space has been sent.

4. The method according to claim 3, characterized in that Also includes: If the data in the sorting space is sent, return to continue selecting the next sorting space from the sorting spaces; If the number of paths corresponding to the selected sorting space is less than or equal to N, then continue to perform subsequent operations after selecting the sorting space; If the number of paths corresponding to the selected sorting space is greater than N, it is determined whether the data of the N-path sorting spaces have been sent. If not, a sorting space is selected from each sorting space and subsequent operations are performed.

5. The method according to claim 3, characterized in that Also includes: After obtaining the response information including the second indication information, returning to continue selecting the next sorting space from each sorting space; If the number of paths corresponding to the selected sorting space is less than or equal to N, then continue to perform subsequent operations after selecting the sorting space; If the number of paths corresponding to the selected sorting space is greater than N, it is determined whether the data of the N-path sorting spaces have been sent. If not, a sorting space is selected from each sorting space and subsequent operations are performed.

6. A data sorting device, characterized in that: include: A determination module is used to determine whether the sending mode is column-based sending, and the sent and sorted columns are the same, or the sent column is the leading column of the sorted column, when the current environment is a distributed environment and there is a sorting plan node below the sending plan node in the plan tree; a merging module, configured to merge a sending operator and a sorting operator if the sending mode is column-by-column sending and the sent and sorted columns are the same, or the sent column is a leading column of the sorted column, and set a flag of the sending operator to a set value; The allocation module is used to allocate the sorting space required for N-way sorting and place each row of data into the sorting space of the corresponding way if the flag bit of the sending operator is a set value during the execution phase; The sending module is used to sort the data in the N-way sorting space and send the sorted N-way data after the data acquisition is completed; The determining module is specifically configured to: Determine whether the sending method is to send by key value, if so, determine whether the sending method is to send by column; Determine whether the key value of the sending operator is consistent with the key value of the sorting operator. If so, determine that the columns sent and sorted are the same; Wherein, the merging module is specifically used for: removing the sort operator below the send operator in the plan tree; Put the functionality of the sort operator into the send operator to complete the sort function within the send operator; The determining module is specifically configured to: The nodes in the plan tree are traversed from the root node to the child nodes. If the traversal result shows that the sending plan node appears first and then the sorting plan node appears, it is determined that there is a sorting plan node below the sending plan node in the plan tree.

7. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images