A method, device, equipment and medium for parallel data distribution

Abstract

The present invention relates to the field of data query technology, and in particular to a method, device, equipment and medium for parallel data distribution. If it is detected that the distribution mode of sub-data is local distribution, then when the data to be queried is partitioned locally, the data is distributed to the thread with the corresponding number according to the distribution key of the sub-data to perform operations. When the data to be queried is not partitioned locally, the data is distributed to the thread with the corresponding number according to the distribution key of the sub-data to perform operations. If it is detected that the distribution mode of sub-data is remote distribution, the sub-data is distributed to the thread with the corresponding number according to the distribution key of the sub-data, and the data distributed in the storage block is distributed to the thread with the corresponding number to perform operations. Local distribution and remote distribution use different distribution modes to distribute to threads, realize parallelism within the query operator, reduce the number of redistributions during execution during data query, and thus improve the distribution efficiency during execution.

Description

Technical Field

[0001] The present invention relates to the field of data query technology, and in particular to a data parallel distribution method, device, equipment and medium. Background Art

[0002] Data distribution is a key technology of distributed databases. Reasonable data distribution can affect the performance, availability and scalability of the system. Through effective data distribution strategies, load balancing can be achieved, network communication overhead can be reduced, and the impact of single point failures can be reduced. At the same time, higher concurrency performance and better system flexibility can also be provided. Therefore, data distribution technology is crucial to the design and implementation of distributed database systems. In the prior art, data distribution under distributed conditions generally adopts a unified distribution method between nodes and within nodes. When many operators execute distributed data, it is difficult to achieve parallel execution between nodes and within nodes, resulting in low execution efficiency. Therefore, how to improve the execution efficiency of data distribution has become an urgent problem to be solved. Summary of the invention

[0003] In view of this, embodiments of the present application provide a data parallel distribution method, apparatus, device and medium to solve the problem of how to distribute data to improve execution efficiency.

[0004] In a first aspect, an embodiment of the present application provides a data parallel distribution method, the data parallel distribution method comprising:

[0005] Obtain a query statement, determine the query operator and the data to be queried of the query statement, and if the query operator meets the parallel condition, determine the storage type of the data to be queried in distributed storage;

[0006] Determine, according to the query operator and the storage type of the data to be queried, a distribution method of any sub-data of the data to be queried in the distributed storage;

[0007] If it is detected that the distribution mode of the sub-data is local distribution, then when the data to be queried has a partition locally, the sub-data is distributed to the required storage blocks with corresponding numbers according to the distribution key of the sub-data, and the thread with corresponding number is determined according to the number of the storage block, and the data distributed in the storage block is distributed to the thread with corresponding number for execution; when the data to be queried does not have a partition locally, the thread with corresponding number is determined according to the distribution key of the sub-data, and the sub-data is distributed to the thread with corresponding number for execution;

[0008] If it is detected that the distribution method of the sub-data is remote distribution, the sub-data is distributed to the required storage block with the corresponding number according to the distribution key of the sub-data, the thread with the corresponding number is determined according to the number of the storage block, and the data distributed in the storage block is distributed to the thread with the corresponding number to perform the operation.

[0009] In a second aspect, an embodiment of the present application provides a data parallel distribution device, the data parallel distribution device comprising:

[0010] An acquisition module is used to acquire a query statement, determine a query operator and data to be queried of the query statement, and if the query operator satisfies a parallel condition, determine a storage type of the data to be queried in distributed storage;

[0011] A determination module, used to determine the distribution mode of any sub-data of the data to be queried in the distributed storage according to the query operator and the storage type of the data to be queried;

[0012] A local distribution module, for, if it is detected that the distribution mode of the sub-data is local distribution, then when the data to be queried has partitions locally, according to the distribution key of the sub-data, the sub-data is distributed to the required storage blocks with corresponding numbers, according to the number of the storage block, the threads with corresponding numbers are determined, and the data distributed in the storage blocks are distributed to the threads with corresponding numbers for execution of operations, and when the data to be queried does not have partitions locally, according to the distribution key of the sub-data, the threads with corresponding numbers are determined, and the sub-data are distributed to the threads with corresponding numbers for execution of operations;

[0013] A remote distribution module is used to distribute the sub-data to the required storage blocks with corresponding numbers according to the distribution key of the sub-data if it is detected that the distribution mode of the sub-data is remote distribution, determine the threads with corresponding numbers according to the number of the storage block, and distribute the data distributed in the storage block to the threads with corresponding numbers to perform operations.

[0014] In a third aspect, an embodiment of the present application provides a network device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data parallel distribution method as described in the first aspect when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data parallel distribution method as described in the first aspect is implemented.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] Obtain a query statement, determine the query operator and the data to be queried of the query statement, if the query operator meets the parallel condition, determine the storage type of the data to be queried in distributed storage, determine the distribution mode of any sub-data of the data to be queried in the distributed storage according to the query operator and the storage type of the data to be queried, if it is detected that the distribution mode of the sub-data is local distribution, then when the data to be queried has partitions locally, distribute the sub-data to the required storage blocks of corresponding numbers according to the distribution key of the sub-data, determine the threads of corresponding numbers according to the number of the storage block, and distribute the data distributed in the storage block to the threads of corresponding numbers to perform operations, when the data to be queried does not have partitions locally, determine the threads of corresponding numbers according to the distribution key of the sub-data, and distribute the sub-data to the threads of corresponding numbers to perform operations, if it is detected that the distribution mode of the sub-data is remote distribution, distribute the sub-data to the required storage blocks of corresponding numbers according to the distribution key of the sub-data, determine the threads of corresponding numbers according to the number of the storage block, and distribute the data distributed in the storage block to the threads of corresponding numbers to perform operations. In this application, the distribution method of data distribution is determined according to the storage type of data, so that local distribution and remote distribution are distributed to threads using different distribution methods, thereby realizing parallelism within the query operator. During the data query process, the number of redistributions during execution is reduced, thereby improving the distribution efficiency during execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0019] Figure 1 This is a schematic diagram of an application environment of a data parallel distribution method provided in Example 1 of the present application;

[0020] Figure 2 It is a flowchart of a data parallel distribution method provided in Example 2 of the present application;

[0021] Figure 3 It is a flowchart of a data parallel distribution method provided in Example 3 of the present application;

[0022] Figure 4 It is a flowchart of a data parallel distribution method provided in Embodiment 4 of the present application;

[0023] Figure 5 It is a flowchart of a data parallel distribution method provided in Embodiment 5 of the present application;

[0024] Figure 6It is a structural diagram of a data parallel distribution device provided in Example 6 of the present application;

[0025] Figure 7 This is a structural diagram of a network device provided in Example 7 of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0028] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0029] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0030] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0032] It should be understood that the size of the serial numbers of the steps in the following embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0033] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0034] A data parallel distribution method provided in the first embodiment of the present application can be applied in the following cases: Figure 1 In the application environment, the client includes but is not limited to PDA, desktop computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, cloud network equipment, personal digital assistant (PDA) and other devices. The network equipment corresponding to the server can be implemented by an independent server or a server cluster composed of multiple servers.

[0035] See also Figure 2 , is a flow chart of a data parallel distribution method provided in Embodiment 2 of the present application. The above-mentioned data parallel distribution processing method is applied to Figure 1 The server in Figure 2 As shown, the data parallel distribution method may include the following steps:

[0036] S201: Obtain a query statement, determine the query operator of the query statement and the data to be queried, and if the query operator meets the parallel condition, determine the storage type of the data to be queried in distributed storage.

[0037] In step S201, when the client terminal needs to perform a query operation on the database, it sends a query request to the distributed node, and the query request includes a query statement. The query statement can be a regular query statement or a complex query statement. Among them, the regular query statement is a query statement that only includes one basic query statement, and the complex query statement is a query statement that includes no less than two basic query statements.

[0038] In this embodiment, a query statement is obtained, wherein the query statement may be a structured query (SQL) statement, and a query operator and data to be queried are determined according to the query statement, wherein the SQL statement includes a SELECT keyword, a query column, a FROM keyword, and a table name, wherein the query column is the part separated by commas between the SELECT keyword and the FROM keyword, and the query column is the result that the SQL statement needs to query and obtain. The query column may include an aggregate function to implement an aggregate query; the FROM keyword and the table name indicate that the data of the query column is obtained from the data table identified by the table name as the query result; the SQL statement may also include query operators such as WHERE, GROUP BY, and / or HAVING, for example, a subquery may follow the FROM keyword and the table name, indicating that the data of the query column is selectively obtained from the data table.

[0039] It should be noted that commonly used aggregate functions can be MAX (maximum value), MIN (minimum value), SUM (sum), AVG (average value), COUNT (number of records), etc. Complex aggregate functions are aggregate functions that can be expressed in the form of one or more simple aggregate functions. Among the commonly used aggregate functions, the AVG function can be expressed in the form of the SUM function divided by the COUNT function, so the AVG function is a complex aggregate function.

[0040] For example, the query statement SELECT a FROM A GROUP BY Aa obtains the results grouped by column Aa from table A.

[0041] If the query operator meets the parallel condition, the storage type of the data to be queried in distributed storage is determined. Among them, the storage types include horizontal sharding storage and replicated storage partition storage. In distributed database storage, horizontal sharding storage refers to dividing the data in the database into multiple fragments according to a certain rule, and each fragment is horizontally sharded and stored on different nodes. This method improves the scalability and performance of the system by allowing different nodes to share the load of data storage and processing. Horizontal sharding usually relies on distribution keys, and data is distributed to different nodes according to the value or range of the distribution key. Replicated storage refers to copying the data in the database to different nodes, which can improve the reliability and fault tolerance of the system. Replication can be divided into master-slave replication and multi-master replication, in which way data backup and recovery can be achieved. Partitioned storage refers to dividing the data in the database into non-intersecting parts according to a certain rule, and each part is distributed on different nodes. Partitioning can be performed based on the range, hash value or other attributes of the data, which can effectively reduce the bottleneck of the system and improve the concurrency performance of the database.

[0042] S202: Determine, according to the query operator and the storage type of the data to be queried, a distribution method of any sub-data of the data to be queried in the distributed storage.

[0043] In step S202, the distribution method of any sub-data of the data to be queried in the distributed storage is determined according to the query operator and the storage type of the data to be queried, wherein the distribution method includes local distribution and remote distribution, local distribution is the distribution method within a node in a distributed database, and remote distribution is the distribution method between nodes in a distributed database.

[0044] In this embodiment, the distribution method of any sub-data of the data to be queried in the distributed storage is determined according to the query operator and the storage type of the data to be queried. Among them, any sub-data in the distributed storage is the data stored in any node in the distributed storage, wherein the data stored in the node may be full data, that is, data stored in a copy, or it may be local data, that is, data stored in horizontal shards. The distribution method of any sub-data in the distributed storage is determined according to the storage type and the query operator, wherein the distribution is to distribute the sub-data to the threads in the node so that different threads can execute the data query at the same time.

[0045] For example, if the storage type of the data to be queried is replicated storage, and the query operator is the GROUP BY operator, a hash grouping algorithm is used for hash grouping to determine whether the data to be queried has a partition key. If the data to be queried has a partition key, determine whether the partition key is a subset of the grouping field, where the grouping field is the field after the query operator in the query statement, such as GROUP BY b, where b is the grouping field. If the partition key is not a subset of the grouping field, determine that the distribution mode of any sub-data of the data to be queried in the distributed storage is local distribution. For example, if GROUP BY a,b, the storage mode of any sub-data in the distributed storage is replicated storage, and the partition key in the sub-data is c, then the data in the node is partitioned by c, and it is considered that the partition key is not a subset of the grouping field. If the data to be queried does not have a partition key, determine that the distribution mode of any sub-data of the data to be queried in the distributed storage is local distribution.

[0046] When the query operator is the GROUP BY operator, determine whether the storage type of the data to be queried is horizontal shard storage. If the storage type of the data to be queried is horizontal shard storage, determine the distribution key of the data to be queried in the distributed storage, and determine whether the distribution key of the data to be queried in the distributed storage is a subset of the grouping field. If the distribution key of the data to be queried in the distributed storage is not a subset of the grouping field, determine that any sub-data of the data to be queried in the distributed storage is first distributed locally and then distributed remotely.

[0047] It should be noted that when the query operator is the GROUP BY operator, if the storage type of the data to be queried is replicated storage, determine whether the data to be queried has a partition key. If the data to be queried has a partition key, determine whether the partition key is a subset of the grouping field. If the partition key is a subset of the grouping field, hash grouping can be performed directly in each node without redistribution. For example, if GROUP BY a,b, the storage method of any sub-data in the distributed storage is replicated storage, that is, each node has the full amount of data. Assuming that there is a partition key a, the thread can scan the partition by the hash value of a, and group by b in the partition, then the query data grouped by a, b can be obtained without redistribution.

[0048] It should be noted that if the query operator is a join operator, the join operator is used to distribute multi-table connections, and it is determined whether each table data in the data to be queried is a duplicate table. If each table data is a duplicate table data, it is determined whether each table data has a partition key. If each table data has a partition key, and the partition key is a subset of the grouping field, and the partition key is the same part of the grouping field, then in each node, multi-table connections can be directly performed without distribution and determination of the distribution method.

[0049] It should be noted that, if the query operator is a join operator, the distribution method of any sub-data of the data to be queried in the distributed storage is determined according to the query operator and the storage type of the data to be queried. For example, the query statement select*fromA join B on Aa=Bb, the data to be queried includes the first data and the second data to be connected by the join operator; wherein the first data is table A, the second data is table B, and Aa=Bb is the connection field. Then, it is determined whether each table data in the first data and the second data is a horizontally sharded distribution table, that is, it is determined whether table A and table B are horizontally sharded distribution tables. If each table data is a distribution table, the distribution key in each table data is determined, and it is determined whether the distribution key of each table data is a subset of the connection field. If the distribution key in each table data is a subset of the grouping field, then in each node, multiple tables can be directly connected without redistribution.

[0050] See also Figure 3 , is a flow chart of a data parallel distribution method provided in Example 3 of the present application. Figure 3 As shown, in step S202, determining the distribution mode of any sub-data of the to-be-queried data in the distributed storage according to the query operator and the storage type of the to-be-queried data may include the following steps:

[0051] S301: Obtain the grouping field in the query statement;

[0052] S302: Determine whether the storage type of the data to be queried is replicated storage;

[0053] S303: If the storage type of the data to be queried is replicated storage, determine whether the data to be queried has a partition key;

[0054] S304: If the data to be queried has a partition key, determine whether the partition key is a subset of the grouping field; if the partition key is not a subset of the grouping field, determine that the distribution mode of any sub-data of the data to be queried in the distributed storage is local distribution.

[0055] In this embodiment, according to the query statement, the corresponding grouping field is determined, wherein the grouping field is the grouping or connection data of the target data to be queried. For example, when hash grouping is concurrent, the grouping field is the grouping after the query operator, for example, GROUP BY a, b in GROUP BY, wherein a, b correspond to the grouping field. It is determined whether the storage type of the data to be queried is replicated storage, that is, whether the stored data in each node in the distributed data storage is full data. If the storage type of the data to be queried is replicated storage, it is determined whether there is a partition key in the data to be queried, that is, whether the table data in the data to be queried is partitioned and stored. For example, partitioned storage can be partitioned according to the primary key in the data table, or partitioned according to the column field in the data table. If there is a partition key in the data to be queried, it is determined whether the partition key is a subset of the grouping field. If the partition key is not a subset of the grouping field, it is determined that the distribution mode of any sub-data of the data to be queried in the distributed storage is local distribution, wherein local distribution is to distribute the sub-data to different threads locally according to the distribution key of the replicated table data, so as to copy the data in different threads to other threads, so that different threads can store the corresponding full data.

[0056] It should be noted that when the partition key of the table in the data to be queried is a subset of the grouping field, the data query is executed directly without data distribution. For example, GROUP BY a,b in GROUP BY, where a and b correspond to the grouping fields, and the partition key of the table in the data to be queried is b, that is, the table in the data to be queried is partitioned and stored by b. Each line queries the corresponding grouped data according to the partition scan, and there is no need to redistribute the data.

[0057] It should be noted that when the query operator is a join operator, the connection field in the query statement is obtained to determine whether the storage type of the data to be queried is replicated storage, wherein the data to be queried includes the first data and the second data to be connected by the join operator; determine whether the storage type of the first data and the second data is replicated storage, and if the storage type of the first data and the second data is replicated storage, determine whether there is a partition key between the first data and the second data, for example, the query statement select*from A join B on Aa=Bb, wherein the first data is table A, the second data is table B, and Aa=Bb is the connection field, and determine whether there is a partition key between table A and table B. If there is a partition key in table A and no partition key in table B, and the partition key of table A is a subset of the connection field, then table B is locally distributed, and the number of distributed threads is equal to the number of partitions of table A, so that after each thread is used to scan each partition in table A, it can be directly processed one-to-one with the data distributed to the thread by table B.

[0058] Optionally, if the storage type of the data to be queried is replicated storage, after determining whether the data to be queried has a partition key, the method further includes:

[0059] If the data to be queried does not have a partition key, it is determined that the distribution mode of any sub-data of the data to be queried in the distributed storage is local distribution.

[0060] In this embodiment, if the storage type of the data to be queried is replicated storage, it is determined whether there is a partition key for the data to be queried. If there is no partition key for the data to be queried, that is, the replicated table data in any node is not partitioned and stored, then when querying the data in groups, it is necessary to calculate the hash value of the distribution key based on the distribution key in the replicated table data, determine the number of the thread mapped to the hash value based on the hash value of the distribution key, use the thread with the corresponding number to scan the data, and distribute the scanned data locally.

[0061] See also Figure 4 , is a flow chart of a data parallel distribution method provided in Embodiment 4 of the present application. Figure 4 As shown, in step S202, determining the distribution mode of any sub-data of the to-be-queried data in the distributed storage according to the query operator and the storage type of the to-be-queried data may include the following steps:

[0062] S401: Determine whether the storage type of the data to be queried is horizontal sharding storage;

[0063] S402: If the storage type of the data to be queried is horizontal shard storage, determine the distribution key of the data to be queried in the distributed storage;

[0064] S403: Determine whether the distribution key of the data to be queried in the distributed storage is a subset of the grouping field. If the distribution key of the data to be queried in the distributed storage is not a subset of the grouping field, determine whether any sub-data of the data to be queried in the distributed storage is first distributed locally and then distributed remotely.

[0065] In this embodiment, it is determined whether the storage type of the data to be queried is horizontal shard storage. If the storage type of the data to be queried is horizontal shard storage, the distribution key of the data to be queried in the distributed storage is determined, and it is determined whether the distribution key of the data to be queried in the distributed storage is a subset of the grouping field. If the distribution key of the data to be queried in the distributed storage is not a subset of the grouping field, it is determined that any sub-data of the data to be queried in the distributed storage is first distributed locally, and then distributed remotely. Among them, local distribution is performed first, that is, the sub-data is first distributed locally to different threads, and then remote distribution is performed, and the thread data in the local is distributed to the threads of different nodes, so that different nodes can obtain the full amount of data, and data group query is performed according to the corresponding full amount of data.

[0066] It should be noted that when the query operator is a join operator, it is determined whether the storage type of the data to be queried is horizontal shard storage; wherein the data to be queried includes the first data and the second data to be connected by the join operator; for example, the query statement select*from A join B on Aa=Bb, wherein the first data is table A, the second data is table B, and Aa=Bb is the connection field. If the storage type of table A is replication storage, and the storage type of table B is horizontal shard storage, and there is a distribution key in table B, and the distribution key of table A and the distribution key of table B are subsets of the grouping fields, and the distribution key of table A and the distribution key of table B are the same key, then table A is distributed locally. If there is no distribution key in table B, or the distribution key of table A and the distribution key of table B are not subsets of the grouping fields, or the distribution key of table A and the distribution key of table B are different distribution keys, then table A and table B are distributed remotely.

[0067] Optionally, if the distribution key of the data to be queried in the distributed storage is not a subset of the grouping field, the method further includes:

[0068] It is determined that any sub-data of the data to be queried in the distributed storage is remotely distributed.

[0069] In this embodiment, it is determined whether the storage type of the data to be queried is horizontally sharded storage. If the storage type of the data to be queried is horizontally sharded storage, the distribution key of the data to be queried in the distributed storage is determined, and it is determined whether the distribution key of the data to be queried in the distributed storage is a subset of the grouping field. If the distribution key of the data to be queried in the distributed storage is not a subset of the grouping field, it is determined that any sub-data of the data to be queried in the distributed storage is directly distributed remotely.

[0070] S203: If it is detected that the distribution mode of the sub-data is local distribution, then when the data to be queried exists in a partition locally, the sub-data is distributed to the required storage block with the corresponding number according to the distribution key of the sub-data, and the thread with the corresponding number is determined according to the number of the storage block, and the data distributed in the storage block is distributed to the thread with the corresponding number to perform operations. When the data to be queried does not exist in a partition locally, the thread with the corresponding number is determined according to the distribution key of the sub-data, and the sub-data is distributed to the thread with the corresponding number to perform operations.

[0071] In step S203, if it is detected that the distribution mode of the sub-data is local distribution, that is, the sub-data is distributed to the local thread. During the local distribution, the sub-data is distributed to the required storage block with the corresponding number according to the distribution key of the sub-data, wherein the number of the corresponding storage block can be calculated by a hash algorithm, and the thread with the corresponding number is determined according to the number of the storage block, and the data distributed in the storage block is distributed to the thread with the corresponding number to perform operations. When there is no partition in the local data to be queried, the thread with the corresponding number is determined according to the distribution key of the sub-data, and the sub-data is distributed to the thread with the corresponding number to perform operations, wherein the distribution key of the sub-data can be any column of the data table, and the data is distributed to the threads of each node according to the corresponding distribution key.

[0072] In this embodiment, if it is detected that the distribution method of sub-data is local distribution, then when the data to be queried is partitioned locally, the sub-data is distributed to the required storage blocks with corresponding numbers according to the distribution key of the sub-data. When distributing the sub-data to the required storage blocks with corresponding numbers, a hash algorithm can be used to calculate the hash value of the corresponding partition parameter in the partition key, number the storage blocks, calculate the remainder according to the corresponding hash value and the number of storage blocks, and according to the remainder, distribute the sub-data to the storage blocks with numbers corresponding to the remainder, determine the threads with corresponding numbers according to the numbers of the storage blocks, and distribute the data distributed in the storage blocks to the threads with corresponding numbers to perform operations.

[0073] When the data to be queried does not exist in a partition, the thread with the corresponding number is determined according to the distribution key of the sub-data, and the sub-data is distributed to the thread with the corresponding number to perform the operation. When distributing the sub-data to the threads with the corresponding numbers, a hash algorithm can be used to calculate the hash value of the distribution parameter in the distribution key, and the remainder is calculated according to the corresponding hash value and the number of threads, and the sub-data is distributed to the threads with the numbers corresponding to the remainder according to the remainder.

[0074] See also Figure 5 , is a flow chart of a data parallel distribution method provided in Embodiment 5 of the present application. Figure 5 As shown, determining the thread with the corresponding number according to the number of the storage block in step S203 may include the following steps:

[0075] S501: Determine a partition number corresponding to the storage block number according to a mapping relationship between the storage block number and the preset partition number;

[0076] S502: Calculate the hash value of the partition number, and determine the thread of the corresponding number according to the hash value of the partition number.

[0077] In this embodiment, the partition number corresponding to the storage block number is determined according to the mapping relationship between the storage block number and the preset partition number, wherein the mapping relationship between the storage block number and the preset partition number can be a one-to-one mapping relationship or a many-to-one mapping relationship. For example, when the storage block number and the preset partition number correspond one-to-one, one storage block number corresponds to one preset partition number, and the number of storage blocks is equal to the number of preset partitions. When the storage block number and the preset partition number correspond many-to-one, data stored in multiple storage blocks can be distributed to one preset partition, and the number of storage blocks is greater than the number of preset partitions.

[0078] According to the corresponding partition number, the hash value of the partition number is calculated, and the thread with the corresponding number is determined according to the hash value of the partition number and the preset number of threads, and the corresponding data is distributed to the thread with the corresponding number to perform the operation.

[0079] S204: If it is detected that the distribution method of the sub-data is remote distribution, the sub-data is distributed to the required storage blocks with corresponding numbers according to the distribution key of the sub-data, the threads with corresponding numbers are determined according to the number of the storage block, and the data distributed in the storage block is distributed to the threads with corresponding numbers to perform operations.

[0080] In step S204, if it is detected that the distribution method of the sub-data is remote distribution, the sub-data is distributed to the required storage blocks with corresponding numbers according to the distribution key of the sub-data. If in the process of horizontal sharding storage, the data stored in the node where the sub-data is located may be data corresponding to the distribution parameters in a distribution key, or may be data in multiple distribution parameters. According to the number of the storage block, the thread with the corresponding number is determined, and the data distributed in the storage block is distributed to the thread with the corresponding number to perform operations.

[0081] In this embodiment, when performing remote distribution, the sub-data is distributed to the required storage blocks of corresponding numbers according to the distribution key of the sub-data, wherein the number of the corresponding storage block can be calculated by a hash algorithm, and the thread of the corresponding number is determined according to the number of the storage block, and the data distributed in the storage block is distributed to the thread of the corresponding number to perform operations. For example, if the data stored in the node is the distribution parameter corresponding to the distribution key A, which is a1 and a2, that is, the remainder value of the hash value of a1 and the number of nodes in the distributed storage is equal to the remainder value of the hash value of a2 and the number of nodes in the distributed storage, and is distributed to the same node for storage, the corresponding remainder value is calculated according to the hash values ​​of a1 and a2 and the number of storage blocks, and the data in the distribution parameter a1 is distributed to the storage blocks of corresponding numbers according to the remainder value, wherein the remainder value is equal to the number of the storage block, and the partition number corresponding to the storage block number is determined according to the mapping relationship between the storage block number and the preset partition number, the hash value of the partition number is calculated, and the thread of the corresponding number is determined according to the hash value of the partition number.

[0082] Obtain a query statement, determine the query operator and the data to be queried of the query statement, if the query operator meets the parallel condition, determine the storage type of the data to be queried in distributed storage, determine the distribution mode of any sub-data of the data to be queried in the distributed storage according to the query operator and the storage type of the data to be queried, if it is detected that the distribution mode of the sub-data is local distribution, then when the data to be queried has partitions locally, distribute the sub-data to the required storage blocks of corresponding numbers according to the distribution key of the sub-data, determine the threads of corresponding numbers according to the number of the storage block, and distribute the data distributed in the storage block to the threads of corresponding numbers to perform operations, when the data to be queried does not have partitions locally, determine the threads of corresponding numbers according to the distribution key of the sub-data, and distribute the sub-data to the threads of corresponding numbers to perform operations, if it is detected that the distribution mode of the sub-data is remote distribution, distribute the sub-data to the required storage blocks of corresponding numbers according to the distribution key of the sub-data, determine the threads of corresponding numbers according to the number of the storage block, and distribute the data distributed in the storage block to the threads of corresponding numbers to perform operations. In this application, the distribution method of data distribution is determined according to the storage type of data, so that local distribution and remote distribution are distributed to threads using different distribution methods. During the data query process, the number of redistributions during execution is reduced, thereby improving the distribution efficiency during execution.

[0083] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a data parallel distribution device provided by Embodiment 6 of the present invention. Figure 2-Figure 5 as well as Figure 2-Figure 5 For the convenience of explanation, only the parts related to this embodiment are shown. Figure 6As shown, the data parallel distribution device 60 includes: an acquisition module 61, a determination module 62, a local distribution module 63, and a remote distribution module 64.

[0084] The acquisition module 61 is used to acquire a query statement, determine the query operator of the query statement and the data to be queried, and if the query operator meets the parallel condition, determine the storage type of the data to be queried in distributed storage.

[0085] The determination module 62 is used to determine the distribution mode of any sub-data of the data to be queried in the distributed storage according to the query operator and the storage type of the data to be queried.

[0086] The local distribution module 63 is used to distribute the sub-data to the required storage blocks with corresponding numbers according to the distribution key of the sub-data if it is detected that the distribution mode of the sub-data is local distribution, when the data to be queried exists partitions locally, determine the threads with corresponding numbers according to the numbers of the storage blocks, and distribute the data distributed in the storage blocks to the threads with corresponding numbers to perform operations; when the data to be queried does not exist partitions locally, determine the threads with corresponding numbers according to the distribution key of the sub-data, and distribute the sub-data to the threads with corresponding numbers to perform operations.

[0087] The remote distribution module 64 is used to distribute the sub-data to the required storage blocks with corresponding numbers according to the distribution key of the sub-data if it is detected that the distribution method of the sub-data is remote distribution, determine the threads with corresponding numbers according to the number of the storage block, and distribute the data distributed in the storage block to the threads with corresponding numbers to perform operations.

[0088] Optionally, the local distribution module 63 includes:

[0089] The mapping unit is used to determine the partition number corresponding to the storage block number according to the mapping relationship between the storage block number and the preset partition number.

[0090] The computing unit is used to calculate the hash value of the partition number, and determine the thread of the corresponding number according to the hash value of the partition number.

[0091] Optionally, the local determination module 62 includes:

[0092] The acquisition unit is used to obtain the grouping fields in the query statement.

[0093] The first judgment unit is used to judge whether the storage type of the data to be queried is replicated storage.

[0094] The second judgment unit is configured to judge whether the data to be queried has a partition key if the storage type of the data to be queried is replicated storage.

[0095] The third judgment unit is used to judge whether the partition key is a subset of the grouping field if the data to be queried has a partition key, and if the partition key is not a subset of the grouping field, determine that the distribution mode of any sub-data of the data to be queried in the distributed storage is local distribution.

[0096] Optionally, the local determination module 62 further includes:

[0097] The first determining unit is used to determine that the distribution mode of any sub-data of the to-be-queried data in the distributed storage is local distribution if the to-be-queried data does not have a partition key.

[0098] Optionally, the local determination module 62 further includes:

[0099] The fourth judgment unit is used to judge whether the storage type of the data to be queried is horizontal sharding storage.

[0100] The second determining unit is used to determine the distribution key of the data to be queried in the distributed storage if the storage type of the data to be queried is horizontal sharding storage.

[0101] The third determination unit is used to determine whether the distribution key of the data to be queried in the distributed storage is a subset of the grouping field. If the distribution key of the data to be queried in the distributed storage is not a subset of the grouping field, it is determined that any sub-data of the data to be queried in the distributed storage is first distributed locally and then distributed remotely.

[0102] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0103] Figure 7 This is a schematic diagram of the structure of a network device provided in Example 7 of this application. Figure 7 As shown, the network device of this embodiment includes: at least one processor ( Figure 7 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned data parallel distribution method embodiments are implemented.

[0104] The network device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 7 It is only an example of a network device and does not constitute a limitation on the network device. The network device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components, for example, it may also include a network interface, etc.

[0105] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0106] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of the network device, and the internal memory provides an environment for the operation of the operating system and the computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of the network device, and in other embodiments, it may also be an external storage device of the network device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the network device. Further, the memory may also include both an internal storage unit of the network device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0107] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiment when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0108] The present application implements all or part of the processes in the above-mentioned embodiment method, and may also be completed through a computer program product. When the computer program product runs on a network device, the network device implements the steps in the above-mentioned method embodiment when executing.

[0109] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0110] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0111] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0112] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0113] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data parallel distribution method, characterized in that: The data parallel distribution method comprises: Obtain a query statement, determine the query operator and the data to be queried of the query statement, and if the query operator meets the parallel condition, determine the storage type of the data to be queried in distributed storage; Obtain the grouping field in the query statement; Determine whether the storage type of the data to be queried is replicated storage; If the storage type of the data to be queried is replicated storage, determining whether the data to be queried has a partition key; If the data to be queried has a partition key, determine whether the partition key is a subset of the grouping field; if the partition key is not a subset of the grouping field, determine that the distribution mode of any sub-data of the data to be queried in the distributed storage is local distribution; If it is detected that the distribution mode of the sub-data is local distribution, then when the data to be queried has a partition locally, the sub-data is distributed to the required storage blocks with corresponding numbers according to the distribution key of the sub-data, and the thread with corresponding number is determined according to the number of the storage block, and the data distributed in the storage block is distributed to the thread with corresponding number for execution; when the data to be queried does not have a partition locally, the thread with corresponding number is determined according to the distribution key of the sub-data, and the sub-data is distributed to the thread with corresponding number for execution; If it is detected that the distribution method of the sub-data is remote distribution, the sub-data is distributed to the required storage block with the corresponding number according to the distribution key of the sub-data, the thread with the corresponding number is determined according to the number of the storage block, and the data distributed in the storage block is distributed to the thread with the corresponding number to perform the operation.

2. The data parallel distribution method according to claim 1, characterized in that: The step of determining a thread with a corresponding number according to the number of the storage block comprises: Determine the partition number corresponding to the storage block number according to the mapping relationship between the storage block number and the preset partition number; A hash value of the partition number is calculated, and a thread of a corresponding number is determined according to the hash value of the partition number.

3. The data parallel distribution method according to claim 1, characterized in that: If the storage type of the data to be queried is replicated storage, after determining whether the data to be queried has a partition key, the method further includes: If the data to be queried does not have a partition key, it is determined that the distribution mode of any sub-data of the data to be queried in the distributed storage is local distribution.

4. The data parallel distribution method according to claim 1, characterized in that: The step of determining, according to the query operator and the storage type of the data to be queried, a distribution mode of any sub-data of the data to be queried in the distributed storage further includes: Determine whether the storage type of the data to be queried is horizontal sharding storage; If the storage type of the data to be queried is horizontal sharding storage, determining the distribution key of the data to be queried in the distributed storage; Determine whether the distribution key of the data to be queried in the distributed storage is a subset of the grouping field. If the distribution key of the data to be queried in the distributed storage is not a subset of the grouping field, determine that any sub-data of the data to be queried in the distributed storage is first distributed locally and then distributed remotely.

5. The data parallel distribution method according to claim 4, characterized in that: If the distribution key of the to-be-queried data in the distributed storage is not a subset of the grouping field, the method further includes: It is then determined that any sub-data of the data to be queried in the distributed storage is remotely distributed.

6. A data parallel distribution device, characterized in that: The data parallel distribution device comprises: An acquisition module is used to acquire a query statement, determine a query operator and data to be queried of the query statement, and if the query operator satisfies a parallel condition, determine a storage type of the data to be queried in distributed storage; A determination module, used to obtain the grouping fields in the query statement; Determine whether the storage type of the data to be queried is replicated storage; If the storage type of the data to be queried is replicated storage, determining whether the data to be queried has a partition key; If the data to be queried has a partition key, determine whether the partition key is a subset of the grouping field; if the partition key is not a subset of the grouping field, determine that the distribution mode of any sub-data of the data to be queried in the distributed storage is local distribution; A local distribution module, for, if it is detected that the distribution mode of the sub-data is local distribution, then when the data to be queried has partitions locally, according to the distribution key of the sub-data, the sub-data is distributed to the required storage blocks with corresponding numbers, according to the number of the storage block, the threads with corresponding numbers are determined, and the data distributed in the storage blocks are distributed to the threads with corresponding numbers for execution of operations, and when the data to be queried does not have partitions locally, according to the distribution key of the sub-data, the threads with corresponding numbers are determined, and the sub-data are distributed to the threads with corresponding numbers for execution of operations; A remote distribution module is used to distribute the sub-data to the required storage blocks with corresponding numbers according to the distribution key of the sub-data if it is detected that the distribution mode of the sub-data is remote distribution, determine the threads with corresponding numbers according to the number of the storage block, and distribute the data distributed in the storage block to the threads with corresponding numbers to perform operations.

7. The data parallel distribution device according to claim 6, characterized in that: The local distribution module includes: A mapping unit, configured to determine a partition number corresponding to the storage block number according to a mapping relationship between the storage block number and a preset partition number; The calculation unit is used to calculate the hash value of the partition number, and determine the thread of the corresponding number according to the hash value of the partition number.

8. A network device, characterized in that: The network device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the data parallel distribution method according to claims 1 to 5 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data parallel distribution method according to claims 1 to 5 is implemented.

Images