Distributed computing method and system under storage and computation separation architecture

Abstract

The invention discloses a distributed computing method and system under a storage and computation separation architecture, the system comprises a control module, a storage module and a computing module, the computing module comprises a group of computing nodes, and each computing node comprises a processing module, a push-down decision module and a cache module; the processing module is used for receiving and processing calculation tasks; the push-down decision module pushes down the query of the hot spot write-in partition to the storage module through a hot spot write-in data push-down mechanism, so that the performance problem caused by frequent hot spot write-in data and cache failure under the storage and calculation separation architecture is solved; through a mechanism of pushing down a cold query, the query of a non-hotspot write-in partition is pushed down to a storage module, so that the problem of cache space waste caused by cache of cold data is solved; through a data pre-fetching push-down mechanism, queries which do not hit the cache are pushed down to a storage module, and meanwhile, the same task is executed asynchronously, so that the problem of high delay when the cache is not hit is solved; and the cache module is used for caching the partition data.

Description

Technical Field

[0001] The present invention belongs to the field of database technology, and specifically relates to a distributed computing method and system under a storage-computing separation architecture. Background Art

[0002] In the traditional storage-computing integrated architecture, data storage and computing are integrated into the same system, so computing nodes can directly access the data of storage nodes, thus avoiding data transmission in the network. However, the storage-computing integrated architecture couples storage and computing, which will face problems such as storage and computing competing for resources and difficulty in dynamically scaling computing capabilities. In order to solve these problems, the storage-computing separation architecture came into being. In the storage-computing separation architecture, storage and computing are independent systems, thus avoiding resource competition between each other, and computing nodes can be dynamically scaled according to user business conditions.

[0003] In a storage-computing separation architecture, computing nodes use the network to obtain data from remote data nodes, which reduces query performance. Therefore, in a storage-computing separation architecture, hot data of remote data nodes are generally cached in computing nodes or some operators are pushed down to storage nodes for execution. However, this implementation faces the following problems: 1. Frequently written data will cause the cache to become invalid, causing the computing nodes to consume a large amount of resources to load invalid cache; 2. For scanning queries, the data cache obtained by the computing node from the data node has little value and will occupy the cache space of the computing node, affecting other queries.

[0004] 3. When the computing node cache misses, the computing node needs to access the data node to obtain data and cache it, resulting in high latency; Summary of the invention

[0005] In order to solve the deficiencies of the prior art and achieve the purpose of saving resources, improving query efficiency and reducing latency, the present invention adopts the following technical solutions: A distributed computing method under a storage-computation separation architecture, applied to computing nodes; Get the tasks corresponding to a single partition; Determine the partition category corresponding to the task according to the task. If it is a hotspot write partition and / or not a hotspot query partition, push the task down so that the task is calculated on the data node. Because the data of the hotspot write partition will change continuously, even if the current computing node caches the data of this part of the partition, the data in the cache will soon become invalid, resulting in a large number of invalid cache loads and I / O operations. In scanning queries, the partition data is usually queried once and simply filtered. Therefore, these partitions often have no hotspots, resulting in a low cache hit rate, which not only reduces the value of the cache, but also occupies the cache space of the computing node, thereby affecting other queries. Otherwise, determine whether the corresponding partition data is cached in the cache of the current computing node. If it is cached, directly pull the partition data for task calculation. If it is not cached, based on the partition, pull the data of the entire partition from the data node for task calculation and cache it in the current computing node. For uncached data with cache prefetching set, asynchronous pull is performed while the task is pushed down to pull the entire partition data of the data node and cache it in the current computing node, thus avoiding the network transmission cost caused by cache misses and making the user unaware of the delay of cache misses during the first query. At the same time, when the partition is accessed next time, it is likely to be in the cache. Provide feedback on the calculation results of the task.

[0006] Further, the computing task is obtained and split into multiple subtasks, wherein the subtask only involves a single partition; Obtaining the location of the subtask partition, and sending the subtask to the corresponding computing node, so that the corresponding computing node determines and executes the single partition category and cache prefetching; Aggregate and / or calculate the results of subtasks.

[0007] Furthermore, the determination of whether the hotspot is written into a partition is based on the interval from when the partition was last updated to the current time. If the interval is less than the time interval threshold, the hotspot is written into the partition.

[0008] Furthermore, the hotspot query partition is determined by comparing the number of times the partition is accessed with a threshold number of times. If the number is greater than the threshold, it is a hotspot query partition.

[0009] A distributed computing system under a storage-computation separation architecture is applied to a computing module, wherein the computing module includes one or more computing nodes, and the computing node includes a processing module, a push-down decision module, and a cache module; The processing module receives and processes computing tasks; The push-down decision module determines the corresponding partition category according to the computing task corresponding to a single partition. If it is a hotspot write partition and / or not a hotspot query partition, the computing task is pushed down so that the computing task is calculated on the data node. Because the data of the hotspot write partition will change continuously, even if the current computing node caches the data of this part of the partition, the data in the cache will soon become invalid, resulting in a large number of invalid cache loads and I / O operations. In scan-type queries, partition data is usually queried once and simply filtered. Therefore, these partitions often have no hotspots, resulting in a low cache hit rate, which not only reduces the value of the cache, but also occupies the cache space of the computing node, thereby affecting other Query, otherwise, determine whether the corresponding partition data is cached in the cache module of the current computing node. If it is cached, directly pull the partition data for task calculation. If it is not cached, pull the data of the entire partition from the data node based on the partition through the cache module for task calculation, and cache it in the cache module of the current computing node; for those that are not cached and cache pre-fetch is set, asynchronous pull is performed while the computing task is pushed down to pull the entire partition data of the data node and cache it in the cache module of the current computing node, thereby avoiding the network transmission cost caused by cache misses, so that users do not perceive the delay of cache misses during the first query. At the same time, when the partition is accessed next time, it is likely to be in the cache; The cache module caches partition data.

[0010] Furthermore, the determination of whether the hotspot is written into a partition is based on the interval from when the partition was last updated to the current time. If the interval is less than the time interval threshold, the hotspot is written into the partition.

[0011] Furthermore, the hotspot query partition is determined by comparing the number of times the partition is accessed with a threshold number of times. If the number is greater than the threshold, it is a hotspot query partition.

[0012] Further, the system includes a control module, a storage module and a computing module; The control module is used to store metadata of the partition data, the metadata including the location of the partition data in the storage module, whether the partition data is cached by the computing module, and the location of the partition cache data in the computing module; The storage module includes a group of data nodes for storing data and having a certain computing capability to execute the computing tasks pushed down by the computing module; The computing module is used to execute computing tasks and cache the data required for the computing tasks; wherein the processing module decomposes the received computing tasks into multiple subtasks, each of which involves only a single partition. The processing module communicates with the control module to obtain the location of the subtask partition and sends the subtask to the corresponding computing node. The processing module summarizes and calculates the results of the subtask. When the cache module does not cache the required partition data, it communicates with the control module to obtain the location of this partition in the storage module, and pulls the data of the entire partition from the storage module and caches it.

[0013] The advantages and beneficial effects of the present invention are: The present invention pushes down queries for hot spot write partitions to the storage module, thereby solving the performance problem caused by frequent hot spot data writing and cache failure under the storage and computing separation architecture; the present invention pushes down queries for non-hot spot write partitions to the storage module, thereby solving the problem of wasting cache space due to caching cold data; the present invention pushes down queries that miss the cache to the storage module, and asynchronously executes the same task at the same time, thereby solving the problem of high latency when the cache misses. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the structure of the system in an embodiment of the present invention.

[0015] Figure 2 It is a schematic diagram of the structure of a computing node in a computing module in an embodiment of the present invention.

[0016] Figure 3 It is a flowchart executed by the push-down module in an embodiment of the present invention.

[0017] Figure 4 is a flow chart of a method in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0019] like Figure 1 As shown, a distributed computing system under a storage-computation separation architecture includes a control module, a computing module, and a storage module that are interconnected. Data in the system is generally organized in partitions, and some data with similar characteristics will be stored in the same partition.

[0020] The control module is mainly responsible for storing the metadata of the partition data, which includes the location of the partition data in the storage module, whether the partition data is cached by the computing module, and the location of the partition cache data in the computing module.

[0021] The storage module is used to store actual data and has a certain computing capability to execute the computing tasks pushed down by the computing module.

[0022] The computing module is used to perform computing tasks and cache the data required for computing tasks, such as Figure 2 As shown, the computing module is usually composed of one or more computing nodes, and each computing node has an independent push-down decision module, a processing module, and a cache module.

[0023] The processing module is responsible for receiving and processing computing tasks. When the processing module receives a computing task, it will break it down into multiple subtasks, where each subtask only involves a single partition. Then the processing module will communicate with the control module to obtain the location of the subtask partition (also known as routing information) and send the subtask to the corresponding computing node. Finally, the processing module will summarize and calculate the results of the subtask.

[0024] When the processing module of computing node A receives a subtask sent by other computing nodes, it will first ask the push decision module whether to push this task to the storage module. If so, the processing module will push this computing task to the storage module. Otherwise, it will query whether the cache module has cached the data of this partition. If it has been cached, it will directly read this part of the data and execute the computing task. If it has not been cached, the cache module will communicate with the control module to obtain the storage location of this partition in the storage module, pull the data of the entire partition from the storage module, and cache it in the cache module.

[0025] The push decision module is mainly responsible for deciding whether the current computing task needs to be pushed down to the storage module, such as Figure 3 As shown, the execution process includes the following steps: First, determine whether the current partition is a hotspot write partition. If so, push this task down to the data node. Because for hotspot write partitions, the partition data will change constantly. Even if the data of this part of the partition is cached, the data in the cache will soon become invalid. This will cause a large number of invalid cache loads and I / O operations. Among them, the hotspot write partition is determined by the interval from the last time the partition was updated to the current time. When the interval is less than the threshold α, it is considered to be a hotspot write partition. The default value of α is 0, which can also be specified by the user.

[0026] Then, determine whether the current partition is a hot query partition. If not, push this task down to the data node. In a scan-type query, the partition data is usually queried once and simply filtered. Therefore, these partitions often have no hot spots, resulting in a low cache hit rate. This situation not only reduces the value of the cache, but also occupies the cache space of the computing node, thereby affecting other queries. Among them, the hot query partition is judged as follows: when the number of times a partition is accessed is greater than the threshold β, it is considered a hot query partition. The default value of β is 0, which can also be specified by the user.

[0027] Next, determine whether to enable the cache prefetch function. If enabled, push the task down to the data node, and asynchronously pull the entire partition data of the data node and cache it. This method avoids the network transmission cost caused by cache misses, so that users do not perceive the delay of cache misses during the first query. At the same time, the partition is likely to be in the cache the next time it is accessed. This function is enabled by default and can also be specified by the user.

[0028] The cache module is responsible for caching partition data. When the push decision module determines that the current computing task does not need to be pushed down to the storage module, the processing module will pull data from the cache module. If the cache module has cached the data of this partition, it will return directly. If there is no cache, the cache module will communicate with the control module to obtain the storage location of this partition in the storage module, pull the data of the entire partition from the storage module, and cache it in the cache module.

[0029] like Figure 4 As shown in the figure, a distributed computing under the storage and computing separation architecture is shown. In this embodiment, it is assumed that there is a storage module composed of three physical machines. The data is divided into three partitions. The partitioning method is hash partitioning. Each physical machine stores the data of one partition. The computing module is also composed of three physical machines. The computing task is select * from t full table query, which specifically includes the following steps: Step 1: The user performs a computing task on a computing node, which is first submitted to the processing module.

[0030] Step 2: After the processing module receives the computing task, if the computing task only involves a single partition and the partition will be routed to the current computing node, jump to step 4; otherwise, decompose the computing task into subtasks with only a single partition. In this embodiment, select * from t will be decomposed into query tasks for three partitions, and then execute step 3.

[0031] Step 3: The processing module accesses the control module, obtains the routing information of each partition, and then sends the subtask to the corresponding computing node.

[0032] Step 4: After receiving the subtask of a single partition, the processing module will ask the push decision module whether to push it down. The push decision module determines whether to push it down based on the partition category and asks the cache module whether it hits. If it does not hit and cache prefetching is enabled, the task is pushed down to the data node, and the entire partition data of the data node is asynchronously pulled and cached. Specifically, the following steps are included: Step 4.1: The push-down decision module determines whether the current partition is a hotspot write partition. If so, the query of this partition will be pushed down to the storage module for execution; otherwise, step 4.2 will be executed; the judgment mechanism of hotspot write partition is: the time T1 since the last update of the current partition, if T1 is less than the threshold delay, it will be considered as a hotspot write partition, and the delay size is set by the user; Step 4.2: The push-down decision module determines whether the current partition is a hot query partition. If not, the query of this partition is pushed down to the storage module for execution. Otherwise, execute step 4.3. Among them, the judgment mechanism of the hot query partition is: if the number of times the current partition is queried exceeds the threshold, it will be considered a hot query partition. The size of the threshold is set by the user.

[0033] Step 4.3: The push decision module asks the cache module whether the cache hits. If it does not hit and cache prefetching is enabled, the task is pushed down to the data node, and the entire partition data of the data node is asynchronously pulled and cached. Otherwise, step 5 is executed. This method avoids the network transmission cost caused by cache misses, so that users do not perceive the delay of cache misses during the first query. At the same time, the partition is likely to be in the cache when it is accessed next time. This function is enabled by default and can be specified by the user.

[0034] Enable cache prefetching, asynchronous pulling, and disable synchronization because there are two operations in asynchronous mode: first, synchronously push the task to the data node, and then asynchronously pull the data of the entire partition and cache it; for OLAP (analytical system), most of the calculation / query tasks are to pull the data of the entire partition and perform some aggregation and other operations. The amount of data in the calculation result is much smaller than the amount of data in the entire partition. Therefore, when cache prefetching is not enabled, the user-perceived overhead is: pulling the data of the entire partition + calculation, and when cache prefetching is enabled, the user-perceived overhead is: calculation + transmission result overhead; therefore, such operations will reduce the problem of extremely high query time consumption when the cache misses in the storage-computing separation system (this is a common problem of the storage-computing separation architecture).

[0035] Step 5: Query the cache module whether the data of this partition is cached. If it is already cached, return directly; otherwise, the cache module communicates with the control module to obtain the location where the partition is stored in the storage module, and then pulls the data of the entire partition from the storage module and caches it in the cache module.

[0036] Step 6: After the processing module obtains the data, it starts to perform calculations and returns the calculation results to the user.

[0037] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed computing method under a storage-computing separation architecture, applied to a computing node, characterized in that: Get the tasks corresponding to a single partition; Determine the partition category corresponding to the task according to the task. If it is a hotspot write partition and / or not a hotspot query partition, push the task down so that the task is calculated on the data node. Otherwise, determine whether the corresponding partition data is cached in the cache of the current computing node. If it is cached, directly pull the partition data for task calculation. If it is not cached, pull the data of the entire partition from the data node based on the partition for task calculation and cache it in the current computing node. For those that are not cached and cache prefetch is set, asynchronous pull is performed while the task is pushed down to pull the entire partition data of the data node and cache it in the current computing node; Provide feedback on the calculation results of the task.

2. A distributed computing method under a storage-computation separation architecture according to claim 1, characterized in that: Get the computation task and split it into multiple subtasks, where the subtask only involves a single partition; Obtaining the location of the subtask partition, and sending the subtask to the corresponding computing node, so that the corresponding computing node determines and executes the single partition category and cache prefetching; Aggregate and / or calculate the results of subtasks.

3. The distributed computing method in a storage-computation separation architecture according to claim 1, characterized in that: The determination of whether the hotspot is written into a partition is based on the interval from when the partition was last updated to the current time. If the interval is less than the time interval threshold, the partition is written into a hotspot.

4. The distributed computing method in a storage-computation separation architecture according to claim 1, characterized in that: The determination of the hotspot query partition is to compare the number of times the partition is accessed with a threshold of the number of times. If the number is greater than the threshold, it is a hotspot query partition.

5. A distributed computing system under a storage-computing separation architecture, applied to a computing module, characterized in that: The computing module includes one or more computing nodes, and the computing nodes include a processing module, a push-down decision module, and a cache module; The processing module receives and processes computing tasks; The push-down decision module determines the corresponding partition category according to the computing task corresponding to a single partition. If it is a hotspot write partition and / or not a hotspot query partition, the computing task is pushed down so that the computing task is calculated on the data node. Otherwise, it is determined whether the corresponding partition data is cached in the cache module of the current computing node. If it is cached, the partition data is directly pulled for task calculation. If it is not cached, the cache module is used to pull the data of the entire partition from the data node for task calculation based on the partition, and cache it in the cache module of the current computing node. For those that are not cached and cache prefetch is set, asynchronous pull is performed while the computing task is pushed down to pull the entire partition data of the data node and cache it in the cache module of the current computing node; The cache module caches partition data.

6. A distributed computing system under a storage-computing separation architecture according to claim 5, characterized in that: The determination of whether the hotspot is written into a partition is based on the interval from when the partition was last updated to the current time. If the interval is less than the time interval threshold, the partition is written into a hotspot.

7. A distributed computing system under a storage-computing separation architecture according to claim 5, characterized in that: The determination of the hotspot query partition is to compare the number of times the partition is accessed with a threshold of the number of times. If the number is greater than the threshold, it is a hotspot query partition.

8. A distributed computing system under a storage-computing separation architecture according to claim 5, characterized in that: The system includes a control module, a storage module and a computing module; The control module is used to store metadata of the partition data, the metadata including the location of the partition data in the storage module, whether the partition data is cached by the computing module, and the location of the partition cache data in the computing module; The storage module includes a group of data nodes for storing data and having a certain computing capability to execute the computing tasks pushed down by the computing module; The computing module is used to execute computing tasks and cache the data required for the computing tasks; wherein the processing module decomposes the received computing tasks into multiple subtasks, each of which involves only a single partition. The processing module communicates with the control module to obtain the location of the subtask partition and sends the subtask to the corresponding computing node. The processing module summarizes and calculates the results of the subtask. When the cache module does not cache the required partition data, it communicates with the control module to obtain the location of this partition in the storage module, and pulls the data of the entire partition from the storage module and caches it.

Images