Partition management method and apparatus, and computing device

Abstract

A partition management method and apparatus, and a computing device. The method comprises: during each map task, firstly determining a partition to which each data entry in each split belongs, wherein there are a plurality of pieces of data in each partition, and there is a correspondence between partitions and reduce tasks; then compiling statistics on data entries belonging to the same partition, so as to obtain a data entry statistical result; and then sending the data entry statistical result to a management node, such that the management node makes decisions on whether to adjust the reduce tasks.

Description

Partition management method, device and computing equipment

[0001] The present application claims priority to the Chinese patent application No. 202410905013.7, filed on July 5, 2024, with the State Intellectual Property Office, and entitled "Partition management method, device and computing equipment", the contents of which are incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application belongs to the field of computing, and particularly relates to a partition management method, device and computing equipment applied to distributed processing of big data. BACKGROUND

[0003] The MapReduce (MR) model is a distributed processing model for big data, which is used for parallel operation of large-scale sharding. The MapReduce runs a job by dividing the job into tasks. The tasks include two kinds of map tasks and reduce tasks. When the MapReduce model processes the data to be processed, the data to be processed is first divided into multiple splits, and then the splits are taken as the input data of the map tasks (different splits are input to different map tasks), the map tasks perform partition processing on the received splits to generate intermediate data (the intermediate data includes multiple data segments, and the data in the same data segment belongs to the same partition, and each partition corresponds to a reduce task), and then the intermediate data is written to the local disk, and the subsequent reduce tasks read (pull) the data of the specified partition to the local disk through the remote procedure call (RPC) mode. This data exchange process is also called shuffle.

[0004] In order to confirm whether the data read by each reduce task is balanced, the map task will count the data volume of each data segment in the intermediate data before the reduce task is executed, and send the counting result to the management node. After the management node aggregates the counting results sent by all map tasks, the data volume of the data to be read by each reduce task can be calculated, and it is judged whether the data read by each reduce task is balanced. If not, the number of reduce tasks is increased / decreased.

[0005] In the above process, the conclusion whether the data read by the reduce task is balanced can be drawn only after the intermediate data is generated by each map task, and the shuffle performance is low. SUMMARY

[0006] The application provides a partition management method and device and computing equipment, which can enable a management node to make a decision in advance on whether to adjust the number of reduce tasks.

[0007] In a first aspect, a partition management method is provided, which includes: a map task determining a partition to which each data entry in a split containing a plurality of data entries belongs; wherein the number of partitions is a plurality; counting a first type of data entry belonging to a first partition in the split to obtain a first data amount of the first type of data entries, the first partition being one of the plurality of partitions, and the first data amount indicating the number or size of the first type of data entries; and sending the first data amount to a management node. It can be understood that the split is input data of the map task, and a split contains a plurality of data entries.

[0008] According to the scheme of the first aspect, after receiving the input split (input data of the map task), the map task first determines the partition to which each data entry belongs. For each partition, the number or size of data entries belonging to the partition is counted. Then, before the map task generates intermediate data, the map task sends the data entry counting result to the management node, so that the management node can obtain the data entry counting result before the map task generates the intermediate data, and can further determine whether to adjust the number of reduce tasks according to the data entry estimation result. The decision on whether to adjust the number of reduce tasks is also called adaptive query execution (AQE) decision.

[0009] In combination with the first aspect, in a possible implementation manner, the first data amount is related to the task amount of a first reduce task; and the first reduce task is used for processing the data entries belonging to the first partition. It can be understood that the more data entries that a reduce task needs to read, the larger the task amount of the reduce task. The fewer data entries that a reduce task needs to read, the smaller the task amount of the reduce task. It can be understood that the number of map tasks is a plurality. After the management node receives the first data amount sent by all map tasks, the size of the first partition is known, and the task amount of the first reduce task can also be known.

[0010] With reference to the first aspect, in a possible implementation manner, the management node is configured to count data amounts of data entries in each partition, and determine whether to adjust the number of reduce tasks according to the counting result, wherein the reduce task is configured to process data entries belonging to a corresponding partition, and the first reduce task is one of the plurality of reduce tasks; and the management node is further configured to update the correspondence between the partition and the reduce task after determining to adjust the number of reduce tasks, and provide an access address of the data entry to the reduce task based on the updated correspondence between the partition and the reduce task. It can be understood that the access address is sent by the map task to the management node.

[0011] With reference to the first aspect, in a possible implementation manner, the map task is configured to store the first type of data entries in a centralized manner after sending the first data amount to the management node, and the sending of the first data amount to the management node specifically includes: sending the first data amount to the management node before the centralized storage.

[0012] The centralized storage is, for example, storing the first type of data entries in a continuous manner in terms of addresses (physical addresses or logical addresses) in a physical storage medium, or storing the first type of data entries in the same physical storage medium.

[0013] In the scheme shown in the present implementation manner, the map task starts to store the first type of data entries in a centralized manner after sending the first type of data entries to the management node. Therefore, the centralized storage of the first type of data entries is performed in parallel with the AQE decision made by the management node based on the counting result of the data entries. In one scheme, the map task reports the first data amount to the management node after storing the first type of data entries in a centralized manner, and the management node makes the AQE decision after receiving the first data amount sent by all map tasks. Compared with the scheme, the AQE decision of the management node can be made in advance in the scheme shown in the present implementation manner. Since the management node needs less time to make the AQE decision, the AQE decision is usually completed before the centralized storage of the first type of data entries, and therefore, the management node can determine the number of reduce tasks before the centralized storage of the first type of data entries is completed.

[0014] With reference to the first aspect, in a possible implementation manner, the map task is configured to store the first type of data entries in a centralized manner after sending the first data amount to the management node, and the sending of the first data amount to the management node specifically includes: sending the first data amount to the management node before the centralized storage.

[0015] In the scheme shown in the present implementation, the position of the first type of data entry in the shard is centrally stored. For example, the shard includes multiple rows of data, and a data entry is a row of data in the shard. The position of the data entry in the shard is the row number of the data entry in the shard, or the row offset of the data entry in the shard.

[0016] In the scheme shown in the present implementation, the position of the first type of data entry in the shard is centrally stored. For example, the shard includes multiple rows of data, and a data entry is a row of data in the shard. The position of the data entry in the shard is the row number of the data entry in the shard, or the row offset of the data entry in the shard.

[0017] In combination with the first aspect, in a possible implementation, after the first type of data entry or the position of the first type of data entry in the shard is centrally stored, the map task provides the first type of data entry to the first reduce task in response to a data entry reading request of the first reduce task based on the access address.

[0018] In the scheme shown in the present implementation, after the map task performs the central storage action, the map task reports the access address to the management node. This enables the management node to send the access address of the corresponding data entry to the first reduce task, and the first reduce task can obtain the first type of data entry according to the access address sent by the management node. Generally, there are multiple map tasks. Once a certain map task or certain map tasks report the access address to the management node, the first reduce task can read the first type of data entry. Instead of waiting for all map tasks to perform the central storage action before reading the first type of data entry, the data reading efficiency of the first reduce task can be improved.

[0019] In combination with the first aspect, in a possible implementation, the number of data entries corresponding to the partition is related to the task size of the reduce task. It can be understood that the more data entries the reduce task needs to read, the larger the task size of the reduce task. The fewer data entries the reduce task needs to read, the smaller the task size of the reduce task. Therefore, by counting the data entries corresponding to the partition, the data entry counting result obtained can reflect not only the number of data entries corresponding to the partition, but also the task size of the reduce task. Therefore, the management node can determine whether the task size of a certain reduce task or certain reduce tasks is too large or too small according to the data entry counting result, so as to determine whether to adjust the number of reduce tasks.

[0020] In a possible implementation manner of the first aspect, the map task determines the partition to which each data entry in the split belongs, including: the map task calculates each data entry in the split to obtain a partition identifier corresponding to the data entry; and the partition identifier indicates the partition to which the corresponding data entry belongs. It can be understood that the partition identifiers of the data entries belonging to the same partition are the same.

[0021] In a possible implementation manner of the first aspect, the method further includes: the computing node stores the split in the memory or the volatile cache pool of the computing node, where the access latency of the computing node to the volatile cache pool is less than the access latency of the computing node to the external storage; and the computing node provides, to the first reduce task through the management node, a storage address of the first type of data entry in the memory or the volatile cache pool as an access address.

[0022] In the scheme of the implementation manner of the first aspect, after the computing node stores the split in the memory or the volatile cache pool of the computing node, the computing node directly provides, to the first reduce task through the management node, the storage address of the first type of data entry in the memory or the volatile cache pool as an access address, thereby omitting the step of storing the first type of data entry centrally, and enabling the first reduce task to directly obtain the first type of data entry through the storage address of the first type of data entry in the memory or the volatile cache pool, thereby improving the data reading efficiency of the first reduce task.

[0023] In a possible implementation manner of the first aspect, the volatile cache pool is formed by pooling the memory of a plurality of computing nodes, and the plurality of computing nodes all support a high-speed interconnection protocol. The high-speed interconnection protocol is characterized by global addressing, that is, all computing nodes (including computing nodes executing map tasks and computing nodes executing reduce tasks) can read data in the volatile cache pool as if reading their own memory, without the need to read through a network, thereby improving the data reading efficiency.

[0024] In a possible implementation manner of the first aspect, the computing node supports a CXL or UB protocol.

[0025] In a possible implementation manner of the first aspect, the data cache pool includes a volatile cache pool and a non-volatile cache pool, and the split is stored in the volatile cache pool or the non-volatile cache pool of the data cache pool.

[0026] With reference to the first aspect, in a possible implementation manner, the computing node stores the shard into the memory or the volatile cache pool of the computing node, specifically comprising: the computing node judging whether the data amount of the shard is less than a threshold; if less than the threshold, the computing node stores the shard into the memory or the volatile cache pool of the computing node.

[0027] In the scheme of the present implementation, the volatile cache pool and the memory of the computing node have the characteristics of fast access speed, small storage space and data loss after power failure. Therefore, storing the shard in the volatile cache pool can improve the execution efficiency of the map task due to the fast storage speed; the shard stored in the volatile cache pool does not need to be partitioned, and the data copying action does not need to be performed, further improving the execution efficiency of the map task; the first reduce task can directly pull the first type of data entry from the volatile cache pool to the local, which improves the execution efficiency of the first reduce task due to the fast pulling speed; since the execution efficiency of the map task and the reduce task is very high, the shard is temporarily stored in the volatile cache pool or the memory of the computing node for a short time, and the loss probability is low. Therefore, storing the smaller shard in the volatile cache pool or the memory of the computing node and considering that the reduce task directly pulls the data entry greatly improves the shuffle efficiency.

[0028] With reference to the first aspect, in a possible implementation manner, the computing node stores the shard into the non-volatile cache pool; the access delay of the computing node to the non-volatile cache pool is less than the access delay of the computing node to the external storage. Compared with the volatile cache pool, the non-volatile cache pool has slower access speed, but has larger storage space and data will not be lost after power failure.

[0029] With reference to the first aspect, in a possible implementation manner, the map task stores the first type of data entry in the non-volatile cache pool; the storage address of the first type of data entry stored in the non-volatile cache pool is taken as an access address, and the access address is provided to the first reduce task by the management node.

[0030] With reference to the first aspect, in a possible implementation manner, the map task stores the storage address of the first type of data entry in the shard in the non-volatile cache pool; the map task takes the address of the storage address stored in the non-volatile cache pool as an access address, and the access address is provided to the first reduce task by the management node; the first data amount is greater than the second data amount, and the second data amount is the data amount of the second type of data entry, and the second type of data entry belongs to the second partition, and the second partition is one of the multiple partitions.

[0031] It can be understood that the data amount of the first type of data entries is large, and if the first type of data entries are stored centrally, a long time will be spent. In the scheme of the implementation mode, the storage addresses of the first type of data entries in the non-volatile cache pool in the shard are stored centrally, and the first type of data entries are not stored centrally. Compared with storing the first type of data entries centrally, the time required for storing the storage addresses of the first type of data entries in the non-volatile cache pool in the shard is less. Therefore, the first reduce task can quickly obtain the position information of the first type of data entries sent by the management node, so as to quickly read the first type of data entries and improve the data reading efficiency of the first reduce task.

[0032] In combination with the first aspect, in a possible implementation mode, when the proportion of the data amount of the first type of data entries in the shard exceeds a threshold value, the storage addresses of the first type of data entries in the non-volatile cache pool in the shard are stored centrally. After the map task calculates each data entry in the shard, the partition corresponding to each data entry can be determined, and the data amount of the data entries corresponding to the partition can be counted. If the proportion of the first type of data entries exceeds the threshold value, it indicates that the shard has data skew. If the first type of data entries are stored centrally, a long time will be spent, which reduces the execution efficiency of the map task. In the scheme of the implementation mode, when the first type of data entries are more, the map task does not store the first type of data entries centrally, which can improve the execution efficiency of the map task.

[0033] In combination with the first aspect, in a possible implementation mode, the non-volatile cache pool includes a first area and a second area; the first area is used to store the shard; and the central storage includes copying the first type of data entries or the positions of the first type of data entries in the shard to the second area.

[0034] It can be understood that the number of the second areas is multiple and corresponds to the partitions one by one. The reduce task has a corresponding relationship with the partition, and the reduce task can pull the data entries stored in the corresponding second area to the local.

[0035] The essence of the map task storing the first type of data entries centrally is to copy the first type of data entries to the second area for centralized storage. In the scheme of the implementation mode, a dedicated storage area is specially divided for the shard and the intermediate data, which can make the data copying orderly.

[0036] In combination with the first aspect, in a possible implementation mode, the method further includes that the size of the second area is determined in advance according to the statistical result of the data entries.

[0037] Since the management node has obtained the data entry statistical result of all map task feedbacks, the data amount of each partition is known, and therefore, according to the data amount of each partition, a second area of a corresponding size can be pre-allocated in the non-volatile cache pool, so as to try to ensure that data entries belonging to a same partition are stored in a similar address space, and the reduce task can be sequentially read, thereby improving the reading efficiency.

[0038] In combination with the first aspect, in a possible implementation, the method further includes: if the shard stored in the non-volatile cache pool satisfies a second condition, encapsulating the address of a data entry of a certain specific partition and the shard into a dictionary, so that the second computing node can read the data entry belonging to the specific partition in the shard to the local according to the indication of the dictionary; and for the data entries of other partitions, performing a partition copy action according to the correspondence between the data entries and the partitions to obtain intermediate data.

[0039] The second aspect provides a partition management method, which includes: a management node receiving data entry statistical results sent by all computing nodes; the data entry statistical result is obtained by counting data entries belonging to a same partition after the computing node determines the partition to which each data entry in a plurality of data entries contained in a shard belongs; the data entry statistical result is used to indicate the number or size of data entries belonging to the same partition; according to the received data entry statistical result, counting the data amount of data entries of each partition, and judging whether to adjust the number of reduce tasks according to the statistical result.

[0040] The third aspect provides a partition management method, which includes: a first reduce task reading a first type of data entry in a volatile cache pool or a non-volatile cache pool to the local according to the indication of a management node; the access delay of the reduce task to the volatile cache pool or the non-volatile cache pool is less than the access delay of the computing node executing a map task to an external storage.

[0041] The fourth aspect provides a partition management device, which includes: a determination module, which determines the partition to which each data entry in a plurality of data entries contained in a shard belongs; wherein the number of partitions is a plurality; a statistical module, which is used to count a first type of data entry belonging to a first partition in the shard to obtain a first data amount of the first type of data entry, the first partition being one of the plurality of partitions, and the first data amount being used to indicate the number or size of the first type of data entry; and a sending module, which is used to send the first data amount to a management node.

[0042] In a possible implementation manner of the fourth aspect, the first data amount is related to a task amount of the first reduce task; and the first reduce task is used for processing data entries belonging to the first partition.

[0043] In a possible implementation manner of the fourth aspect, the management node is configured to count data amounts of data entries of each partition, and determine whether to adjust the number of reduce tasks according to the counting result; the reduce task is used for processing data entries belonging to a corresponding partition; the first reduce task is one of the plurality of reduce tasks; and the correspondence between the partition and the reduce task is updated after it is determined to adjust the number of reduce tasks; and the management node is further configured to provide an access address of the data entry to the reduce task based on the updated correspondence between the partition and the reduce task.

[0044] In a possible implementation manner of the fourth aspect, the apparatus further includes a providing module configured to provide the first type of data entry to the first reduce task in response to a data entry reading request of the first reduce task based on the access address.

[0045] In a possible implementation manner of the fourth aspect, the apparatus further includes a centralized storage module configured to centrally store the first type of data entry; and the sending module is specifically configured to send the first data amount to the management node before the central storage.

[0046] In a possible implementation manner of the fourth aspect, the apparatus further includes a centralized storage module configured to centrally store the location of the first type of data entry in the shard; and the sending module is specifically configured to send the first data amount to the management node before the central storage.

[0047] In a possible implementation manner of the fourth aspect, the apparatus further includes a transferring module configured to store the shard into a memory or a volatile cache pool managed by the partition management apparatus, wherein an access delay of the partition management apparatus to the volatile cache pool is less than an access delay of the partition management apparatus to an external storage managed by the partition management apparatus; and the sending module is further configured to provide, to the first reduce task through the management node, a storage address of the first type of data entry in the shard in the memory or the volatile cache pool as the access address.

[0048] In a possible implementation manner of the fourth aspect, the apparatus further includes a transferring module configured to store the shard into a non-volatile cache pool; and an access delay of the partition management apparatus to the non-volatile cache pool is less than an access delay of the partition management apparatus to an external storage managed by the partition management apparatus.

[0049] In combination with the fourth aspect, in a possible implementation manner, the apparatus further includes a centralized storage module, configured to store the first type of data entries in the non-volatile cache pool in a centralized manner; and the sending module is further configured to provide, to the first reduce task through the management node, a storage address of the centralized storage of the first type of data entries in the non-volatile cache pool as an access address.

[0050] In combination with the fourth aspect, in a possible implementation manner, the apparatus further includes a centralized storage module, configured to store the first type of data entries in the non-volatile cache pool in a centralized manner;

[0051] The sending module is further configured to provide, to the first reduce task through the management node, an address of the centralized storage of the storage address in the non-volatile cache pool as an access address; the first data amount is greater than the second data amount, and the second data amount is a data amount of the second type of data entries, the second type of data entries belonging to the second partition, and the second partition being one of the multiple partitions.

[0052] In the fifth aspect, a partition management apparatus is provided, which includes a receiving module configured to receive data entry statistical results sent by all computing nodes; the data entry statistical result is obtained by counting data entries belonging to a same partition after each data entry in a plurality of data entries determined to be contained in a shard belongs to a partition, and the data entry statistical result is used to indicate a number or a size of data entries belonging to the same partition; and an adjusting module configured to count data amounts of data entries of each partition according to the received data entry statistical results, and determine whether to adjust a number of reduce tasks according to the counting results.

[0053] In the sixth aspect, a partition management apparatus is provided, which includes a reading module configured to read, according to an indication of a management node, a first type of data entries in a volatile cache pool or a non-volatile cache pool to a local; a time delay of reading data from the volatile cache pool or the non-volatile cache pool is less than a time delay of reading data from an external storage of a computing node executing a map task.

[0054] In the seventh aspect, a computing device is provided, which includes a memory configured to store computer instructions, and one or more processors configured to execute the computer instructions stored in the memory, so that the computing device executes the method in the first to third aspects and any implementation manners thereof.

[0055] In the eighth aspect, a computer readable storage medium is provided, which stores computer program codes, and when the computer program codes are executed by a computing device, the computer implements the method in the first to third aspects and any implementation manners thereof.

[0056] In a ninth aspect, a computer program product is provided, which includes a computer program stored in a computer readable storage medium, and the computer program is loaded by a processor to implement the method in the first to third aspects and any implementation manner thereof.

[0057] The second, third, fourth, fifth, sixth, seventh, eighth and ninth aspects have similar various possible designs as the first aspect and can produce corresponding technical effects, which are not described here.

[0058] In a tenth aspect, a partition management method is provided, which includes: a map task determining a partition to which each data entry in a plurality of data entries included in a split belongs; wherein the number of partitions is a plurality; and a storage address of a first type of data entry belonging to a first partition in the split being taken as an access address, and being provided to a first reduce task by a management node; wherein the first partition is one of the plurality of partitions, and the first reduce task is one of a plurality of reduce tasks.

[0059] In the scheme shown in the implementation mode, the data entries in the split are not stored in a centralized manner according to the partitions, and the reduce task directly reads data from the split, thereby saving the time spent on centralized storage and improving the shuffle efficiency.

[0060] In combination with the tenth aspect, in a possible implementation manner, the data amount of the data entries in the split is less than a threshold value. It can be understood that when the data amount of the data entries in the split is less than the threshold value, the reduce task spends less time reading the data entries, and the specified data entries can be read to the local more quickly.

[0061] In an eleventh aspect, a partition management method is provided, which includes: a map task determining a partition to which each data entry in a plurality of data entries included in a split belongs; wherein the number of partitions is a plurality; and a storage address of a first type of data entry belonging to a first partition in the split being stored in a centralized manner; an address of the centrally stored storage address being taken as an access address, and being provided to a first reduce task by a management node; a data amount of the first type of data entry belonging to the first partition being greater than a data amount of a second type of data entry belonging to a second partition; wherein the first partition is one of the plurality of partitions, the second partition is one of the plurality of partitions, and the first reduce task is one of a plurality of reduce tasks.

[0062] In the scheme shown in the implementation mode, the first type of data entry is not stored in a centralized manner, and the reduce task directly reads data from the split, thereby saving the time spent on centralized storage and improving the shuffle efficiency.

[0063] With reference to the eleventh aspect, in a possible implementation, the proportion of the data amount of the first type of data entry in the shard exceeds a threshold. It can be understood that the number of the first type of data entry is large, and the centralized storage of the first type of data entry can save more time. BRIEF DESCRIPTION OF DRAWINGS

[0064] FIG. 1 is a structural schematic diagram of a data processing system;

[0065] FIG. 2 is a structural schematic diagram of a computing node;

[0066] FIG. 3 is a structural schematic diagram of a storage device;

[0067] FIG. 4 is a schematic diagram of a data file and an index file obtained after a map task calculates and partition copy shards;

[0068] FIG. 5 is a schematic diagram of a shuffle process in the related art;

[0069] FIG. 6 is a flowchart of a partition management method provided by an embodiment of the present application;

[0070] FIG. 7 is a data flow diagram of a shuffle process provided by an embodiment of the present application;

[0071] FIG. 8 is a schematic diagram of metadata provided by an embodiment of the present application;

[0072] FIG. 9 is a schematic diagram of another metadata and an estimated partition data amount provided by an embodiment of the present application;

[0073] FIG. 10 is a schematic diagram of sending a partition data amount to a management node provided by an embodiment of the present application;

[0074] FIG. 11 is a schematic diagram of a management node adjusting a reduce task number provided by an embodiment of the present application;

[0075] FIG. 12 is a schematic diagram of another management node adjusting a reduce task number provided by an embodiment of the present application;

[0076] FIG. 13 is a flowchart of a shuffle process provided by an embodiment of the present application;

[0077] FIG. 14 is a flowchart of another shuffle process provided by an embodiment of the present application;

[0078] FIG. 15 is a structural schematic diagram of a storage system provided by an embodiment of the present application;

[0079] FIG. 16 is a structural schematic diagram of a data cache pool provided by an embodiment of the present application;

[0080] FIG. 17 is a schematic diagram of a result of performing partitioned copy on data according to an embodiment of the present application;

[0081] FIG. 18 is a schematic diagram of storage locations of a slice and metadata according to an embodiment of the present application;

[0082] FIG. 19 is a schematic diagram of a structure of a data cache pool according to an embodiment of the present application;

[0083] FIG. 20 is a flowchart of a process of configuring a data cache pool by a management node according to an embodiment of the present application;

[0084] FIG. 21 is a schematic diagram of a connection manner of a compute node and a memory terminal / PMEM terminal according to an embodiment of the present application;

[0085] FIG. 22 is a schematic diagram of another connection manner of a compute node and a memory terminal / PMEM terminal according to an embodiment of the present application;

[0086] FIG. 23 is a flowchart of another partition management method according to an embodiment of the present application;

[0087] FIG. 24 is a schematic diagram of obtaining partition data according to a dictionary according to an embodiment of the present application;

[0088] FIG. 25 is a schematic diagram of storing a dictionary according to an embodiment of the present application;

[0089] FIG. 26 is a schematic diagram of a structure of a partition management apparatus according to an embodiment of the present application;

[0090] FIG. 27 is a schematic diagram of another structure of a partition management apparatus according to an embodiment of the present application;

[0091] FIG. 28 is a schematic diagram of another structure of a partition management apparatus according to an embodiment of the present application;

[0092] FIG. 29 is a schematic diagram of a structure of a data processing system according to an embodiment of the present application;

[0093] FIG. 30 is a schematic diagram of a structure of a compute node according to an embodiment of the present application. DETAILED DESCRIPTION

[0094] The embodiment of the present application provides a partition management method, which can be applied to a shuffle process in a spark computing framework, and the spark computing framework is a distributed computing framework. Other computing frameworks with similar data processing logic can also apply the scheme. The following describes the shuffle process in the spark computing framework as an example.

[0095] FIG. 1 is a structural diagram of a spark computing framework. As shown in FIG. 1, the spark computing framework includes a management node 100, a computing node 200, a storage device 300, etc. The management node and the computing node can be hardware, such as a PC or a server, or can be implemented by software, such as a virtual machine or a container. The storage device can provide a storage space for storing data. The management node 100 allocates computing tasks, such as map tasks and reduce tasks, to each computing node 200 in the spark computing framework, and can monitor the progress of each computing node 200 in performing the computing tasks. The map task refers to a task of performing computation and partitioning on data in a split, and the reduce task refers to a task of aggregating the computation results of multiple map tasks. A computing node 200 can obtain data (a split) required by a computing task from itself or other computing nodes 200 or the storage device 300, perform the corresponding computing task, and store the computation result generated by performing the computing task in a local memory or a local disk or the storage device 300. A split is a data set, which includes multiple data (data).

[0096] It can be understood that a split is input data of a map task. In the embodiments of the present application, one data is also referred to as one data entry.

[0097] It should be noted that one computing node 200 can perform one or more map tasks or reduce tasks in parallel.

[0098] Specifically, the management node 100 and the computing node 200 can store the computation result to a remote storage device 300 through a network. When the storage device 300 is close to the management node 100 and the computing node 200, the storage device 300 can also communicate with the management node 100 and the computing node 200 through a backplane, a network cable or a non-network cable.

[0099] For example, the management node 100 and the computing node 200 can be servers. As shown in FIG. 2, each computing node 200 can include one or more processors 210 and a communication interface 220 connected to the processor 210. The management node 100 can have the same structure as the computing node 200 shown in FIG. 2, and also include a processor and a communication interface.

[0100] The communication interface 2200 can be used to receive messages or data, such as input data of a map task or input data of a reduce task.

[0101] The communication interface 2200 can also be configured to send messages or data, for example, to send output data of a map task or output data of a reduce task, etc.

[0102] The processor 210 is a control center of the computing node (or the management node), and is connected with various parts of the computing node (or the management node) through various interfaces and lines, such as the communication interface 220, a baseboard management controller (not shown in FIG. 2), etc.

[0103] In an embodiment of the present application, the processor 210 can be a central processing unit (CPU) configured to implement related processing of a computing task. Alternatively, each processor 210 can include one or more processing units. The processor 210 can also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, etc.

[0104] The computing node (or the management node) can also include a memory (not shown in FIG. 2), which can be configured to store software programs and modules. The processor 210 can implement various functional applications and data processing of the computing node (or the management node) by reading the software codes and modules stored in the memory. The memory herein is an internal memory, or a solid state drive (SSD), a hard disk drive (HDD) or other persistent storage medium. The internal memory is also referred to as a main memory.

[0105] The storage device can be a single storage apparatus, or a storage system composed of multiple storage apparatuses (such as a storage server, a distributed storage device, a SAN storage device), and the multiple storage apparatuses constituting the storage system can be the memory in the computing node. As shown in FIG. 3, the storage device can logically include a processing unit 310, a storage unit 320 and a communication unit 330. Correspondingly, the storage device physically includes one or more processors, memories and external interfaces, and the memories are, for example, phase-change memories (PCM), SSDs or hard disks HDDs. Further, the storage device also includes an internal memory.

[0106] The communication unit 330 can transmit data to the management node or the computing node through a network, for example, to transmit input data of a map task or output data of a reduce task, etc. The storage unit 320 can store the data received from the management node or the computing node, the processing unit 310 can control the data transmission between the communication unit 330 and the management node or the computing node, and process the data stored in the storage unit 320, etc.

[0107] In the embodiments of the present application, the storage device can be a Hadoop Distributed File System (HDFS), a NAS file storage device, a block storage device, or an object store in the cloud.

[0108] The shuffle process will be described in detail as follows:

[0109] Generally, shuffle is divided into two parts: data preparation performed by a map task (also referred to as shuffle write) and data pulling performed by a reduce task (also referred to as shuffle read).

[0110] After shuffle write processes the input data (split), intermediate data is generated. The data in the intermediate data is the same as that in the split, but the data is sorted differently. The intermediate data includes multiple data segments. The data in the same data segment belongs to the same partition, and will be pulled to the local by the same reduce task. That is, the intermediate data generated by each map task is sorted and stored according to the partition. Based on this, the map task usually writes the intermediate data into the corresponding data segment of the corresponding data file according to the partition id, and records the offset of each data segment in the data file in the index file. Each reduce task is usually used to pull the data of the specified data segment. Based on this, shuffle read can read (pull) the data of the specified data segment from the data file corresponding to each map task according to the offset of the specified data segment recorded in the index file. FIG. 4 shows an example of the data file and the index file.

[0111] As shown in FIG. 4, the split input data of the map task includes 7 pieces of data, i.e., data1-data7. The data file generated by the map task includes 3 data segments (data segment 1, data segment 2, and data segment 3), and the data in the 3 data segments belongs to partition 1, partition 2, and partition 3 respectively. Then, the data data1, data3, and data4 included in data segment 1 belong to partition 1; the data data2 and data5 included in data segment 2 belong to partition 2; and the data data6 and data7 included in data segment 3 belong to partition 3. The number 0 in the index file represents the start position of data segment 1; the number 3 represents the offset of data segment 2; the number 5 represents the offset of data segment 3; and the number 7 represents the total number of data in the data file.

[0112] It should be noted that the partition is a logical concept, and the data segment in the intermediate data logically belongs to a certain partition, which means that it will be processed by a designated reduce task.

[0113] FIG. 5 introduces a process diagram of shuffle. As shown in FIG. 5, there are 2 splits to be processed, i.e., split 1 and split 2. Split 1 is the input data of map task 1; and split 2 is the input data of map task 2. The number of reduce tasks is 3, i.e., reduce task 1, reduce task 2, and reduce task 3.

[0114] In FIG. 5, map task 1 and map task 2 are executed by a map node (a computing node for executing a map task), and reduce task 1, reduce task 2, and reduce task 3 are executed by a reduce node (a computing node for executing a reduce task).

[0115] It should be noted that the data of a map task executed by a map node can be one or more, and the number of reduce tasks executed by a reduce node can be one or more. A computing node can execute both map tasks and reduce tasks. The embodiments of the present application do not limit this.

[0116] In the shuffle write stage, each map task processes one shard. In FIG. 5, map task 1 processes shard 1 and map task 2 processes shard 2. After the shuffle write is performed, the map tasks produce an output file (containing intermediate data), each of which includes three data segments (different data segments belong to different partitions), which are pulled by three reduce tasks for processing.

[0117] Specifically, the intermediate data produced by map task 1 includes data segment 1 (containing data1, data3, data4, belonging to partition 1), data segment 2 (containing data2, data5, belonging to partition 2), and data segment 3 (containing data6, data7, belonging to partition 3); the output file produced by map task 2 includes data segment 1 (containing data8, data14, belonging to partition 1), data segment 2 (containing data9, data11, belonging to partition 2), and data segment 3 (containing data10, data12, data13, belonging to partition 3). Among them, the data segment belonging to partition 1 is pulled by reduce task 1 for processing; the data segment belonging to partition 2 is pulled by reduce task 2 for processing; and the data segment belonging to partition 3 is pulled by reduce task 3 for processing.

[0118] In the shuffle read stage, each reduce task respectively pulls the data of the specified data segment in the intermediate data produced by two map tasks through shuffle read as the input data of the downstream operator of the reduce task. Among them, the data obtained by reduce task 1 includes the data segment belonging to partition 1 in map task 1 and map task 2, the data obtained by reduce task 2 includes the data segment belonging to partition 2 in map task 1 and map task 2, and the data obtained by reduce task 3 includes the data segment belonging to partition 3 in map task 1 and map task 2.

[0119] Specifically, as shown in FIG. 5, the data pulled by reduce task 1 includes data1, data3, data4, data8, data14, which will be the input data of the downstream operator of reduce task 1; the data pulled by reduce task 2 includes data2, data5, data9, data11, which will be the input data of the downstream operator of reduce task 2; and the data pulled by reduce task 3 includes data6, data7, data10, data12, data13, which will be the input data of the downstream operator of reduce task 3.

[0120] The downstream operators of the reduce task 1 process the input data and output data part 1; the downstream operators of the reduce task 2 process the input data and output data part 2; and the downstream operators of the reduce task 3 process the input data and output data part 3. The downstream operators are, for example, Hash Aggregate or Hash Join.

[0121] The specific steps of the scheme shown in FIG. 5 are as follows:

[0122] 1. The map task calculates the partition ID from the shard.

[0123] 2. The map task generates the intermediate data according to the partition ID. The intermediate data includes three data segments, and the data in the three data segments belong to partition 1, partition 2 and partition 3 respectively.

[0124] 3. The map task calculates the size information of the data segment according to the intermediate data.

[0125] 4. The map task sends the size information of the data segment to the management node.

[0126] 5. The management node makes an AQE decision (i.e., whether to adjust the number of reduce tasks and how to adjust the number of reduce tasks) according to the size information sent by all map tasks. The AQE decision will be described in detail below.

[0127] 6. The reduce task pulls the data belonging to the specified partition to the local memory according to the indication of the management node.

[0128] The above six steps are executed sequentially. All map tasks generate the intermediate data and persist the intermediate data to the disk file system, and then report the size information of each data segment to the management node. After the management node aggregates the size information sent by all map tasks, it knows how much data belongs to each partition, and then makes a decision on whether to adjust the number of reduce tasks. After the management node makes the decision, the reduce task can pull the data from the disk of the map node. This scheme does not support streaming reading.

[0129] It should be noted that, as described above, the number of reduce tasks that can be processed by one reduce node can be multiple, and therefore, increasing or decreasing the number of reduce tasks can also increase or decrease the number of reduce tasks in the same reduce node, or increase or decrease the number of reduce tasks in different reduce nodes.

[0130] In one solution, the intermediate data is temporarily stored in the memory of the map node. This solution can reduce the time consumed for storing the intermediate data compared to persisting the intermediate data to the disk file system; and once the intermediate data is stored in the memory of the map node, the reduce task can read the data belonging to the specified partition remotely to the local without waiting for the shuffle read to be executed after the shuffle write of all the map tasks is completed, thus supporting the streaming read.

[0131] The process of the streaming read is described below.

[0132] Similarly, assume that there are 2 map tasks and 3 reduce tasks, namely map task 1, map task 2, reduce task 1, reduce task 2, and reduce task 3. The process of the streaming read is as follows:

[0133] Once the intermediate data generated by the map task 1 is stored in the memory of the map node, the reduce task 1, the reduce task 2, and the reduce task 3 can read the data segment belonging to the specified partition to the local memory.

[0134] Once the intermediate data generated by the map task 2 is stored in the memory of the map node, the reduce task 1, the reduce task 2, and the reduce task 3 can read the data segment belonging to the specified partition to the local memory.

[0135] Without waiting for the map task 1 and the map task 2 to store the intermediate data to the memory of the map node, the reduce task (shuffle read) is started to be executed.

[0136] Since the data segments in the intermediate data can be unbalanced in data volume, the data volume of the data belonging to a certain partition can be too large or too small, that is, the data volume pulled by a certain reduce task can be too large or too small. If it is too large, the execution time of the downstream operator of the reduce task will be too long, and the downstream operators of other reduce tasks can need to wait for the downstream operator of the reduce task to be executed. If it is too small, the execution time of the downstream operator of the reduce task will be very short, causing waste of computing resources.

[0137] In order to solve the technical problem caused by too large or too small data volume of a part of partitions, the spark computing framework supports an adaptive query execution (AQE) function. The AQE function includes: in a shuffle write stage, a management node aggregates size information of each data segment in intermediate data generated by each map task, and finally calculates data size of all data belonging to each partition, so as to determine whether the number of reduce tasks needs to be adjusted. If the data volume of a certain partition is too large, the number of reduce tasks is increased, and the data in the partition is distributed to multiple reduce tasks, so as to prevent a downstream operator of a certain reduce task from spending too long time. If the data volume of some partitions is too small, the number of reduce tasks is reduced, and the partitions are distributed to the same reduce task, so as to prevent waste of computing resources. Then, the management node instructs the reduce task to pull corresponding data. It should be noted that each reduce task corresponds to a downstream operator, and the number of downstream operators is increased / decreased when the number of reduce tasks is increased / decreased.

[0138] In the above scheme supporting stream reading, the AQE function is not supported, and the reason is as follows: after the map task stores the intermediate data in the memory of the map node, the intermediate data is read by the reduce task. Therefore, before the reduce task (shuffle read) is executed, the size information of each partition cannot be obtained, and it is impossible to determine whether the data volume of the partition is too large or too small, and then the number of reduce tasks cannot be adjusted, that is, the AQE function cannot be supported, thereby affecting the shuffle performance.

[0139] Based on this, an embodiment of the present application provides a partition management method. In the method, the map task first calculates each data in the shard to determine the partition to which each data in the shard belongs, and then counts the number / data volume of the data belonging to the same partition to obtain a data entry statistical result. Before performing partition copy (concentrating storage of the data belonging to the same partition) on the shard, the data entry statistical result is sent to the management node, so that the management node can timely make a decision on whether to adjust the number of reduce tasks.

[0140] The scheme provided by the embodiment of the present application will be described below with reference to FIG. 6. As shown in FIG. 6, the partition management method of the embodiment of the present application includes steps 601 to 603.

[0141] In step 601, the map task determines the partition to which each data entry in the split belongs, wherein the number of partitions is multiple.

[0142] Specifically, the map task calculates each data row in the split to determine the partition to which each data in the split belongs.

[0143] As mentioned above, the number of partitions is multiple, and each partition corresponds to a reduce task, which is used to process data belonging to the corresponding partition. Specifically, the reduce task is used to pull data belonging to the specified partition to the local memory.

[0144] Specifically, the source of the split processed by the map task is various. In one example, if the split is stored in the map node executing the map task, the map task obtains the split from the memory of the map node. In another example, if the split is stored in another computing node, the map task obtains the split from the other computing node.

[0145] In some embodiments, the map task obtains metadata after calculating the split. The metadata is used to indicate the partition to which each data in the split belongs. Through the indication of the metadata, the data in the split is copied according to the partition to obtain intermediate data. For example, as shown in FIG. 4, after performing the partition copy on the split, the data file is obtained, which is the intermediate data. It should be noted that the partition copy has not been performed on the split, and the intermediate data has not been obtained.

[0146] It can be understood that the partition copy, that is, the data belonging to the same partition is stored in a centralized manner. According to the indication of the metadata, after performing the partition copy on the split, the data belonging to the same partition is stored in a centralized manner. The centralized storage, for example, is stored in adjacent address space (physical space or logical space) in a physical medium, or is stored in the same physical medium, and the purpose is to enable the reduce task to quickly read the data belonging to the same partition.

[0147] It should be noted that the data structure of the metadata has various implementation manners.

[0148] In one example, the metadata includes multiple partition identifiers. Each data in the split can calculate a partition identifier. Therefore, each partition identifier in the metadata is in a one-to-one correspondence with each data in the split. The partition identifier is used to indicate to which partition the corresponding data belongs. The metadata of the embodiment of the application is described below in combination with FIG. 8.

[0149] Figure 8 illustrates two shards (shard 1 and shard 2) and two map nodes (map task 1 and map task 2). It is noted that the number of shards can be greater than 2 and the number of partitions in each shard can be greater than 3, which are not limited by the embodiments of the present application.

[0150] As shown in Figure 8, shard 1 contains 7 data (data 1 ~ data 7) and shard 2 contains 7 data (data 8 ~ data 14). After map task 1 processes shard 1, metadata 1 is obtained, which contains numbers 1, 2 and 3. Each data in shard 1 corresponds to a data identifier. After map task 2 processes shard 2, metadata 2 is obtained, which also contains numbers 1, 2 and 3. The partition identifier is used to indicate to which partition the corresponding data belongs. Number 1 indicates that the corresponding data belongs to partition 1; number 2 indicates that the corresponding data belongs to partition 2; and number 3 indicates that the corresponding data belongs to partition 3. As shown in Figure 8, data 1, data 3, data 4, data 8 and data 14 belong to partition 1; data 2, data 5, data 9 and data 11 belong to partition 2; and data 6, data 7, data 10, data 12 and data 13 belong to partition 3.

[0151] As mentioned above, the data in the shard is copied according to the partition identifier, and the intermediate data is obtained. Therefore, by counting the number of times that numbers 1, 2 and 3 appear, the number of data in each data segment of the intermediate data can be predicted.

[0152] In another example, the metadata includes a partition identifier and an address identifier. The partition identifier is used to indicate to which partition the data is allocated, and the address identifier is used to indicate the position (e.g. offset) of the data in the shard. The metadata of the embodiments of the present application is described below in conjunction with Figure 9. Similarly, Figure 9 illustrates two shards (shard 1 and shard 2) and two map nodes (map task 1 and map task 2). Map task 1 is used to process shard 1 and map task 2 is used to process shard 2. The number of shards can be greater than or less than 2, and the number of map tasks can be greater than or less than 2, which are not limited by the embodiments of the present application.

[0153] As shown in FIG. 9, the metadata 1 contains partition id and row id. The metadata 2 contains partition id and row id. The number of partition ids is the same as the number of partitions, and the number of row ids is the same as the number of data in the split. The row ids of the data belonging to the same partition correspond to the same partition id.

[0154] As shown in FIG. 9, the data in the 1st, 3rd and 4th rows (data1, data3, data4) in the split 1 belong to partition 1; the data in the 2nd and 5th rows (data2, data5) in the split 1 belong to partition 2; the data in the 6th and 7th rows (data6, data7) in the split 1 belong to partition 3. The data in the 1st and 7th rows (data8, data14) in the split 2 belong to partition 1; the data in the 2nd and 4th rows (data9, data11) in the split 2 belong to partition 2; the data in the 3rd, 5th and 6th rows (data10, data12, data13) in the split 2 belong to partition 3.

[0155] As mentioned above, the data in the split is partitioned according to the indication of the partition id and row id, and the intermediate data is obtained. Therefore, the number of row ids corresponding to each partition id can be used to predict how many data will be contained in each data segment of the intermediate data.

[0156] In some embodiments, the method used in the calculation of the data is a hash calculation method. The hash calculation method is as follows: first, the hash value of the shuffle key field in a piece of data is calculated, and then the hash value is subjected to a modulo operation to obtain the partition id. The divisor in the modulo operation is the number of partitions, i.e., the number of reduce tasks. The key field is one of the multiple fields contained in the data, and the data is classified by the field. For example, assuming that a piece of data contains three fields: name, student ID and score, if the data in all splits is classified according to the score, with data having a score greater than or equal to 60 being classified as one category and data having a score less than 60 being classified as another category, then the score is the shuffle key field.

[0157] In step 602, the map task counts the data belonging to each partition to obtain a data entry statistical result. The data entry statistical result can reflect the data volume of the data belonging to each partition.

[0158] Specifically, the map task counts the first type of data entries belonging to the first partition in the shard, and obtains a first data amount of the first type of data entries, the first partition being one of the multiple partitions, the first data amount being used to indicate the number or size of the first type of data entries.

[0159] The map task performs the same action on the data entries corresponding to other partitions in the multiple partitions, and obtains multiple first data amounts, each first data amount corresponding to a partition and being used to indicate the number or size of the data entries belonging to the partition. The first data amounts corresponding to all the partitions are collectively referred to as a data entry statistical result. For example, as shown in FIG. 4, there are three partitions, and then the map task obtains three first data amounts after counting the data entries according to the partitions, the three first data amounts respectively corresponding to partition 1, partition 2, and partition 3, and respectively being used to indicate the number or size of the data entries belonging to partition 1, partition 2, and partition 3.

[0160] It should be noted that the map task does not perform partition copy on the shard at this time, and has not obtained the intermediate data.

[0161] In some embodiments, the data possessed by each partition can be counted according to the metadata.

[0162] Specifically, the data entry statistical result can be the number of data entries, or the data size of the data.

[0163] In one example, the data amount is the number of all data belonging to the respective partitions.

[0164] As shown in FIG. 8, the number of times that the numbers 1, 2, and 3 appear corresponds to the number of data entries of the data belonging to partition 1, partition 2, and partition 3. The map task 1 counts that the number 1 appears 3 times, the number 2 appears 2 times, and the number 3 appears 2 times. The map task 2 counts that the number 1 appears 2 times, the number 2 appears 2 times, and the number 3 appears 3 times. The number of data entries of partition 1, partition 2, and partition 3 can be obtained by combining the number of times that the numbers 1, 2, and 3 appear counted by the map task 1 and the map task 2.

[0165] As shown in FIG. 9, the map task 1 can count the number of data belonging to the partition 1, the partition 2 and the partition 3 according to the metadata 1. In the map task 1, the data belonging to the partition 1 contains 3 data; the data belonging to the partition 2 contains 2 data; and the data belonging to the partition 3 contains 2 data.

[0166] As shown in FIG. 9, the map task 2 can count the number of data belonging to the partition 1, the partition 2 and the partition 3 according to the metadata 2. In the map task 2, the data belonging to the partition 1 contains 2 data; the data belonging to the partition 2 contains 2 data; and the data belonging to the partition 3 contains 3 data.

[0167] The number of data of the partition 1, the partition 2 and the partition 3 can be obtained according to the number of data counted by the map task 1 and the map task 2.

[0168] In another example, the data amount is the data size of all data belonging to each partition.

[0169] As described above, after the map task counts the number of data of each partition, the map task can estimate the data size of all data belonging to each partition according to the number of bytes corresponding to the data type multiplied by the number of data. When the column data in the data is all fixed-length data type (int8, int16, etc.), the size of each data is also a fixed value, so the data size of all data belonging to each partition can be accurately calculated; when some column data in the data is a variable-length type (string), the estimated value of the number of bytes of one data can be obtained according to the metadata of the shard or the sampling estimation method, so the data size of all data belonging to each partition can be approximately estimated.

[0170] In step 603, the map task sends the first data amount to the management node.

[0171] Specifically, the map task sends the first data amount (or the data entry counting result) corresponding to all partitions to the management node.

[0172] Specifically, before performing the partition copy action on the shard, the map task feeds back the data entry counting result to the management node, so that the management node can determine whether to adjust the number of reduce tasks according to the data entry counting result, for example, whether to increase or decrease the number of reduce tasks.

[0173] The partition copy (or the partition copy action) can be understood as copying the data in the shard, so that the data belonging to the same partition is stored adjacently (or concentratedly), obtaining the data file (intermediate data) as shown in FIG. 4.

[0174] Specifically, after the map task counts the data amount of the data belonging to each partition, the counted data amount is immediately sent / feedback to the management node. As shown in FIG. 10, the map task 1 sends the data amount 1 to the management node 100, and the map task 2 sends the data amount 2 to the management node 100.

[0175] It should be noted that FIG. 10 only shows an example in which the data amount is the number of data. The map task 2 can also estimate the size of the data based on the software program, take the estimated data size as the data amount, and directly send the estimated data size to the management node.

[0176] It should be noted that all map tasks will perform the above steps 601 to 603, count the data amount of the data belonging to each partition in the shard processed by the node, and send the data amount to the management node 100. The management node 100 collects the data amount to count the data amount of each partition.

[0177] The foregoing part describes the steps 601 to 603 in detail, and the shuffle process realized based on the method shown in FIG. 6 is described in detail in combination with FIG. 7.

[0178] As shown in FIG. 7, in the shuffle write stage, the map task calculates each data in the shard to obtain metadata (e.g., partition ID) (step ①), and the map task can copy the data in the shard according to the indication of the metadata (step ⑤) to form intermediate data. According to this logic, the method of the embodiment estimates / counts the data amount of the data belonging to each partition according to the metadata before the metadata is calculated (step ①) and the intermediate data is generated (step ⑤) (step ②), and the data amount is summarized to the management node (step ③), so that the management node can determine whether the number of reduce tasks needs to be adjusted according to the received data amount (step ④); without waiting for all map tasks to end (generate intermediate data), the data amount of each partition is informed to the management node, so that the management node can make a decision (AQE decision) on whether to adjust the number of reduce tasks before the map task generates the intermediate data. At the same time, since the map task is copied according to the indication of the metadata, the AQE decision and the partition copy action of the map task are performed in parallel (steps ④ and ⑤ are performed in parallel). Once the intermediate data is generated by a map task, the reduce task can pull (read) the data segment belonging to the specified partition to the local, without waiting for all map tasks to generate intermediate data before starting to perform the data pulling task (shuffle read), so as to support streaming reading. In summary, the partition management method of the embodiment of the application supports AQE and streaming reading at the same time.

[0179] The foregoing steps 601 to 603 are performed by the map task, and the following step 604 to be performed by the management node is described.

[0180] In some embodiments, as shown in FIG. 6, the partition management method further includes step 604. It should be noted that step 604 is an optional step.

[0181] In step 604, the management node determines whether the reduce task needs to be adjusted according to the data entry statistical result sent by the map task. For example, whether to increase / decrease the number of reduce tasks.

[0182] Specifically, after receiving the data entry statistics sent by each map task, the management node can count the data volume that each reduce task will pull, i.e., the data volume of each partition. The data volume that the reduce task pulls directly determines the execution time of the downstream operator of the reduce task. Therefore, the management node can estimate whether the execution time of the downstream operator of each reduce task is balanced. If balanced, the management node makes a decision not to adjust the number of reduce tasks; if not balanced, the management node makes a decision on whether to adjust the number of reduce tasks and how to adjust.

[0183] It should be noted that the management node 100 can quickly determine whether the number of reduce tasks needs to be adjusted after receiving the data entry statistics sent by each map task. If adjustment is needed, resources can be quickly applied to increase the number of reduce tasks, or resources can be released to reduce the number of reduce tasks.

[0184] In one example, as shown in FIG. 10, the data volume received by the management node 100 is the number of data entries per partition, based on which the management node 100 counts that reduce task 1 will pull 5 data entries, reduce task 2 will pull 4 data entries, and reduce task 3 will pull 5 data entries. It should be noted that at this time, the reduce tasks have not pulled data.

[0185] In another example, the management node 100 receives the number of data entries as shown in FIG. 10, and in combination with the data type, the data size of all data that reduce task 1, reduce task 2, and reduce task 3 will pull can be estimated.

[0186] In yet another example, the data volume received by the management node 100 is the data size of all data belonging to each partition, based on which the data size of all data that reduce task 1, reduce task 2, and reduce task 3 will pull can be counted.

[0187] In some embodiments, the management node 100 determines whether the number of reduce tasks needs to be adjusted according to the data volume that each reduce task will pull.

[0188] In one example, as shown in FIG. 10, the number of data entries that each reduce task will pull is balanced, and therefore the management node 100 decides not to adjust the number of reduce tasks.

[0189] In another example, the management node 100 determines that the data amount to be pulled by one or more reduce tasks is large (e.g., the number of data is large), and increases the number of reduce tasks, and assigns the data in the partition with the large data amount to different reduce tasks.

[0190] Specifically, as shown in FIG. 11, assuming that there are originally three reduce tasks (reduce 1-reduce 3), the management node 100 determines that the number of data to be pulled by reduce 1, reduce 2, and reduce 3 is 10, 4, and 5 respectively, and determines that the downstream operators of reduce 1 need to spend a long time to execute because reduce 1 will pull too many data. The downstream operators of reduce 2 and reduce 3 have finished execution, and need to wait for the execution of the downstream operator of reduce 1. Based on this, the management node 100 increases the number of reduce tasks to four (reduce 1, reduce 2, reduce 3, and reduce 4). The partition with the data amount of 10 is pulled by reduce 1 and reduce 4, each of which pulls a part of the data; for example, reduce 1 will pull 5 data, and reduce 4 will pull 5 data. The number of data to be pulled by reduce 2 is still 4, and the number of data to be pulled by reduce 3 is still 5; the data amount to be pulled by each reduce task is balanced.

[0191] In yet another example, the management node 100 determines that the data amount to be pulled by some reduce tasks is small (e.g., the number of data is small), and reduces the number of reduce tasks, and assigns the data to the same reduce task.

[0192] Specifically, as shown in FIG. 12, it is assumed that there are five reduce tasks (reduce 1-reduce 5). The management node 100 counts that the number of data to be pulled by reduce 1, reduce 2, reduce 3, reduce 4, and reduce 5 is 6, 5, 5, 2, and 2 respectively, and then the management node 100 judges that the number of data to be pulled by reduce 4 and reduce 5 is too small, and reduce 4 and reduce 5 can be executed soon, causing resource waste. Based on this, the management node 100 reduces the number of reduce tasks, and after adjustment, reduce 4 pulls the data that should be pulled by reduce 4 and reduce 5. After adjustment, the number of data to be pulled by reduce 1 is 6; the number of data to be pulled by reduce 2 is 5; the number of data to be pulled by reduce 3 is 5; the number of data to be pulled by reduce 4 is 4; and the amount of data to be pulled by each reduce task is balanced.

[0193] It should be noted that the above examples only describe whether the number of reduce tasks needs to be adjusted and how to adjust, and the number of data involved in the above description in the shard, the increase and decrease strategy of the reduce task do not constitute a limitation of the adjustment strategy.

[0194] The foregoing embodiments mainly describe the first few steps of the shuffle process (the map task calculates and counts the shards to obtain the data entry statistical result, and sends the data entry statistical result to the management node; the management node makes a decision on whether to adjust the reduce task according to the received data entry statistical result), and the complete flow of the shuffle process is described next.

[0195] The foregoing describes the tasks performed by the map task and the management node, and the entire shuffle process is described next in conjunction with FIG. 13.

[0196] FIG. 13 is a flowchart of a shuffle process provided by an embodiment of the present application, and the shuffle process includes steps 801-809.

[0197] The steps 801, 802, 803, and 804 are similar to the steps 601, 602, 603, and 604 in the foregoing FIG. 6 respectively, and will not be described here again.

[0198] How to adjust the number of reduce tasks (step 806) is similar to the examples shown in the foregoing FIG. 11 and FIG. 12, and will not be described here again.

[0199] Step 805, the map task performs partition copy on the shard to obtain intermediate data.

[0200] In some embodiments, the map task performs partition copy on the data in the shard according to the indication of the metadata, so that the data belonging to the same partition are stored in the same set.

[0201] It should be noted that the AQE decision of the management node (including steps 804 and 806) and the partition copy of the map task can be performed in parallel, and the AQE decision does not affect the partition copy action of the map task. After the map task sends the data entry statistics result to the management node, the map task can perform partition copy on the shard according to the indication of the metadata.

[0202] Step 807, the map task sends the storage address of the data to the management node.

[0203] Specifically, the storage address of the data sent by the map task to the management node is the latest storage address of the data. If the data is not subjected to the partition copy action, the latest storage address is the storage address of the data in the shard; if the data is subjected to the partition copy action, the latest storage address is the storage address of the data in the intermediate data; if the data is encapsulated into a dictionary, the latest address is the storage address of the data indicated in the dictionary.

[0204] Specifically, the map task sends the storage address of the data to the management node as the access address.

[0205] Step 808, the management node sends the storage address / access address of the data fed back by the completed map task to the reduce task.

[0206] Specifically, different reduce tasks pull different data, and therefore, the storage address / access address sent by the management node to the reduce task is only the storage address / access address of the data to be pulled by the reduce task (i.e., the specified storage address / access address of the data).

[0207] Step 809, the reduce task pulls the specified data to the local memory according to the indication of the storage address / access address sent by the management node.

[0208] Specifically, the first reduce task sends a data entry read request based on the access address to the map task. The first reduce task is one of all the reduce tasks. The first reduce task sends the data entry read request to all the map tasks to read the specified data entry in each map task to the local. All the reduce tasks perform the action as the first reduce task.

[0209] The specific process of the reduce task pulling data is that the reduce task sends a read data request to the management node, and the request contains the partition information (which partition the data belongs to, or the partition id) to be read; the management node sends the storage address of the data belonging to the partition to the reduce task; the reduce task pulls the specified data to the local memory according to the indication of the storage address. Repeat the above process until all the shuffle read is completed, and there is no data to read.

[0210] Specifically, the reduce task sends a data read request to the map task based on the access address; the map task provides the response data entry to the reduce task in response to the data entry read request based on the access address of the reduce task.

[0211] It should be noted that the reduce task can pull data according to the indication of the received storage address without waiting for all the map tasks to complete.

[0212] It should be noted that the above step 805 is optional in order to improve the shuffle efficiency. Specifically, as described above, the essence of partition copy is to copy data from one location to another, which is time-consuming. Therefore, the partition copy is one of the bottlenecks of shuffle. In order to reduce the partition copy, the embodiment of the present application provides another flowchart of the shuffle process as shown in FIG. 14.

[0213] As shown in FIG. 14, the shuffle process of the embodiment of the present application includes steps 901 to 908.

[0214] In the embodiment of the present application, the map node stores the shards in the data cache pool, partitions (or does not partition, whether to partition is determined according to the data volume) the shards in the data cache pool according to the indication of the metadata, to obtain intermediate data, and the intermediate data is still stored in the data cache pool. The reduce node can pull the data of the specified partition from the data cache pool according to the indication of the management node.

[0215] The data cache pool in the embodiment of the present application is located in the second layer in the storage system as shown in FIG. 15. It has a faster access speed relative to the external storage in the third layer. Compared with persisting intermediate data to the disk file system, the speed of writing and reading data is faster, the IO overhead is smaller, and the shuffle efficiency (partition copy efficiency, pull efficiency) is high. In some embodiments, the data cache pool is included in the storage device 300 as shown in FIG. 1.

[0216] In some embodiments, the data cache pool is a compute express link (CXL) / unified bus (UB) based data cache pool. Compared with the RDMA data cache pool, the CXL / UB data cache pool has the advantage of global addressing, and any node can access the data in the data cache pool without serialization, and can read and write data like accessing local memory.

[0217] As shown in the storage system in FIG. 15, from the first layer to the third layer, the storage capacity increases step by step, the access speed decreases step by step, and the cost decreases step by step. As shown in FIG. 15, the first layer includes registers and caches located in the CPU, and the caches can include level 1 cache, level 2 cache and level 3 cache. The memory included in the second layer can be used as the main memory of the computer system. For example, DRAM, DDR SDRAM, storage-class memory (SCM), persistent memory (PMEM). The main memory can be referred to as main memory or memory, i.e., the memory that exchanges information with the CPU. The memory included in the third layer can be used as the auxiliary memory of the computer system. For example, network memory, SSD, HDD. The auxiliary memory can be referred to as auxiliary storage or external storage. Compared with the main memory, the external storage has a large storage capacity and a slow access speed. It can be seen that the closer the memory is to the CPU, the smaller the capacity, the faster the access speed, the larger the bandwidth, and the lower the delay. Therefore, the memory included in the third layer stores data that is not frequently accessed by the CPU, thereby improving the reliability of the data. The memory included in the second layer can be used as a cache device to store data frequently accessed by the CPU, thereby significantly improving the access performance of the system.

[0218] The steps 901 to 908 of the embodiment of the present application are introduced below.

[0219] In step 901, the map node receives the map task sent by the management node 100, and the map task includes the shard (input data) and the number of partitions to be divided (the number of partitions).

[0220] Step 902, the map task in the map node calculates the shard to obtain metadata of the shard. Step 902 is similar to step 601, and thus is not described herein.

[0221] Step 902-1, the map task in the map node stores the shard and the metadata in the data cache pool 400. It is to be noted that the shard stored in the data cache pool 400 has not been subjected to partition copy in this step.

[0222] In some embodiments, a first area is divided in the data cache pool 400, and the first area is used to store shards.

[0223] In the embodiments of the present application, the shard is stored in the data cache pool 400, and thus the storage speed is faster than that of storing data in a disk.

[0224] Step 903, the map task in the map node estimates the data volume of the data belonging to each partition according to the metadata; and step 904, the map task in the map node feeds back the data volume to the management node 100. Steps 903 and 904 are similar to steps 602 and 603 respectively, and thus are not described herein.

[0225] Step 905, the management node 100 makes an AQE decision according to the data volume fed back by the map node. The AQE decision is used to determine how the reduce task pulls the data of the corresponding partition.

[0226] It is to be noted that there are multiple map tasks, and the management node 100 needs to aggregate the data volume provided by all the map tasks to make the AQE decision.

[0227] In some embodiments, the AQE decision includes two kinds. The first kind is that the management node 100 determines not to adjust the number of reduce tasks according to the obtained data volume. The second kind is that the management node 100 determines to change the number of reduce tasks according to the obtained data volume. How the management node 100 makes the AQE decision has been described in step 604, and thus is not described herein.

[0228] Step 906, the map node performs partition copy according to the indication of the metadata to obtain intermediate data. The map node performs partition copy, that is, according to the indication of the metadata, the data stored in the data cache pool 400 is copied to a specific area in the data cache pool 400 according to the partition. The specific area has a one-to-one correspondence with the partition.

[0229] Step 907, the reduce node reads the data in the data cache pool according to the indication of the AQE decision. The reduce node pulls the data, that is, according to the indication of the AQE decision, the map node copies the data in the specified area to the local.

[0230] As an alternative, the management node can generate multiple AQE tasks according to the AQE decision, each reduce task corresponding to an AQE task. The management node sends the AQE task to the corresponding reduce task. The AQE task instructs the corresponding reduce task to pull the specified data in the data cache pool 400.

[0231] In some embodiments, multiple second areas are divided in the data cache pool 400, and the multiple second areas and the multiple partitions have a one-to-one correspondence. Each second area is used to store data belonging to the same partition.

[0232] FIG. 16 shows an example of a data cache pool containing a first area and a second area. As shown in FIG. 16, the data cache pool 400 includes a first area 401 and multiple second areas (402a, 402b, 402c). Among them, the first area 401 is used to store shards from the map node. The second area 402a is used to store data belonging to partition 1, the second area 402b is used to store data belonging to partition 2, and the second area 402c is used to store data belonging to partition 3.

[0233] The map node performs partitioned copying of the shard, that is, according to the indication of the metadata, the map node copies the data of the shard in the first area to the specified second area (also referred to as the specified area). FIG. 17 shows an example of copying data.

[0234] As shown in FIG. 17, the first area 401 stores shard 1 and shard 2. Assuming that the AQE decision indicates that the number of reduce tasks does not need to be adjusted. data1, data3, data4 belong to partition 1; data2, data5 belong to partition 2; data6, data7 belong to partition 3. Then, the map task copies data1, data3, data4 to the second area 402a; copies data2, data5 to the second area 402b; copies data6, data7 to the second area 402c. Through the above process, the intermediate data (the three parts of data stored in 402a, 402b, 402c, that is, three data segments) is obtained.

[0235] Step 908: The reduce task in the reduce node pulls the data in the corresponding second area to local. This step is similar to step 809, and thus is not described here.

[0236] Suppose there are 2 map tasks (map task 1 and map task 2) in the map node in FIG. 14, and 3 reduce tasks (reduce task 1, reduce task 2, reduce task 3) in the reduce node. Since the embodiment of the present application supports stream reading, if map task 1 generates intermediate data first, each reduce task can pull the data in the corresponding partition to the local memory of the reduce node after map task 1 generates the intermediate data. There is no need to wait for map task 2 to generate intermediate data. For example, reduce task 1 pulls data in the second area 402a to the local memory; reduce task 2 pulls data in the second area 402b to the local memory; and reduce task 3 pulls data in the second area 402c to the local memory.

[0237] In some embodiments, in order to further improve the partition efficiency and the pulling efficiency, the foregoing step 906 is an optional step. For example, if a certain shard contains a small number of data, the reduce task can directly pull the data in the shard to local according to the indication of the metadata (for example, the foregoing partition id and row id, at this time the management node does not adjust the number of reduce tasks). There is no need for the map task to perform partition copy (or memory copy) on the shard, and the shuffle efficiency is improved.

[0238] As shown in FIG. 18, the shard containing a small number of data is located in the data cache pool, and when the reduce task reads the data belonging to partition 1, the reduce task can directly read according to the row id (row number) stored in the metadata. The same is true for partition 2 and partition 3.

[0239] Since the number of data in the shard is small, and it is known to which partition each data belongs, the next target data can also be prefetched when the current row of data is pulled, thereby further improving the pulling efficiency.

[0240] In some embodiments, the data cache pool includes a volatile cache pool (e.g., a memory pool) and a non-volatile cache pool (e.g., a PMEM pool). As shown in FIG. 19, the memory pool includes one or more memory terminals, which can be the memory of a computing node or the memory of a separate storage device; the PMEM pool includes one or more PMEM terminals, which can be the PMEM terminal of a computing node or the PMEM of a separate storage device. The configuration of the data cache pool is performed by the management node / computing node.

[0241] The memory terminal can be separated from the computing node or a separate memory pool. It is characterized by fast access speed and high price, but the data is easy to lose after power failure, and once the shuffle data is lost, the entire task needs to be recalculated.

[0242] The PMEM pool is a shared memory resource composed of all or part of the persistent memory provided by each computing node, which is characterized by higher latency than ordinary memory and lower price than ordinary memory, but has non-volatility, high density, byte addressability, and faster read-write speed than disk. In the embodiments of the present application, the memory pool and the PMEM pool are used to form the data cache pool, which avoids persisting data to the disk and can solve the IO overhead caused by reading and writing the disk.

[0243] The above describes the characteristics of the memory pool and the PMEM pool, and how to configure the memory pool / PMEM pool is described below in conjunction with FIGS. 20-22.

[0244] As shown in FIG. 20, the configuration of the data cache pool is performed by the cache pool manager in the management node (or computing node), which specifically includes setting the size of the memory pool / PMEM pool and mapping the host-managed device memory (HDM) of the memory terminal / PMEM terminal to the host physical address (HPA) of the management node. The CPU of the management node and the memory of the memory terminal / PMEM terminal support new high-speed interconnection protocols (CXL / UB, etc.).

[0245] As described above, the memory pool / PMEM pool based on CXL / UB is characterized by global addressing, and all computing nodes can read the data in the memory pool / PMEM pool without the need for network communication; as intermediate storage in shuffle, it can avoid data serialization / deserialization and reduce disk read-write overhead.

[0246] The memory of the memory terminal (or the PMEM of the PMEM terminal) is regarded as the host managed device memory (HDM) which can be mapped to the host physical address (HPA) in the memory system of the CPU of the management node as the dynamic random access memory (DRAM). Therefore, the application running on the CPU of the management node can access the HDM through the traditional memory instruction (load / store). Thanks to this feature, the HDM request is regarded as a traditional memory request, and the request is first cached in the CPU cache of the management node. Once the cache controller of the management node clears the line related to the HDM address space, one or more requests are packaged into flits by the root complex (RC) of the management node, and the RC transmits the flits to the memory terminal / PMEM terminal through the read / write interface provided by the high-speed interconnection protocol via the PCIe bus. The controller (memory engine / PMEM engine) inside the memory terminal / PMEM terminal converts the flits into multiple memory requests and processes them.

[0247] In some embodiments, the size of the memory pool / PMEM pool can be determined according to hardware resources or the size of the data to be processed.

[0248] Fig. 21 and Fig. 22 show two connection modes between the computing node and the memory pool (or PMEM pool): direct connection (as shown in Fig. 21) and switch connection (as shown in Fig. 22). In direct connection, each computing node is connected to each memory terminal in the memory pool or PMEM terminal in the PMEM pool through the PCIe bus. In switch connection, the computing node and the memory terminal / PMEM terminal are connected to the same switch network. It should be noted that the switch network can be a network composed of multiple switch bridges.

[0249] The difference between the two connection modes is:

[0250] (1) The direct connection requires the memory terminal / PMEM terminal to be of the multi-headed single logical device (MH-SLD) type, while the device type of the switch connection is multiple logical device (MLD).

[0251] (2) The number of devices in direct connection is small. Taking CXL as an example, the number of directly connected devices depends on the number of PCIe ports of the computing node and the memory terminal / PMEM terminal, and is usually several or dozens; while through switch connection, up to 4095 memory terminals / PMEM terminals can be connected.

[0252] (3) In the case that the number of nodes (the number of memory terminals / PMEM terminals) in the memory pool / PMEM pool is the same, the latency of direct connection is less than that of switch connection.

[0253] The embodiments of the present application are not limited to any connection mode, and the user can select a suitable connection mode according to the type of the device, the size of the memory pool / PMEM pool, and the like.

[0254] The above describes how to configure the data cache pool in combination with FIGS. 20 to 22, and the following introduces how to use the configured data cache pool to improve the shuffle efficiency in combination with FIG. 23.

[0255] In some embodiments, in order to further improve the partition efficiency and the pulling efficiency, as shown in FIG. 23, the partition management method of the embodiments of the present application includes steps 1001 to 1008.

[0256] The steps 1001 and 1002 are similar to the aforementioned steps 901 and 601, and the step 1003 is similar to the step 902-1, which are not described herein.

[0257] The step 1004, the map node (the map task therein) judges whether the shard meets the condition (the first condition) for storing in the memory pool; if yes, the step 1005-1 is executed; if not, the step 1005-2 is executed.

[0258] In some embodiments, the first condition can be that the data amount of the shard is small, or that the fault tolerance requirement is low. It can be understood that the specific first condition can also be set according to the actual situation.

[0259] The step 1005-1, the map node (the map task therein) stores the shard in the memory pool.

[0260] Compared with the PMEM pool, the memory pool has a smaller storage capacity and a faster access speed. Therefore, there are several benefits of storing the shard in the memory pool:

[0261] (1) The shard is stored in the memory pool, the storage speed is fast, and the execution efficiency of the map task is improved;

[0262] (2) After the shard is stored in the memory pool, there is no need for partitioning, and no data copying is needed, further improving the execution efficiency of the map task;

[0263] (3) The reduce task pulls data from the memory pool to the local, the pulling speed is fast, and the execution efficiency of the reduce task is improved;

[0264] (4) Since the execution efficiency of the map task and the execution efficiency of the reduce task are both high, the time of temporarily storing data in the memory pool is short, and the loss probability is low.

[0265] In summary, the small fragments are stored in the memory pool and directly pulled by the reduce task, which greatly improves the shuffle efficiency.

[0266] It can be understood that, since the fragments stored in the memory pool do not need to be copied, the second area described above does not need to be divided in the memory pool, and the memory pool can be regarded as the first area described above.

[0267] In some embodiments, a data amount threshold can be set, and if the data amount of the fragment is less than or equal to the threshold, the fragment is stored in the memory pool, and if the data amount is greater than or equal to the threshold, the fragment is stored in the PMEM pool. The threshold size is determined according to the task, and the embodiments of the present application do not limit the threshold size.

[0268] It should be noted that the map node can also determine whether to store the fragment in the memory pool through other parameters. For example, the re-computation cost after data loss, the fault tolerance requirement for data, and the like. In some implementations, if the re-computation cost after data loss is large, the fragment is stored in the PMEM pool, and if the cost is small, the fragment is stored in the memory pool. In some embodiments, if the fault tolerance requirement for data is high, the fragment is stored in the PMEM pool, and if the fault tolerance requirement for data is not high, the fragment is stored in the memory pool. For example, if complete fault tolerance is required, all fragments are placed in the PMEM pool for persistence.

[0269] Step 1005-2, the map node (the map task therein) stores the fragment in the PMEM pool. Specifically, the fragment is stored in the first area of the PMEM pool.

[0270] Step 1006, the map node (the map task therein) determines whether the data belonging to a certain partition in the fragment satisfies a second condition. If the second condition is not satisfied, step 1007-1 is performed, and if the second condition is satisfied, step 1007-2 is performed.

[0271] Specifically, the second condition is, for example, to determine whether the proportion of the data belonging to a certain partition in the fragment exceeds a threshold. If the proportion of the data belonging to a certain partition in the fragment exceeds a certain threshold, it indicates that the fragment has data skew. The partition meeting the condition can be referred to as a specific partition / skewed partition.

[0272] In the embodiments of the present application, for the partition with data skew, the partition copy is not performed, and the storage address of the data in the partition is directly stored in the corresponding second area, further reducing the number of data copying and improving the shuffle efficiency.

[0273] Step 1007-1, the map node (the map task therein) copies the data belonging to the corresponding partition in the fragment to the corresponding second area.

[0274] Specifically, the map node copies the data of the shard in the first region of the PMEM pool to the corresponding second region (buffer1 or buffer2 or buffer3) in the PMEM pool according to the indication of the metadata.

[0275] It can be understood that, since the shard stored in the PMEM pool is partitioned, the first region and the second region as described above are divided in the PMEM pool.

[0276] The buffers 1-3 as shown in FIG. 23 are pre-allocated storage spaces, i.e. the second regions as described above. The number of buffers is the same as the number of reduce tasks (before AQE decision), and each buffer is used to store data belonging to the same partition, and the size of the buffer is determined by the data size of the corresponding partition. As shown in FIG. 23, buffer1 is used to store data belonging to partition1, buffer2 is used to store data belonging to partition2, and buffer3 is used to store data belonging to partition3.

[0277] It can be understood that, since the management node has obtained the data size of all the map tasks, it can know the data size of each partition, and thus can pre-allocate storage spaces of corresponding sizes in the PMEM pool according to the data size of each partition, so as to store data of the same partition to the nearest address space as much as possible, so that the reducer reads sequentially and improves the shuffle performance.

[0278] After the map node (in the map task) completes the partitioning and copying operation as described above, the intermediate data is obtained. The storage address of the intermediate data is sent to the management node. The storage address here indicates the address identifier of the data and to which partition the data belongs. It can be understood that the storage address sent by the map node to the management node includes the partition id and the address identifier. The address identifier can include the first address and the offset of each data.

[0279] Step 1007-2, the map node (in the map task) stores the storage address of the data belonging to the specific partition / skewed partition in the shard into the corresponding second region.

[0280] Specifically, the map node (the map task in the map node) directly encapsulates the shard and the storage address array indicating the data in the skewed partition into a dictionary, and stores the address and the position array of the shard in a corresponding buffer. The corresponding buffer is originally used to store the data of the skewed partition. The address and the position array of the shard are the storage addresses of the data.

[0281] The dictionary is introduced below in conjunction with FIG. 24. As shown in FIG. 24, data of the dictionary represents an array (shard) containing 10 data. ids represents a position array containing data indicating the positions of the target data. For example, 0 in the ids indicates the first data "10" in the array, 9 indicates the eighth data "17" in the array, and so on. Through this rule, the real array of the flat part can be restored.

[0282] How to store the dictionary is introduced below in conjunction with FIG. 25. As shown in FIG. 25, the shard contains 7 data. After metadata calculation, it is known that the number of data belonging to partition 1 is 5, the number of data belonging to partition 2 is 2, and the number of data belonging to partition 3 is 0. The number of data belonging to partition 1 exceeds the threshold (assuming that the threshold is 50%), and therefore, the data amount corresponding to partition 1 is large, and there is data skew, and partition 1 is a skewed partition / special partition; when copying the partition data, the data belonging to partition 1 is not copied, that is, data 1, data 3, data 4, data 6, and data 7 are not copied from the first area to buffer 1. The shard and the metadata (position array) of partition 1 are encapsulated into a dictionary, and the address of the shard and the position array (for example, row id) indicating the positions of data 1, data 3, data 4, data 6, and data 7 are stored in buffer 1. It can be understood that in some embodiments, the address stored in buffer 1 is the first address of the shard. Based on the first address and the position array, the storage addresses of all data belonging to partition 1 can be known.

[0283] For the data belonging to partition 1, the corresponding reduce task obtains the storage addresses of the data from buffer 1, and pulls the corresponding data from the first area according to the indication of the storage addresses.

[0284] For the data (data 2 and data 5) belonging to partition 2, the data is copied to buffer 2 of the PMEM pool. The corresponding reduce task pulls the data from buffer 2.

[0285] In the embodiments of the present application, when performing the map task, it is determined whether the data belonging to a certain partition needs to be copied according to the proportion of the data belonging to each partition. When the proportion of the data belonging to a certain partition is greater than a threshold (for example, greater than 50%, which is called a skewed partition), the data belonging to the skewed partition is not copied (that is, the corresponding data is not copied from the first area to the second area), but the metadata and the shard belonging to the skewed partition are packaged into a dictionary. The dictionary usually contains two parts of content: the shard and the position array (or metadata), wherein the position array is an address array, and the target data belongs to the skewed partition. The shard is stored in the first area of the PMEM pool, and the position array is stored in the second area of the PMEM pool, which corresponds to the skewed partition. The reduce task can pull the corresponding data in the shard in the first area to the local according to the indication of the position array. In this way, it can be avoided to copy a large amount of data from the first area to the second area, reduce the delay caused by copying, and improve the shuffle efficiency.

[0286] Step 1008, the reduce node (in the reduce task) pulls the data belonging to the corresponding partition to the local according to the indication of the management node.

[0287] As before, the path of the reduce node (in the reduce task) to pull the data is also different according to whether the shard performs the partition copy action. The following will be introduced in several cases.

[0288] Case 1, the shard is stored in the memory pool. The reduce node (in the reduce task) pulls the data belonging to the specified partition from the shard in the memory pool. For example, the shard and the metadata shown in FIG. 18 are stored in the memory pool shown in FIG. 23. The reduce node (in the reduce task) can pull the data belonging to the specified partition in the shard to the local according to the indication of the management node. The steps of the reduce node to pull the data have been described in the foregoing part, and will not be repeated here.

[0289] Case 2, the shard is stored in the PMEM pool, and there is no skewed partition. The reduce node (in the reduce task) pulls the data belonging to the corresponding partition from the corresponding second area of the PMEM pool according to the indication of the management node. For example, the shard and the data after partitioning shown in FIG. 17 are stored in the PMEM pool shown in FIG. 23. The steps of the reduce node (in the reduce task) to pull the data have been described in the foregoing part, and will not be repeated here.

[0290] Case 3, the fragments are stored in the PMEM pool, and there is a skewed partition. For the skewed partition, the reduce node (the reduce task therein) pulls the corresponding data in the first region of the PMEM pool to the local according to the indication of the position array in the corresponding second region of the PMEM pool. For the remaining partitions,

[0291] It should be noted that Figure 23 does not show the step of storing metadata. The metadata is stored in the memory pool in priority. When the data partition in the PMEM pool is needed, the corresponding metadata is copied into the PMEM pool. The map node (the map task therein) copies the data in the fragment to the corresponding buffer (the second region) according to the indication of the metadata. The reduce node (the reduce task therein) pulls the data belonging to the corresponding partition from the second region of the PMEM pool according to the indication of the management node.

[0292] It should be noted that Figure 23 also does not show the step of making the AQE decision by the management node according to the data amount fed back by the map task. The step of making the AQE decision by the management node occurs before the reduce node pulls the data.

[0293] In some embodiments, the number of fragments that need to be processed by the map node is multiple. After the map node processes N fragments, the map node reports the latest storage address of the data to the management node, and the reduce task can start reading the data without waiting for the shuffle write of all fragments to be completed. For example, assuming that the number of fragments is 100 and the value of N is 10, after the map node processes 10 fragments, the map node reports the latest storage address of the data to the management node.

[0294] In one scheme, for each group of fragments, the map task stores the fragments in the memory pool, and then copies the data from the memory pool to the PMEM pool according to the indication of the metadata to form the intermediate data in the PMEM pool. Compared with this scheme, the scheme of the embodiment of the present application considers the actual situation of the fragments and adopts different copying strategies. Not all fragments are first stored in the memory, and not all fragments are copied to form the intermediate data. Relatively, the number of data copying is minimized, and the shuffle performance is improved.

[0295] The above scheme is described by taking the storage of the fragments in the volatile cache pool or the non-volatile cache pool as an example. It can be understood that, if the memory space of the map node is sufficient, after the map node calculates and determines the partition to which each data in the fragment belongs, the fragment can also not be copied from the memory of the map node, and the reduce task reads the data from the memory of the computing node.

[0296] In some embodiments, if the data amount of a shard is small, the map task sends the storage address of the data in the shard in the map node to the reduce task through the management node as an access address. After obtaining the access address, the reduce task accesses the map node based on the access address to obtain the corresponding data.

[0297] In some embodiments, if the data amount of a shard is small, the map task stores the position of the data belonging to the same partition in the shard. The position of the data in the shard is, for example, the row number / offset of the data in the shard. The storage address of the data in the map node is, for example, the storage address of the row number / offset in the map node and the first address of the shard, or the storage address of the row number / offset in the map node.

[0298] In some embodiments, if the data amount of a shard is small, the map task stores the position of the data belonging to the same partition in the shard. The position of the data in the shard is, for example, the row number / offset of the data in the shard. The storage address of the data in the map node is, for example, the storage address of the row number / offset in the map node and the first address of the shard, or the storage address of the row number / offset in the map node.

[0299] In the above scheme, for the data belonging to a certain partition, when partition copy is needed, all the data belonging to the certain partition is stored; when partition copy is not needed, the position of the data belonging to the certain partition in the shard is stored. When no partition copy is performed, the time spent on partition copy can be saved, and the shuffle efficiency is improved.

[0300] In the above, the partition management method provided by the embodiments of the present application is described in detail in combination with FIG. 1 to FIG. 25. In the following, the partition management device and the data processing system provided by the embodiments of the present application for implementing the above-mentioned partition management method will be described in combination with FIG. 26 to FIG. 30. The partition management device and the data processing system are used to execute the above-mentioned partition management method.

[0301] FIG. 26 and FIG. 27 are structural schematic diagrams of possible partition management devices provided by the embodiments. These partition management devices can be used to implement the functions of the map node or the management node in the above-mentioned method embodiments, and thus can also achieve the beneficial effects of the above-mentioned method embodiments.

[0302] Fig. 26 is a structural schematic diagram of a possible partition management apparatus provided in the embodiment. As shown in Fig. 26, the partition management apparatus 1300 includes a determining module 1301 and a counting module 1302. The partition management apparatus 1300 is configured to implement the function of the map task in the method embodiments shown in Figs. 7 and 13.

[0303] The determining module 1301 is configured to determine a partition to which each data entry in a plurality of data entries contained in a split belongs; wherein the number of the partitions is a plurality.

[0304] Specifically, the determining module is configured to receive a split, and perform calculation on data entries (data) in the split to determine a partition to which each data entry in the split belongs; the partition has a corresponding relationship with a reduce task that processes the data entry belonging to the partition.

[0305] Specifically, the determining module 1301 performs calculation on the split to obtain metadata, and the metadata includes partition information, which is used to indicate the corresponding relationship between the data entries and the partitions. According to the indication of the partition information, the data entries can be divided (or copied) to a specified location to obtain intermediate data. The intermediate data includes a plurality of areas, and the data entries in each area belong to the same partition. Therefore, the partition information included in the metadata embodies a partition plan of how to partition the split.

[0306] The counting module 1302 is configured to count a first type of data entry belonging to a first partition in the split to obtain a first data amount of the first type of data entry, and the first partition is one of the plurality of partitions, and the first data amount is used to indicate the number or size of the first type of data entry. The counting module 1302 is further configured to send the first data amount to a management node.

[0307] Specifically, the counting module 1302 is configured to count the data entries owned by each partition to obtain a data entry counting result. The data entry counting result can embody the task amount of the reduce task. The counting module 1302 is further configured to send the data entry counting result to the management node, so that the management node determines whether to adjust the number of the reduce tasks according to the received data entry counting result.

[0308] Specifically, the counting module 1302 is configured to determine the data amount of each partition in the partition plan according to the metadata. Since the metadata can embody the partition plan, the data amount of each partition can be obtained through the metadata. After obtaining the data amount, the counting module 1302 sends the data amount to the management node.

[0309] Optionally, the data entry statistics result or the first data amount can also be sent to the management node by the sending module 1306.

[0310] Specifically, the first data amount is related to a task amount of the first reduce task; the first reduce task is used for processing data entries belonging to the first partition.

[0311] Specifically, the management node is used for counting data amounts of data entries of each partition, and judging whether to adjust the number of reduce tasks according to the counting result; the reduce task is used for processing data entries belonging to a corresponding partition; the first reduce task is one of the plurality of reduce tasks; after determining to adjust the number of reduce tasks, the correspondence between the partition and the reduce task is updated; the management node is also used for providing an access address of the data entry to the reduce task based on the updated correspondence between the partition and the reduce task.

[0312] Optionally, the apparatus further comprises a providing module (not shown in FIG. 26) configured to provide the first type of data entry to the first reduce task in response to a data entry reading request of the first reduce task based on the access address.

[0313] Optionally, the apparatus further comprises a transferring module 1303 configured to store the shard into a data cache pool; and the apparatus has a smaller access latency to the data cache pool than to an external storage managed by the apparatus.

[0314] Optionally, the apparatus further comprises a centralized storage module 1304 configured to centrally store the first type of data entry; and the sending module is specifically configured to send the first data amount to the management node before the centralized storage.

[0315] Optionally, the apparatus further comprises a centralized storage module 1304 configured to centrally store the location of the first type of data entry in the shard; and the sending module is specifically configured to send the first data amount to the management node before the centralized storage.

[0316] Specifically, the transferring module 1303 is configured to: judge whether the shard meets a first condition; and in a case where the shard meets the first condition, store the shard into a volatile cache pool of the data cache pool; and in a case where the shard does not meet the first condition, store the shard into a non-volatile cache pool of the data cache pool.

[0317] Optionally, the first condition comprises that a data amount of the shard is less than a threshold.

[0318] Optionally, the apparatus further includes a centralized storage module 1304 configured to, in the case that the shard is stored in the volatile cache pool, not perform the partition copy operation on the shard.

[0319] Optionally, the apparatus further includes a sending module 1306 configured to, in the case that the shard is stored in the volatile cache pool, send the storage address of the data entry contained in the shard in the volatile cache pool to the management node.

[0320] Optionally, the apparatus further includes a centralized storage module 1304 configured to, in the case that the shard is stored in the non-volatile cache pool, determine whether the shard meets a second condition; in the case that the shard meets the second condition, perform a partition copy operation (or a centralized storage operation) on the part of the data entries belonging to the shard, so that the part of the data entries are stored in the non-volatile cache pool according to the partition; or, in the case that the shard does not meet the second condition, perform the partition copy operation on all the data entries in the shard, so that all the data entries are stored in the non-volatile cache pool according to the partition.

[0321] Specifically, the centralized storage module 1304 is configured to copy the data entries to the designated positions (or designated areas) in the non-volatile cache pool according to the correspondence between the data entries and the partitions, and the designated positions correspond to the reduce tasks.

[0322] Specifically, the centralized storage module 1304 is configured to, in the case that the shard meets the second condition, store the storage address of the data entry belonging to a certain specific partition in the non-volatile cache pool in the specific partition; the specific partition is one of the plurality of partitions, and the number of the data entries owned by the specific partition is greater than the number of the data entries owned by each of the remaining partitions; and the partition is a specific area in the non-volatile cache pool.

[0323] Optionally, the second condition includes that the proportion of the data entries belonging to a certain partition in the shard exceeds a threshold.

[0324] Optionally, the partition management apparatus is located at a computing node, and the apparatus further includes a transfer module 1303 configured to store the shard in the memory or the volatile cache pool of the computing node, wherein the access delay of the partition management apparatus to the volatile cache pool is less than the access delay of the partition management apparatus to the external storage managed by the partition management apparatus. The sending module 1306 included in the apparatus is further configured to provide, to the first reduce task through the management node, the storage address of the first type of data entry in the shard in the memory or the volatile cache pool as the access address.

[0325] Optionally, the partition management apparatus further comprises a transferring module 1303, configured to store the shards in the non-volatile cache pool; wherein the access latency of the partition management apparatus to the non-volatile cache pool is less than the access latency of the partition management apparatus to the external storage managed by the partition management apparatus.

[0326] Optionally, the partition management apparatus further comprises a centralized storage module 1304, configured to store the first type of data entries in the non-volatile cache pool; the sending module 1306 comprised by the apparatus is further configured to provide, by the management node, the storage address of the first type of data entries stored in the non-volatile cache pool as the access address to the first reduce task.

[0327] Specifically, the determining module 1301 obtains the metadata by performing calculation on all the data entries, and the metadata is used to indicate the correspondence between the data entries and the partitions.

[0328] FIG. 27 is a structural schematic diagram of a possible partition management apparatus provided in the embodiment. As shown in FIG. 27, the partition management apparatus 1400 comprises an AQE decision module (or adjustment module) 1401 and a receiving module 1402. The partition management apparatus 1400 is configured to implement the function of the management node in the method embodiments shown in FIG. 7, FIG. 13 and FIG. 14.

[0329] The receiving module 1402 is configured to receive the data entry statistical results sent by all the map tasks; the data entry statistical result is obtained by performing statistics on the data entries belonging to the same partition after the map task determines the partitions to which the plurality of data entries contained in the shard belong, and the data entry statistical result is used to indicate the number or size of the data entries belonging to the same partition.

[0330] The AQE decision module 1401 is configured to determine the data amount of the data entries of each partition according to the received data entry statistical results, and determine whether to adjust the number of reduce tasks according to the statistical results.

[0331] FIG. 28 is a structural schematic diagram of a possible partition management apparatus provided in the embodiment. As shown in FIG. 28, the partition management apparatus 1500 comprises a data reading module 1501. The partition management apparatus 1500 is configured to implement the function of the reduce task in the method embodiment shown in FIG. 13, the function of the reduce node in the method embodiment shown in FIG. 14, and the function of the reduce node in the method embodiment shown in FIG. 23.

[0332] The data reading module 1501 is configured to read, according to the indication of the metadata stored in the volatile cache pool, the data entries belonging to the specified partition in the shards stored in the volatile cache pool to the local; or read, according to the indication of the dictionary stored in the non-volatile cache pool, the data entries belonging to the specified partition in the shards stored in the non-volatile cache pool to the local; or read, to the local, the data entries belonging to the specified partition in the intermediate data stored in the non-volatile cache pool; wherein the access speed of the volatile cache pool is greater than the access speed of the non-volatile cache pool.

[0333] It should be noted that the partition management apparatus can be a software in the form of a program, a plug-in, etc. running on a computing device, or a physical computing device entity. When the partition management apparatus is a software in the form of a program, a plug-in, etc. running on a computing device, the partition management apparatus runs on a physical computing device entity.

[0334] FIG. 29 is a structural schematic diagram of a possible data processing system provided by an embodiment of the present application. The data processing system can be used to implement the functions of the computing nodes (map nodes or reduce nodes) or the management node in the method embodiments, and thus can also achieve the beneficial effects of the method embodiments.

[0335] As shown in FIG. 29, the data processing system includes a plurality of computing nodes 200 and a management node 100. The computing nodes 200 are configured to implement the functions of the map nodes or the reduce nodes in the method embodiments, and the management node 100 is configured to implement the functions of the management node in the method embodiments. Therefore, the data processing system in the embodiment of the present application can achieve the beneficial effects of the method embodiments.

[0336] Specifically, the computing node 200 includes a shuffle manager 230 and a local memory 240, and the management node 100 includes a shuffle manager 110, an AQE manager 120 and a local memory 130.

[0337] The shuffle manager 110 of the management node 100 is configured to send a map task to the shuffle manager 230 of the computing node 200, and receive the data volume (or the data entry statistical result) sent by the shuffle manager 230 of the computing node 200, and send the received data volume (or the data entry statistical result) to the AQE manager 120.

[0338] Specifically, the shuffle manager 110 of the management node 100 is configured to receive the AQE decision sent by the AQE manager 120, and generate an AQE task (used to indicate the correspondence between the reduce task and the data entry) according to the AQE decision; and send the AQE task to the shuffle manager 230 of the computing node 200.

[0339] Optionally, the shuffle manager 110 of the management node 100 is configured to receive the read data request sent by the shuffle manager 230 of the computing node 200; and send the address of the partition to the shuffle manager 230 of the computing node 200.

[0340] Optionally, the AQE manager 120 is configured to make an AQE decision according to the data amount (or the data entry statistical result) sent by the shuffle manager 110 of the management node 100, and send the AQE decision to the shuffle manager 110 of the management node 100. It can be understood that the AQE manager 120 is configured to perform the steps 803 and 804.

[0341] It can be understood that when the computing node 200 is configured to execute the map task, the shuffle manager 230 of the computing node 200 is configured to perform the steps 601, 602, 603, 805, 806, the steps performed by the map node in FIG. 14, and the steps performed by the map node in FIG. 23.

[0342] When the computing node 200 is configured to execute the reduce task, the shuffle manager 230 of the computing node 200 is configured to perform the step 806, the steps performed by the reduce node in FIG. 14, and the steps performed by the reduce node in FIG. 23.

[0343] Optionally, the management node 100 further comprises a cache pool manager 140; the data processing system further comprises a data cache pool 400, and the data cache pool 400 is configured to implement the function of the data cache pool in the method implementation. The cache pool manager 140 of the management node 100 is configured to configure the data cache pool. It can be understood that the cache pool manager 140 is configured to perform the function of the cache pool manager in FIG. 20.

[0344] Optionally, the data cache pool 400 includes a memory pool 410 and a PMEM pool 420. The memory pool 410 includes a plurality of memory terminals, and the PMEM pool 420 includes a plurality of PMEM terminals. The memory pool 410 and the memory terminals contained therein have similar functions to those of the memory pool and the memory terminals in FIG. 19. The PMEM pool 420 and the PMEM terminals contained therein have similar functions to those of the PMEM pool and the PMEM terminals in FIG. 19. Details are not described herein.

[0345] In some embodiments, to further improve the shuffle performance, as shown in FIG. 30, the computing node of the embodiment of the application further includes a metadata computing accelerator 250. The metadata computing accelerator 250 is a hardware accelerator for metadata computing. It can be understood that the metadata computing accelerator 250 is used to perform the aforementioned steps 902 and / or 903.

[0346] One of the shuffle bottlenecks is that a large number of partition copy operations (involving memcpy, memmove, etc. memory operations) are required, and the CPU of the computing node needs to wait for the I / O to be completed before proceeding to the next step of calculation, resulting in waste of CPU resources. By introducing the metadata computing accelerator 250, the use of the CPU of the computing node can be reduced, and the efficiency of the CPU can be improved. The memcpy / memmove is a library function, and by executing the memcpy / memmove, a specified number of bytes can be copied from one storage area to another storage area.

[0347] In some embodiments, the shuffle manager 230 of the computing node 200 pushes the task of computing metadata to the metadata computing accelerator 250. In some embodiments, the metadata computing accelerator can be a hardware accelerator specially used for computing hash values, which solidifies the hash computing logic in the chip and can accelerate the computation.

[0348] In some embodiments, to further improve the shuffle performance, as shown in FIG. 30, the data processing system of the embodiment of the application further includes a data copy accelerator 260. The data copy accelerator 260 is a hardware accelerator used to perform the aforementioned steps 805, 907, 1005-1, 1005-2, 1007-1, 1007-2, etc.

[0349] The shuffle manager 230 of the computing node 200 will issue the partition to be copied and the corresponding partition id and row id to the data copy accelerator 260, and the data copy accelerator 260 will copy the corresponding row from the partition to the corresponding second area.

[0350] After the metadata computing accelerator 250 and the data copying accelerator 260 are added, not only the computing efficiency can be improved, but also the performance can be further improved through parallel execution of the two added metadata computing accelerator 250 and the data copying accelerator 260.

[0351] To further improve the performance, in the embodiment of the present application, the shuffle manager 230 of the computing node 200 asynchronously pushes down the request (the request of computing metadata, or the request of copying the partition). The so-called asynchronous means that the CPU does not need to wait for the metadata computing accelerator 250 / data copying accelerator 260 to return the result, and the CPU can directly return after the request is issued, and do other work; the CPU will be signaled by the metadata computing accelerator 250 / data copying accelerator 260 when the work is completed, and then the CPU processes the subsequent work.

[0352] The embodiment of the present application provides a computer storage medium, which stores a computer program, and the computer program is executed by a processor to implement the partition management method in the above method embodiment.

[0353] The embodiment of the present application provides an electronic device / computing device, which includes one or more processors, and the processor is configured to support the electronic device / computing device to implement the corresponding function in any of the above partition management methods. The electronic device / computing device can further include a memory, which is coupled with the processor and saves necessary program instructions and data of the electronic device / computing device. The electronic device / computing device can further include a communication interface, which is used for communication between the electronic device and other devices or communication networks.

[0354] The embodiment of the present application provides a chip system, which includes a processor, and the processor is used to support the chip system to implement the function of any of the above partition management methods. In a possible design, the chip system further includes a memory, and the memory is used to save necessary program instructions and data of the chip system. The chip system can be composed of a chip, or can include the chip and other discrete devices.

[0355] The embodiment of the present application provides a computer readable storage medium, which stores computer program code, and when the computer program code is executed by a computing device, the computer implements the method shown in the above method embodiment.

[0356] The embodiment of the present application provides a computer program product, which includes a computer program stored in a computer readable storage medium, and the computer program is loaded by a processor to implement the method shown in the above method embodiment.

[0357] In the above embodiments, the description of each embodiment focuses on different aspects, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0358] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all described as a combination of a series of actions, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, some steps can be performed in other order or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application. Those skilled in the art can select some steps in the above method embodiments to solve the corresponding technical problems.

[0359] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented by other ways. For example, the device embodiments described above are only schematic, and for example, the division of the above units is only a logical function division, and there can be another division way in actual implementation, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different parts can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical or other forms.

[0360] The units / modules described as separate components above can or can not be physically separate, and the components shown as units / modules can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units / modules can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0361] In addition, each functional unit / module in each embodiment of the present application can be integrated in one processing unit, or each unit / module can be physically present separately, or two or more units / modules can be integrated in one unit / module. The integrated unit / module can be realized in the form of hardware or in the form of a software functional unit.

[0362] The integrated unit module described above, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application or all or part of the technical solutions that essentially contribute to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc., and specifically can be a processor in the computer device) to execute all or part of the steps of the above-mentioned method of each embodiment of the present application. Among them, the aforementioned storage medium can include: a U disk, a mobile hard disk, a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), and various storage program codes.

[0363] The above embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of partition management, characterized by, The method comprises: determining a partition to which each data entry in a plurality of data entries contained in a split belongs; wherein the number of the partitions is multiple; counting a first type of data entry belonging to a first partition in the split to obtain a first data amount of the first type of data entry, the first partition being one of the multiple partitions, the first data amount indicating the number or size of the first type of data entry; sending the first data amount to a management node.

2. The partition management method of claim 1, wherein: The first data amount is related to the size of a task amount of a first reduce task, the first reduce task being used to process the data entry belonging to the first partition.

3. The partition management method according to claim 2, wherein: the management node is used to count the data amount of the data entry of each partition and determine whether to adjust the number of reduce tasks according to the counting result, the reduce task being used to process the data entry belonging to the corresponding partition, the first reduce task being one of the multiple reduce tasks; after determining to adjust the number of reduce tasks, updating the correspondence between the partition and the reduce task; the management node is further used to provide an access address of the data entry to the reduce task based on the updated correspondence between the partition and the reduce task.

4. The partition management method according to claim 2 or 3, characterized by, The method further comprises: in response to a data entry reading request of the first reduce task based on the access address, providing the first type of data entry to the first reduce task.

5. The partition management method of claim 1, wherein, The method further comprises: centrally storing the first type of data entry; wherein the sending of the first data amount to the management node specifically comprises: sending the first data amount to the management node before the central storage; or, centrally storing the position of the first type of data entry in the split; wherein the sending of the first data amount to the management node specifically comprises: sending the first data amount to the management node before the central storage.

6. The partition management method according to claim 2, wherein, The method further comprises: the computing node stores the split in the memory or the volatile cache pool of the computing node, wherein the access delay of the computing node to the volatile cache pool is less than the access delay of the computing node to the external storage; the computing node provides the storage address of the first type of data entry in the memory or the volatile cache pool to the first reduce task through the management node.

7. The partition management method of claim 2, wherein, The method further comprises: the computing node stores the split in the non-volatile cache pool; the access delay of the computing node to the non-volatile cache pool is less than the access delay of the computing node to the external storage.

8. The partition management method according to claim 7, wherein, The method further comprises: centrally storing the first type of data entry in the non-volatile cache pool; providing the storage address of the centrally stored first type of data entry in the non-volatile cache pool to the first reduce task through the management node.

9. The partition management method according to claim 7, wherein, The method further comprises: store the first type of data entries in the shard in a set of storage addresses in the non-volatile cache pool; provide, by the management node, the set of storage addresses in the non-volatile cache pool as access addresses to the first reduce task; the first data amount is greater than a second data amount, and the second data amount is a data amount of a second type of data entries, the second type of data entries belonging to a second partition, the second partition being one of the plurality of partitions.

10. A method of partition management, the method comprising: The method comprises: receiving data entry statistics sent by all computing nodes; the data entry statistics being statistics of data entries belonging to the same partition after each data entry in a plurality of data entries contained in the shard is determined to belong to a partition; the data entry statistics being used to indicate the number or size of data entries belonging to the same partition; statistically determining the data amount of data entries of each partition according to the received data entry statistics, and determining whether to adjust the number of reduce tasks according to the statistical result.

11. A partition management apparatus characterized by comprising: comprise: a determination module configured to determine a partition to which each data entry in a plurality of data entries contained in a shard belongs; wherein the number of the partitions is a plurality; a statistical module configured to statistically determine a first type of data entries belonging to a first partition in the shard, to obtain a first data amount of the first type of data entries, the first partition being one of the plurality of partitions, and the first data amount being used to indicate the number or size of the first type of data entries; a sending module configured to send the first data amount to a management node.

12. The partition management apparatus according to claim 11, wherein: The first data amount is related to the task amount of the first reduce task; and the first reduce task is used to process data entries belonging to the first partition.

13. The partition management apparatus according to claim 12, wherein Further comprise: a providing module configured to provide the first type of data entries to the first reduce task in response to a data entry reading request of the first reduce task based on the access addresses.

14. The partition management apparatus according to claim 11, wherein Further comprise: a centralized storage module configured to centrally store the first type of data entries; wherein the sending module is specifically configured to: before the centralized storage, send the first data amount to the management node; or a centralized storage module configured to centrally store the positions of the first type of data entries in the shard; wherein the sending module is specifically configured to: before the centralized storage, send the first data amount to the management node.

15. The partition management apparatus of claim 12, wherein, The apparatus further comprises: a transfer module configured to store the shard in a memory or a volatile cache pool managed by the partition management apparatus, wherein the access delay of the partition management apparatus to the volatile cache pool is less than the access delay of the partition management apparatus to the external storage managed by the partition management apparatus; the sending module is further configured to provide, by the management node, storage addresses of the first type of data entries in the shard in the memory or the volatile cache pool as access addresses to the first reduce task.

16. The partition management apparatus of claim 12, wherein, The apparatus further comprises: A migration module is configured to store the shards in a non-volatile cache pool; wherein the access latency of the partition management device to the non-volatile cache pool is less than the access latency of the partition management device to the external storage managed by the partition management device.

17. The partition management apparatus according to claim 16, wherein The apparatus further comprises: A centralized storage module is configured to store the first type of data entries in the non-volatile cache pool in a centralized manner; The sending module is further configured to provide the storage address of the first type of data entries stored in the non-volatile cache pool in a centralized manner as an access address to the first reduce task through the management node.

18. The partition management apparatus of claim 16, wherein, The apparatus further comprises: A centralized storage module is configured to store the storage addresses of the first type of data entries in the shards in the non-volatile cache pool in a centralized manner; The sending module is further configured to provide the address of the storage addresses stored in the non-volatile cache pool in a centralized manner as an access address to the first reduce task through the management node. The first data amount is greater than a second data amount, and the second data amount is the data amount of the second type of data entries, which belong to a second partition, and the second partition is one of the plurality of partitions.

19. A partition management apparatus characterized by comprising: The apparatus comprises: A receiving module is configured to receive the data entry statistical results sent by all the computing nodes; the data entry statistical results are obtained by counting the data entries belonging to the same partition after the computing nodes determine the partitions to which the plurality of data entries included in the shards belong; the data entry statistical results are used to indicate the number or size of the data entries belonging to the same partition; An adjusting module is configured to count the data amounts of the data entries of each partition according to the received data entry statistical results, and determine whether to adjust the number of reduce tasks according to the counting results.

20. A computing device, comprising: The apparatus comprises: A memory is configured to store computer instructions; One or more processors are configured to execute the computer instructions stored in the memory, so that the computing device executes the method in any one of the preceding claims 1 to 10. The apparatus comprises: A memory is configured to store computer instructions; One or more processors are configured to execute the computer instructions stored in the memory, so that the computing device executes the method in any one of the preceding claims 1 to 10.

Images