Key-value data processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a key-value data processing method and device, electronic equipment and storage medium. The method comprises: obtaining a to-be-processed task and meta-information corresponding to the to-be-processed task; processing the to-be-processed task through a mapping task in a computing engine to obtain key-value data, and transmitting the key-value data to a shuffle processing node, the shuffle processing node being a node independently encapsulated outside the computing engine; and running sorting logic corresponding to the meta-information through the shuffle processing node to sort the key-value data. According to the scheme of the present disclosure, a general sorting mechanism is used to process key-value data generated by multiple computing engines, thereby meeting the sorting requirements of key-value data of multiple computing engines at the same time.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a key-value data processing method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] Current computing engines such as Mapreduce (a kind of mapping reduction model), Spark (a kind of computing engine) have shuffle mechanism. The shuffle mechanism undertakes data transmission between MapTask (mapping task) and ReduceTask (reduction task) in the task.

[0003] At present, the shuffle mechanism implementation mainly includes sort-based shuffle (shuffle based on sorting) and hash-based shuffle (shuffle based on hash). Mapreduce mainly adopts the sort-based shuffle mode, which sorts the key-value data according to partition (partition) and key (key) at the mapping task end, and outputs to the local disk to form an ordered shuffle file. In this way, the shuffle file obtained by the reduction task end is locally ordered. Spark mainly adopts the hash-based shuffle mode, which sorts the keys at the mapping task end, and outputs to the local disk according to the partition classification to form a disordered shuffle file. In this way, the reduction task end needs to perform global sorting on the obtained shuffle file set.

[0004] However, the current internal stream shuffle mechanism of enterprises mostly only supports Spark engine. In the case of coexistence of Spark and MapReduce jobs in the enterprise, the hash-based shuffle mode of the Spark engine cannot meet the key-value data sorting requirements of the MapReduce job, and therefore, there is an urgent need for a flexible sorting mode for key-value data of multiple computing engines in the case of simultaneous operation of multiple computing engines. SUMMARY

[0005] The present disclosure provides a key-value data processing method, device, electronic equipment, computer readable storage medium and computer program product, to provide a flexible sorting mode for key-value data of multiple computing engines in the case of simultaneous operation of multiple computing engines. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of the embodiments of the present disclosure, a key-value data processing method is provided, and the method comprises:

[0007] obtaining a to-be-processed task and meta information corresponding to the to-be-processed task;

[0008] Processing the task to be processed by a mapping task in a computing engine to obtain key-value data, and transmitting the key-value data to a shuffle processing node, which is a node independently encapsulated outside the computing engine;

[0009] The sorting logic corresponding to the meta information is run through the shuffle processing node to sort the key-value data.

[0010] In one embodiment, a first class loader is deployed in the shuffle processing node, and the meta information includes a storage path of the sorting logic file package;

[0011] The running of the sorting logic corresponding to the meta-information by the shuffle processing node includes:

[0012] Acquire the sorting logic file package stored in the storage path through the first class loader in the shuffle processing node;

[0013] The sorting logic file package is loaded through the first class loader to run the sorting logic.

[0014] In one embodiment, a second class loader is further deployed in the shuffle processing node;

[0015] Before acquiring the sorting logic file package stored in the storage path through the first class loader in the shuffle processing node, the method further includes:

[0016] In a case where the to-be-processed task is submitted through a preset tool, it is determined through the first class loader that no sorting logic corresponding to the preset tool exists in the second class loader.

[0017] In one embodiment, the method further comprises:

[0018] When it is determined through the first class loader that the second class loader contains the sorting logic corresponding to the preset tool, the first class loader executes the sorting logic corresponding to the preset tool.

[0019] In one embodiment, a third class loader is further deployed in the shuffle processing node; the meta information further includes key class information of the task to be processed;

[0020] Before acquiring the sorting logic file package stored in the storage path through the first class loader in the shuffle processing node, the method further includes:

[0021] It is determined through the first class loader that no sorting logic corresponding to the key class information exists in the third class loader.

[0022] In one embodiment, the method further comprises:

[0023] When it is determined through the first class loader that the sorting logic corresponding to the key class information already exists in the third class loader, the sorting logic corresponding to the key class information is executed.

[0024] In one embodiment, the method further comprises:

[0025] Registering the meta information to a global management component, where the global management component is a component independently packaged outside the computing engine;

[0026] The meta information is sent to the shuffle processing node through the global management component.

[0027] In one embodiment, there are multiple shuffle processing nodes; and transmitting the key-value data to the shuffle processing node includes:

[0028] Acquire a mapping relationship between a partition and a shuffle processing node through a shuffle write node, wherein the mapping relationship is pre-built by the global management component according to the partition of the task to be processed, and the shuffle write node is a node independently encapsulated outside the computing engine;

[0029] Determine the key value data corresponding to each partition through the shuffle write node, and send the key value data corresponding to each partition to the shuffle processing node corresponding to each partition according to the mapping relationship;

[0030] The step of running a sorting logic corresponding to the meta-information through the shuffle processing node to sort the key-value data includes:

[0031] The sorting logic is run through the shuffle processing node corresponding to each of the partitions to sort the key-value data corresponding to each of the partitions.

[0032] According to a second aspect of an embodiment of the present disclosure, a device for processing key-value data is provided, the device comprising:

[0033] An acquisition module is configured to acquire tasks to be processed and meta information corresponding to the tasks to be processed;

[0034] a mapping module configured to process the to-be-processed task through a mapping task in a computing engine to obtain key-value data, and transmit the key-value data to a shuffle processing node, wherein the shuffle processing node is a node independently encapsulated outside the computing engine;

[0035] The data sorting module is configured to execute sorting logic corresponding to the meta information through the shuffle processing node to sort the key-value data.

[0036] In one embodiment, a first class loader is deployed in the shuffle processing node, and the meta information includes a storage path of the sorting logic file package;

[0037] The data sorting module includes:

[0038] A file package acquiring unit is configured to acquire the sorting logic file package stored in the storage path through the first class loader in the shuffle processing node;

[0039] The first loading unit is configured to load the sorting logic file package through the first class loader to run the sorting logic.

[0040] In one embodiment, a second class loader is further deployed in the shuffle processing node;

[0041] The data sorting module further includes:

[0042] The submission tool determination unit is configured to, when the to-be-processed task is submitted through a preset tool, determine, through the first class loader, that no sorting logic corresponding to the preset tool exists in the second class loader.

[0043] In one embodiment, the data sorting module further includes:

[0044] The second loading unit is configured to execute, when it is determined by the first class loader that the sorting logic corresponding to the preset tool exists in the second class loader, running the sorting logic corresponding to the preset tool through the first class loader.

[0045] In one embodiment, a third class loader is further deployed in the shuffle processing node; the meta information further includes key class information of the task to be processed;

[0046] The data sorting module further includes:

[0047] The query unit is configured to execute, through the first class loader, a determination that a sorting logic corresponding to the key class information does not exist in the third class loader.

[0048] In one embodiment, the data sorting module further includes:

[0049] The third loading unit is configured to execute the sorting logic corresponding to the key class information when it is determined by the first class loader that the sorting logic corresponding to the key class information already exists in the third class loader.

[0050] In one embodiment, the apparatus further comprises:

[0051] a registration module configured to register the meta information with a global management component, wherein the global management component is a component independently packaged outside the computing engine;

[0052] The information sending module is configured to send the meta information to the shuffle processing node through the global management component.

[0053] In one embodiment, the number of the shuffle processing nodes is multiple;

[0054] The mapping module includes:

[0055] a mapping relationship acquisition unit configured to acquire a mapping relationship between a partition and a shuffle processing node through a shuffle write node, wherein the mapping relationship is pre-constructed by the global management component according to the partition of the task to be processed, and the shuffle write node is a node independently encapsulated outside the computing engine;

[0056] a data sending unit configured to determine the key-value data corresponding to each of the partitions through the shuffle write node, and send the key-value data corresponding to each of the partitions to the shuffle processing node corresponding to each of the partitions according to the mapping relationship;

[0057] The data sorting module is configured to execute the sorting logic through the shuffle processing nodes corresponding to each of the partitions to sort the key-value data corresponding to each of the partitions.

[0058] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0059] processor;

[0060] a memory for storing instructions executable by the processor;

[0061] The processor is configured to execute the instructions to implement the key-value data processing method described in any one of the above embodiments.

[0062] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the key-value data processing method described in any one of the above embodiments.

[0063] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, which comprises instructions executed by a processor of an electronic device, so that the electronic device can perform the key-value data processing method of any of the above-mentioned embodiments.

[0064] The technical solutions provided by the embodiments of the present disclosure at least have the following beneficial effects:

[0065] By deploying the independently encapsulated shuffle processing node outside the computing engine, after obtaining the key-value data by processing the to-be-processed task through the mapping task, the shuffle processing node can obtain the sorting logic corresponding to the meta information of the to-be-processed task, and sort the key-value data by using the sorting logic, so that in the case of simultaneously existing multiple computing engines, a general sorting mechanism can be used to process the key-value data generated by the multiple computing engines, thereby meeting the sorting requirements of the key-value data of the multiple computing engines. In addition, by providing an independently deployed shuffle processing node, the embodiments of the present disclosure maximally reduce the degree of modification of the original computing engine code in a decoupled manner, have high universality, and are convenient to deploy and maintain.

[0066] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0067] The accompanying drawings incorporated in the specification and forming a part of the specification illustrate the embodiments consistent with the present disclosure, and together with the specification, serve to explain the principles of the present disclosure, and do not limit the present disclosure.

[0068] Figure 1 is an application environment diagram of a key-value data processing method according to an exemplary embodiment.

[0069] Figure 2 is a flowchart of a key-value data processing method according to an exemplary embodiment.

[0070] Figure 3 is a flowchart of a key-value data processing method according to an exemplary embodiment.

[0071] Figure 4 is a flowchart of a key-value data processing method according to an exemplary embodiment.

[0072] Figure 5 is a flowchart of a key-value data processing method according to an exemplary embodiment.

[0073] Figure 6 The figure is a schematic diagram showing a shuffle processing node according to an exemplary embodiment.

[0074] Figure 7 The figure is a flowchart of a method for processing key-value data according to an exemplary embodiment.

[0075] Figure 8 The figure is a schematic diagram showing a shuffling service according to an exemplary embodiment.

[0076] Figure 9 The figure is a block diagram of a device for processing key-value data according to an exemplary embodiment.

[0077] Figure 10 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0078] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0079] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0080] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0081] The key-value data processing method provided by the present disclosure can be applied to Figure 1In the application environment shown. The terminal 110 interacts with the server 120 through the network. The server 120 is deployed with at least one computing engine and at least one independently packaged shuffle processing node outside the computing engine. The server 120 obtains the tasks to be processed uploaded by the terminal 110, as well as the metadata corresponding to the tasks to be processed. The tasks to be processed are processed by the mapping tasks in the computing engine to obtain key-value data, and the partition to which each key-value data belongs is determined. The key-value data belonging to the same partition are aggregated, and the aggregated key-value data are sent to the shuffle processing node. The shuffle processing node obtains the sorting logic corresponding to the metadata of the tasks to be processed, and sorts the key-value data under each partition according to the sorting logic, so that the reduce task can pull the sorted key-value data under its own corresponding partition.

[0082] The terminal 110 may be, but is not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. The portable wearable devices may be smart watches, smart bracelets, etc. The server 120 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0083] Figure 2 is a flowchart of a method for processing key-value data according to an exemplary embodiment. Figure 2 As shown, the following steps are included.

[0084] In step S210 , a task to be processed and meta information corresponding to the task to be processed are obtained.

[0085] A pending task is a collection of tasks that a user requests the computing engine to perform during a problem solving process or transaction processing. A pending task can be uploaded to the computing engine by a user through a terminal device or scheduled to the computing engine by a task scheduler.

[0086] Meta-information is information about information, which is used to describe the structure, semantics, purpose and usage of the information. In the embodiment of the present disclosure, the meta-information can be information related to the processing of key-value data, such as the key class information (KeyClass) of the task, the storage path of the processing logic file package of the key-value data, the type of computing engine to which the task belongs, etc., which are not described here one by one. Among them, the key class information can be used to uniquely represent the class. In object-oriented programming, a class is a structure of an object-oriented computer programming language that describes the behavior rules of some objects, and these objects are called instances of the class.

[0087] The meta-information and the task have a mapping relationship, so that after the server obtains the task to be processed, it can obtain the meta-information corresponding to the task to be processed according to the mapping relationship.

[0088] In step S220 , the task to be processed is processed by the mapping task in the computing engine to obtain key-value data, and the key-value data is transmitted to the shuffle processing node.

[0089] Among them, the shuffle processing node is an independently packaged node deployed outside the computing engine. It can have its own properties and methods and can independently sort key-value data and perform other processing.

[0090] Specifically, after starting a pending task, the compute engine initiates a map task to process it, generating a series of key-value data. For each key-value data output by each map task, a hash value is calculated, then the hash value is modulo the number of reduce tasks to determine the partition of each key-value data. The map task aggregates the key-value data in the same partition and sends the aggregated key-value data to the shuffle processing node.

[0091] In step S230 , the sorting logic corresponding to the meta information is executed by the shuffle processing node to sort the key-value data.

[0092] Specifically, after receiving the key-value data corresponding to each partition, the shuffle processing node retrieves the sorting logic corresponding to the metadata of the task to be processed from the pre-deployed mapping relationship between sorting logic and metadata, loads the sorting logic, and sorts the key-value data corresponding to each partition. The sorted key-value data corresponding to each partition is stored in a file system (e.g., HDFS, Hadoop Distributed File System).

[0093] In one example, the metadata includes the type of computing engine corresponding to the task to be processed. The pre-deployed sorting logic corresponding to Spark is hash-based shuffling, and the pre-deployed sorting logic corresponding to MapReduce is sort-based shuffling. If the computing engine type of the task to be processed is Spark, then after receiving the key value data aggregated by partition, the shuffling processing node may not sort the key value data of each partition, but directly execute the hash-based shuffling method. If the computing engine type of the task to be processed is MapReduce, then after receiving the key value data aggregated by partition, the shuffling processing node may sort the key value data of each partition according to the sorting method corresponding to MapReduce.

[0094] In another example, the meta-information includes key information corresponding to the task to be processed. A mapping relationship between the key information and the sorting logic is pre-deployed. After receiving the partitioned key value data, the shuffle processing node can sort the key value data of each partition according to the sorting logic corresponding to the key information of the task to be processed based on the mapping relationship between the key information and the sorting logic.

[0095] In one embodiment, after the shuffle processing node completes processing of the pending tasks, the computing engine may start the reduction tasks, so that each reduction task pulls the key-value data under its corresponding partition from the file system.

[0096] In the above-mentioned method for processing key-value data, an independently encapsulated shuffle processing node is deployed outside the computing engine. After the key-value data is obtained by processing the task to be processed through the mapping task, the shuffle processing node can obtain the sorting logic corresponding to the metadata of the task to be processed, and the key-value data is sorted using the sorting logic. In the case of the simultaneous existence of multiple computing engines, a universal sorting mechanism can be used to process the key-value data generated by multiple computing engines, thereby meeting the sorting requirements of the key-value data of multiple computing engines at the same time. In addition, the embodiment of the present disclosure, by providing an independently deployed shuffle processing node, minimizes the degree of modification to the native computing engine code through decoupling, has high versatility, and is easy to deploy and maintain.

[0097] In an exemplary embodiment, the method further includes: registering the meta information with a global management component; and sending the meta information to the shuffle processing node through the global management component.

[0098] The global management component is an independently packaged component deployed outside the computing engine, which can be used for, but not limited to, global resource scheduling, global task management, lifecycle management of shuffle processing nodes, and processing heartbeat requests of shuffle processing nodes.

[0099] Specifically, after initiating a pending task and its corresponding metadata, the computing engine can register the metadata of the pending task with the global management component. In one embodiment, the global management component can distribute the metadata of the pending task to the shuffle processing node. In another embodiment, the shuffle processing node can proactively obtain the metadata of the pending task from the registration information of the global management component after receiving the aggregated key-value data corresponding to each partition.

[0100] In this embodiment, by deploying an independently packaged global management component outside the computing engine, resources can be centrally scheduled and information can be centrally managed, facilitating global control and management.

[0101] In an exemplary embodiment, the number of shuffle processing nodes is multiple. Figure 3 As shown, in step S220, the key value data is transmitted to the shuffle processing node, including:

[0102] In step S310, a mapping relationship between partitions and shuffle processing nodes is obtained through the shuffle write node. The mapping relationship is pre-constructed by the global management component according to the partitions of the task to be processed.

[0103] The shuffle write node can be a standalone, packaged node deployed outside the compute engine. In one embodiment, the shuffle write node can be deployed in conjunction with a mapping task of the compute engine, belonging to the same process as the mapping task and sharing the same JVM (Java Virtual Machine). In one example, the shuffle write node can be embedded in the mapping task on the compute engine side as an SDK (Software Development Kit).

[0104] Specifically, when a pending task is initiated, the global management component retrieves the partitions of the pending task, generates a mapping between shuffle processing nodes and partitions, and sends this mapping to the shuffle write node. In one example, if the pending task includes A partitions, the global management component can determine A currently available shuffle processing nodes and generate a one-to-one correspondence between the partitions and the shuffle processing nodes.

[0105] In step S320 , the key-value data corresponding to each partition is determined by the shuffle write node, and the key-value data corresponding to each partition is sent to the shuffle processing node corresponding to each partition according to the mapping relationship.

[0106] Specifically, after the mapping task processes the task to be processed to obtain key-value data, the key-value data is sent to the shuffle write node. The shuffle write node sends the key-value data corresponding to each partition to the shuffle processing node corresponding to each partition based on the mapping relationship between the partition and the shuffle processing node.

[0107] In one embodiment, shuffle write nodes and mapping tasks can have a one-to-one correspondence. That is, if there are M mapping tasks, a shuffle write node corresponding to each mapping task is deployed. Each shuffle write node is used to send the key-value data sent by the corresponding mapping task to the shuffle processing node corresponding to the partition.

[0108] In this embodiment, in step S230, the sorting logic corresponding to the meta-information is run by the shuffle processing node to sort the key-value data, which can be achieved by step S330. In step S330, the sorting logic is run by the shuffle processing node corresponding to each partition to sort the key-value data corresponding to each partition.

[0109] In the embodiment, the mapping relationship between the partition and the shuffle processing node is established by deploying the independently packaged shuffle write node and the plurality of shuffle processing nodes. Compared with the data isolation processing of each mapping task in the related art, the plurality of mapping task output data can be aggregated according to the partition, so that the reduction task does not need to pull data from the local file corresponding to each mapping task, thereby reducing the data IO (input / output) times and improving the input / output efficiency of the shuffle.

[0110] In an example embodiment, as shown in Figure 4 The first type of loader is deployed in the shuffle processing node, and the meta information includes the storage path of the sorting logic file package.

[0111] The class loader is responsible for loading classes, and generates an instance object for all classes loaded into the memory. Once a class is loaded, such as in the JVM, the same class will not be loaded again. As described in the above embodiment, each class corresponds to unique key class information. The sorting logic file package can refer to a file package formed by packaging a class written in code. For example, a Jar (a computer file format) package.

[0112] In the embodiment, in step S230, the sorting logic corresponding to the meta information is run by the shuffle processing node, which can be implemented by the following steps:

[0113] In step S410, the sorting logic file package stored in the storage path is obtained by the first type of loader in the shuffle processing node.

[0114] In step S420, the sorting logic file package is loaded by the first type of loader to run the sorting logic.

[0115] In one embodiment, when the to-be-processed task is started, the storage path of the sorting logic file package can also be obtained, and the storage path is registered to the global management component, so that the shuffle processing node obtains the storage path from the global management component. The first type of loader in the shuffle processing node obtains the sorting logic file package stored in the storage path, and loads the sorting logic file package to run the sorting logic file package to sort the key value data under each partition.

[0116] In another embodiment, when the to-be-processed task is started, the sorting logic file package of the to-be-processed task can also be obtained. The sorting logic file package is uploaded to the file system. The storage path of the sorting logic file package in the file system is obtained, and the storage path is registered to the global management component, so that the shuffle processing node obtains the storage path from the global management component. The first type of loader in the shuffle processing node obtains the sorting logic file package stored in the storage path, and loads the sorting logic file package to run the sorting logic file package to sort the key value data under each partition.

[0117] In another embodiment, there may be a one-to-one correspondence between the tasks to be processed and the first class loader. A corresponding first class loader is instantiated for each task to be processed, which can ensure task-level class loading isolation and avoid mutual influence between classes.

[0118] Figure 5 The schematic diagram of processing key-value data based on the specified sorting logic file package is shown as an example. Figure 5 As shown, this can be achieved by following the steps below:

[0119] (1) When a pending task (Job) is started, the sorting logic file package (Jar package) containing the custom sorting logic is first uploaded to the HDFS remote storage.

[0120] (2) Upload the storage path of the sorting logic file package in HDFS and the key class information of the sorting logic file package to the global management component.

[0121] (3) The shuffle processing node obtains the metadata of the task to be processed from the global management component. The metadata includes the storage path of the sorting logic file package.

[0122] (4) When the shuffle processing node determines that the key-value data has reached a certain threshold and needs to be output to HDFS, it dynamically pulls the Jar package from HDFS to the local computer on demand and loads the corresponding class into memory.

[0123] This embodiment supports user-defined upload of sorting logic file packages, which can flexibly meet users' high-level sorting needs. By deploying the first class loader to implement task-level class loading, the loading of classes of different tasks will not affect each other, achieving task-level class loading isolation, thereby ensuring data consistency.

[0124] In an exemplary embodiment, in some cases, the amount of tasks submitted through the preset tool is large and has the same key class information. For example, in some systems, 90% of the tasks are submitted through Hive (a data warehouse tool), and the tasks to be processed submitted through Hive have the same key class information. Therefore, in this embodiment, a second class loader is also deployed in the shuffle processing node. The second class loader can be regarded as a cache layer of the first class loader. In step S410, before the first class loader in the shuffle processing node obtains the sorting logic file package stored in the storage path, the following processing can also be performed.

[0125] Specifically, after the shuffling processing node receives the key value data corresponding to each partition, when it is determined that the task to be processed is submitted through the preset tool, the first class loader is used to determine whether there is a sorting logic corresponding to the preset tool in the second class loader. If it is determined through the first class loader that there is no sorting logic corresponding to the preset tool in the second class loader, then the storage path of the sorting logic file package is obtained, and the sorting logic file package stored in the storage path is obtained to sort the key value data. If it is determined through the first class loader that there is a sorting logic corresponding to the preset tool in the second class loader, then the sorting logic can be loaded through the first class loader to sort the key value data of each partition.

[0126] In this embodiment, by predicting the sorting logic corresponding to the task to be processed based on the submission tool of the task to be processed, the number of times the sorting logic file package is pulled and loaded can be reduced, thereby speeding up the processing efficiency of key-value data, reducing the network and disk IO consumption caused by loading the sorting logic file package, and improving the performance of the shuffling service.

[0127] In an exemplary embodiment, a third class loader is also deployed in the shuffle processing node. The third class loader can be considered a cache layer for the first class loader and can be the same loader as the second class loader. In this embodiment, the meta-information also includes key class information of the task to be processed. In step S410, before obtaining the sorting logic file package stored in the storage path through the first class loader in the shuffle processing node, the process also includes: determining, through the first class loader, whether the sorting logic corresponding to the key class information does not exist in the third class loader.

[0128] Specifically, after receiving the key value data corresponding to each partition, the shuffle processing node determines, through the first class loader, whether the sorting logic corresponding to the key class information exists in the third class loader. If not, the node proceeds to obtain the storage path of the sorting logic file package and loads the sorting logic file package stored in the storage path. If so, the node executes the sorting logic corresponding to the key class information through the first class loader.

[0129] In one embodiment, key information, preset tools, and priorities of sorting logic file packages can be configured, and the key value data is sorted using the sorting method with the highest priority.

[0130] In this embodiment, by carrying key class information in the task to be processed and pre-determining whether the sorting logic corresponding to the key class information already exists, the number of times the sorting logic file package is pulled and loaded can be reduced, thereby speeding up the processing efficiency of key-value data and reducing the network and disk IO consumption caused by loading the sorting logic file package.

[0131] In a specific embodiment, Figure 6It is understood that, when there are multiple shuffle processing nodes, Figure 6 Represents the workflow of each shuffle processing node. Figure 6 In the example, the second and third class loaders are the same loader ( Figure 6 The CommonClassLoader in is a globally shared class loader. The number of first class loaders (SessionClassLoader) includes multiple. There is a one-to-one correspondence between the tasks to be processed and the first class loaders. This can be achieved by the following steps:

[0132] (1) When the shuffle processing node starts, it obtains the metadata of the task to be processed from the global management component and preloads the existing sorting logic file package from the local database through the second class loader.

[0133] (2) Using the first class loader corresponding to the task to be processed, the second class loader is used to load whether there is a sorting logic corresponding to the key class information of the preset tool. Alternatively, using the first class loader corresponding to the task to be processed, the second class loader is used to load whether there is a sorting logic corresponding to the key class information of the task to be processed.

[0134] (3) If none of the items in (2) exist, a sorting logic file package is obtained from HDFS according to the storage path contained in the metadata, and the sorting logic file package is run to sort the key value data corresponding to each partition.

[0135] Figure 7 is a flowchart of a method for processing key-value data according to an exemplary embodiment. Figure 7 As shown, this method is applied to the shuffle service deployed outside the computing engine. Figure 9 A schematic diagram of a shuffling service is shown as an example. Figure 9 As shown in the figure, the shuffle service includes independently packaged task management components (App Shuffle Master, ASM), global management components (Shuffle Master), shuffle writer nodes (Shuffle Writer), shuffle processing nodes (Shuffle Worker), and shuffle reader nodes (Shuffle Reader). The following describes the functions of each component.

[0136] Global management component: responsible for global resource scheduling, global task management, lifecycle management of shuffle processing nodes, heartbeat requests of shuffle processing nodes, etc.

[0137] The Task Management component can be deployed alongside the Task Manager (ApplicationMaster) in the compute engine, belonging to the same process and sharing the same JVM. In one example, the Task Management component can be embedded in the Task Manager as an SDK. The Task Management component is responsible for managing resources for individual tasks, processing RPC (Remote Procedure Call) requests for shuffle write and shuffle read nodes, and managing the lifecycles of shuffle write and shuffle read nodes.

[0138] Shuffle write nodes: These nodes can be embedded in the map tasks on the compute engine as SDKs. They are responsible for sending the key-value data processed by the map tasks to the corresponding shuffle processing nodes according to the partition dimension. The shuffle processing nodes safely exit after fully persisting the sorted key-value data.

[0139] Shuffle processing node: Responsible for aggregating and sorting key-value data according to the partition dimension and writing it to the remote HDFS. After the write is completed, the task management component and the shuffle writing node are notified of the persistent results.

[0140] The shuffle read node can be deployed alongside the reduce task on the compute engine side, belonging to the same process and sharing the same JVM. In one embodiment, the shuffle read node can be embedded in the reduce task as an SDK. In another embodiment, the shuffle read node and the reduce task can have a one-to-one correspondence. The shuffle read node is responsible for pulling the shuffle file set to be processed from HDFS, performing local deduplication based on the consistency metadata, and returning it to the compute engine side.

[0141] exist Figure 8 Based on the shuffle service shown, take the task to be processed as Mapreduce Job as an example, refer to Figure 7 , which explains how to process key-value data.

[0142] In step S702, a Mapreduce Job (ie, a task manager in Mapreduce) is started, and the task management component is started accordingly.

[0143] In step S704, the sorting logic file package corresponding to the Mapreduce Job is stored in HDFS, and the corresponding storage path is obtained. The metadata of the Mapreduce Job is registered in the global management component. The metadata includes the storage path and key information of the sorting logic file package of the Mapreduce Job.

[0144] In step S706, the task management component applies for resources from the global management component, requesting the global management component to determine the mapping relationship between the partitions of the task to be processed and the shuffle processing nodes. After the application is successful, the shuffle service mode begins.

[0145] In step S708, the task manager starts the mapping task. The shuffle write node starts with the mapping task, and both share the same process. After starting, the shuffle write node obtains the mapping relationship between the partition and the shuffle processing node from the task management component.

[0146] In step S710, each mapping task writes the processed key-value data into the local buffer of the corresponding shuffle write node through the interface. The shuffle write node then actively sends the key-value data in the buffer to the shuffle processing node corresponding to the partition according to the mapping relationship between the partition and the shuffle processing node.

[0147] In step S712, each shuffle processing node sorts the key value data according to the partition dimension and persists the sorted key value data to HDFS. Sorting the key value data can be specifically achieved by the following steps:

[0148] (1) When a Mapreduce job is submitted using the default tool (Hive), the first class loader determines whether the second class loader has sorting logic corresponding to the Hive key class information. If so, the sorting logic corresponding to the Hive key class information is run to sort the key value data in each partition.

[0149] (2) When the Mapreduce Job is not submitted through Hive, the first class loader determines whether the sorting logic corresponding to the Mapreduce Job's key class information exists in the second class loader. If so, the sorting logic corresponding to the Mapreduce Job's key class information is run to sort the key value data under each partition.

[0150] (3) When the sorting logic corresponding to the key class information of the Mapreduce Job does not exist in the second class loader, the storage path of the sorting logic file package is obtained from the global management component, the sorting logic file package is obtained from the storage path, and the sorting logic file package is loaded to sort the key value data under each partition.

[0151] In step S714, each shuffle processing node sends the sorted key-value data of each partition to the task management component in the storage path of HDFS.

[0152] In step S716, after the map phase completes, the task manager starts the reduce task. The shuffle read node starts with the reduce task. After starting, the shuffle read node obtains the storage path corresponding to each partition from the task manager component, reads the key-value data from HDFS based on the storage path corresponding to each partition, performs local deduplication, and returns the read key-value data to the reduce task on the compute engine.

[0153] It should be understood that, although the various steps in the above flow chart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above flow chart may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0154] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can be referred to each other, and each embodiment focuses on the differences from other embodiments. For related parts, please refer to the description of other method embodiments.

[0155] Figure 9 FIG. 1 is a block diagram of a key-value data processing device X00 according to an exemplary embodiment. Figure 9 The device includes an acquisition module X02, a mapping module X04, and a data sorting module X06.

[0156] The acquisition module X02 is configured to execute the acquisition of tasks to be processed and the metadata corresponding to the tasks to be processed; the mapping module X04 is configured to execute the processing of the tasks to be processed through the mapping task end in the computing engine to obtain key-value data, and transmit the key-value data to the shuffle processing node, which is a node independently encapsulated outside the computing engine; the data sorting module X06 is configured to execute the sorting logic corresponding to the metadata through the shuffle processing node to sort the key-value data.

[0157] In an exemplary embodiment, a first class loader is deployed in the shuffle processing node, and the metadata includes the storage path of the sorting logic file package; the data sorting module X06 includes: a file package acquisition unit, configured to execute the acquisition of the sorting logic file package stored in the storage path through the first class loader in the shuffle processing node; a first loading unit, configured to execute the loading of the sorting logic file package through the first class loader to run the sorting logic.

[0158] In an exemplary embodiment, a second class loader is also deployed in the shuffling processing node; the data sorting module X06 also includes: a submission tool determination unit, which is configured to execute, when the task to be processed is submitted through a preset tool, determining through the first class loader that there is no sorting logic corresponding to the preset tool in the second class loader.

[0159] In an exemplary embodiment, the data sorting module X06 further includes: a second loading unit, configured to execute the sorting logic corresponding to the preset tool through the first class loader when it is determined through the first class loader that there is sorting logic corresponding to the preset tool in the second class loader.

[0160] In an exemplary embodiment, a third class loader is also deployed in the shuffling processing node; the meta information also includes key class information of the task to be processed; the data sorting module X06 also includes: a query unit, configured to execute through the first class loader to determine whether there is a sorting logic corresponding to the key class information in the third class loader.

[0161] In an exemplary embodiment, the data sorting module X06 further includes: a third loading unit, configured to execute the sorting logic corresponding to the key class information when it is determined by the first class loader that the sorting logic corresponding to the key class information already exists in the third class loader.

[0162] In an exemplary embodiment, the device X00 also includes: a registration module, configured to execute the registration of metadata to the global management component, which is a component independently packaged outside the computing engine; an information sending module, configured to execute the sending of metadata to the shuffling processing node through the global management component.

[0163] In an exemplary embodiment, there are multiple shuffle processing nodes; the mapping module X04 includes: a mapping relationship acquisition unit, configured to execute the acquisition of the mapping relationship between the partition and the shuffle processing node through the shuffle write node, the mapping relationship is pre-constructed by the global management component according to the partition of the task to be processed, and the shuffle write node is a node independently encapsulated outside the computing engine; a data sending unit, configured to execute the determination of the key value data corresponding to each partition through the shuffle write node, and send the key value data corresponding to each partition to the shuffle processing node corresponding to each partition according to the mapping relationship; a data sorting module X06, configured to execute the sorting logic through the shuffle processing node corresponding to each partition to sort the key value data corresponding to each partition.

[0164] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0165] Figure 10 1 is a block diagram of an electronic device S00 for processing key-value data according to an exemplary embodiment. For example, the electronic device S00 may be a server. Figure 10 The electronic device S00 includes a processing component S20, which further includes one or more processors and a memory resource represented by a memory S22 for storing instructions executable by the processing component S20, such as an application. The application stored in the memory S22 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component S20 is configured to execute the instructions to perform the above method.

[0166] The electronic device S00 may further include a power supply component S24 configured to perform power management of the electronic device S00, a wired or wireless network interface S26 configured to connect the electronic device S00 to a network, and an input / output (I / O) interface S28. The electronic device S00 may operate based on an operating system stored in the memory S22, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

[0167] In an exemplary embodiment, a computer-readable storage medium including instructions is further provided, such as a memory S22 including instructions. The instructions can be executed by a processor of the electronic device S00 to perform the above method. The storage medium can be a computer-readable storage medium, such as a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0168] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions, and the instructions can be executed by a processor of the electronic device S00 to implement the above method.

[0169] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. can also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments and will not be described one by one here.

[0170] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the features disclosed herein. It is intended that the disclosure be construed as including any paterns of this disclosure that can be derived from the description and illustrations presented herein without departing from the scope and spirit of the disclosure. The specification and examples are exemplary only, with the true scope and spirit of the disclosure being indicated by the claims.

[0171] It is to be understood that the disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the disclosure is limited only by the claims that follow.

Claims

1. A method for processing key-value data, characterized in that: The method comprises: Obtaining tasks to be processed and meta information corresponding to the tasks to be processed; Processing the task to be processed by a mapping task in a computing engine to obtain key-value data, and transmitting the key-value data to a shuffle processing node, which is a node independently encapsulated outside the computing engine; Running, by the shuffle processing node, a sorting logic corresponding to the meta-information to sort the key-value data, the sorting logic including a sorting logic corresponding to the meta-information of the to-be-processed task determined based on a mapping relationship between pre-deployed sorting logic and meta-information; Wherein, a first class loader is deployed in the shuffling processing node, and the meta information includes a storage path of the sorting logic file package; The running of the sorting logic corresponding to the meta-information by the shuffle processing node includes: Acquire the sorting logic file package stored in the storage path through the first class loader in the shuffle processing node; The sorting logic file package is loaded through the first class loader to run the sorting logic.

2. The method for processing key-value data according to claim 1, wherein: A second class loader is also deployed in the shuffling processing node; Before acquiring the sorting logic file package stored in the storage path through the first class loader in the shuffle processing node, the method further includes: In a case where the to-be-processed task is submitted through a preset tool, it is determined through the first class loader that no sorting logic corresponding to the preset tool exists in the second class loader.

3. The method for processing key-value data according to claim 2, characterized in that: The method further comprises: When it is determined through the first class loader that the second class loader contains the sorting logic corresponding to the preset tool, the first class loader executes the sorting logic corresponding to the preset tool.

4. The method for processing key-value data according to claim 1, wherein: A third class loader is also deployed in the shuffle processing node; the meta information also includes key class information of the task to be processed; Before acquiring the sorting logic file package stored in the storage path through the first class loader in the shuffle processing node, the method further includes: It is determined through the first class loader that no sorting logic corresponding to the key class information exists in the third class loader.

5. The method for processing key-value data according to claim 4, characterized in that: The method further comprises: When it is determined through the first class loader that the sorting logic corresponding to the key class information already exists in the third class loader, the sorting logic corresponding to the key class information is executed.

6. The method for processing key-value data according to any one of claims 1 to 5, characterized in that: The method further comprises: Registering the meta information to a global management component, where the global management component is a component independently packaged outside the computing engine; The meta information is sent to the shuffle processing node through the global management component.

7. The method for processing key-value data according to claim 6, characterized in that: The number of the shuffling processing nodes is multiple; The transmitting the key-value data to the shuffle processing node includes: Acquire a mapping relationship between a partition and a shuffle processing node through a shuffle write node, wherein the mapping relationship is pre-built by the global management component according to the partition of the task to be processed, and the shuffle write node is a node independently encapsulated outside the computing engine; Determine the key value data corresponding to each partition through the shuffle write node, and send the key value data corresponding to each partition to the shuffle processing node corresponding to each partition according to the mapping relationship; The step of running a sorting logic corresponding to the meta-information through the shuffle processing node to sort the key-value data includes: The sorting logic is run through the shuffle processing node corresponding to each of the partitions to sort the key-value data corresponding to each of the partitions.

8. A key-value data processing device, characterized in that: The device comprises: An acquisition module is configured to acquire tasks to be processed and meta information corresponding to the tasks to be processed; a mapping module configured to process the to-be-processed task through a mapping task in a computing engine to obtain key-value data, and transmit the key-value data to a shuffle processing node, wherein the shuffle processing node is a node independently encapsulated outside the computing engine; a data sorting module configured to execute, through the shuffle processing node, a sorting logic corresponding to the meta-information to sort the key-value data, wherein the sorting logic is based on a sorting logic corresponding to the meta-information of the to-be-processed task obtained from a mapping relationship between pre-deployed sorting logic and meta-information; Wherein, a first class loader is deployed in the shuffling processing node, and the meta information includes a storage path of the sorting logic file package; The data sorting module includes: A file package acquiring unit is configured to acquire the sorting logic file package stored in the storage path through the first class loader in the shuffle processing node; The first loading unit is configured to load the sorting logic file package through the first class loader to run the sorting logic.

9. The key-value data processing device according to claim 8, characterized in that: A second class loader is also deployed in the shuffling processing node; The data sorting module further includes: The submission tool determination unit is configured to, when the to-be-processed task is submitted through a preset tool, determine, through the first class loader, that no sorting logic corresponding to the preset tool exists in the second class loader.

10. The key-value data processing device according to claim 9, characterized in that: The data sorting module further includes: The second loading unit is configured to execute, when it is determined by the first class loader that the sorting logic corresponding to the preset tool exists in the second class loader, running the sorting logic corresponding to the preset tool through the first class loader.

11. The key-value data processing device according to claim 8, characterized in that: A third class loader is also deployed in the shuffle processing node; the meta information also includes key class information of the task to be processed; The data sorting module further includes: The query unit is configured to execute, through the first class loader, a determination that a sorting logic corresponding to the key class information does not exist in the third class loader.

12. The key-value data processing device according to claim 11, characterized in that: The data sorting module further includes: The third loading unit is configured to execute the sorting logic corresponding to the key class information when it is determined by the first class loader that the sorting logic corresponding to the key class information already exists in the third class loader.

13. The key-value data processing device according to any one of claims 8 to 12, characterized in that: The device further comprises: a registration module configured to register the meta information with a global management component, wherein the global management component is a component independently packaged outside the computing engine; The information sending module is configured to send the meta information to the shuffle processing node through the global management component.

14. The key-value data processing device according to claim 13, characterized in that: The number of the shuffling processing nodes is multiple; the mapping module includes: a mapping relationship acquisition unit configured to acquire a mapping relationship between a partition and a shuffle processing node through a shuffle write node, wherein the mapping relationship is pre-constructed by the global management component according to the partition of the task to be processed, and the shuffle write node is a node independently encapsulated outside the computing engine; a data sending unit configured to determine the key-value data corresponding to each of the partitions through the shuffle write node, and send the key-value data corresponding to each of the partitions to the shuffle processing node corresponding to each of the partitions according to the mapping relationship; The data sorting module is configured to execute the sorting logic through the shuffle processing nodes corresponding to each of the partitions to sort the key-value data corresponding to each of the partitions.

15. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the key-value data processing method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the key-value data processing method according to any one of claims 1 to 7.

17. A computer program product comprising instructions, characterized in that: When the instruction is executed by a processor of an electronic device, the electronic device is enabled to execute the key-value data processing method according to any one of claims 1 to 7.

Images