Data processing method and device based on YARN

Through the YARN node management component, the problem of low resource utilization of the k8s cluster is solved by applying for processing resources for the k8s cluster and the task executor is independently deployed, and the flexible management and efficient utilization of resources are achieved.

CN119003156AInactive Publication Date: 2024-11-22BEIJING VOLCANO ENGINE TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411018080.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When using YARN to manage k8s cluster resources, the resource utilization rate is low and external management and control systems need to be deployed for resource expansion or reduction, resulting in complex solutions and inflexible resource release.

Method used

Through the YARN node management component, apply for processing resources in the k8s cluster for data processing tasks, deploy task executors independent of the node management components, and release resources after the task is completed, improving resource utilization efficiency.

Benefits of technology

It improves the resource utilization rate of the k8s cluster, simplifies resource management, avoids the limitations of resource binding to node management components, and realizes flexible resource release and management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003156A_ABST
    Figure CN119003156A_ABST
Patent Text Reader

Abstract

The invention discloses a data processing method based on a YARN (another resource coordinator), and the method comprises the steps: receiving a data processing task from a big data processing platform, and applying a processing resource in a first cluster for the data processing task through a YARN NM (node management) assembly, and the first cluster is a cluster to which k8s (kubernetes) is applied. And deploying a first pod in the first cluster based on the target resource applied for the data processing task, deploying a task executor in the first pod, and executing the data processing task by using the task executor in the first pod. According to the method and the device, the task executor and the NM component are not bound in one pod any more, and the task executor is deployed in the first pod independent of the pod where the NM component is located, so that resources occupied by the first pod can be released according to the life cycle of the task executor, and the resource utilization efficiency of the first cluster is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a YARN-based data processing method and device. Background Art

[0002] Yet Another Resource Negotiator (YARN) is a general resource management and scheduling platform that can provide unified resource management and scheduling services for upper-layer applications.

[0003] YARN is usually combined with a big data processing platform. As a specific example, various computing clusters can be integrated together, and YARN can receive data processing tasks from the big data processing platform and allocate resources to the task from the aforementioned integrated cluster, so as to utilize the allocated resources to execute the data processing task.

[0004] At present, when the aforementioned computing cluster includes a cluster to which k8s (full English name: kubernetes) is applied, how to use YARN to schedule the resources of the computing cluster is a problem that has yet to be solved. Summary of the invention

[0005] In order to solve or at least partially solve the above technical problems, the present application provides a YARN-based data processing method and device.

[0006] In a first aspect, the present application provides a YARN-based data processing method, which is applied to a server on which YARN technology is deployed. The method includes:

[0007] Receive data processing tasks from the big data processing platform;

[0008] Using a YARN node management component to apply for processing resources in a first cluster for the data processing task, where the first cluster is a cluster to which k8s is applied;

[0009] Deploy a first pod in a first cluster based on the target resources applied for the data processing task by the YARN node management component, and deploy a task executor in the first pod;

[0010] The data processing task is executed in the first pod using the task executor.

[0011] Optionally, the resources managed by the server only include clusters to which k8s is applied, the clusters to which k8s is applied include the first cluster, and the YARN node management component is deployed in a second pod included in the first cluster, and the second pod is different from the first pod.

[0012] Optionally, the resources managed by the server also include a second cluster to which an elastic computing service is applied, and the YARN node management component is deployed in the second cluster.

[0013] Optionally, the method further includes:

[0014] When available resources of the second cluster are insufficient, the YARN node management component is deployed in the second cluster.

[0015] Optionally, the deploying a task executor in the first pod includes:

[0016] An initialization container and a task execution container are deployed in the first pod, the initialization container is used to perform a resource localization operation, the resource localization operation is used to complete the environment preparation for executing the data processing task, and the task execution container is used to execute the data processing task based on the environment prepared by the initialization container.

[0017] Optionally, the resource localization operation includes:

[0018] Download and store user-submitted applications, configuration items of the big data processing platform, execution scripts, and execution environments;

[0019] Based on the stored application, configuration items, execution script and execution environment, the environment preparation for executing the data processing task is completed.

[0020] Optionally, the method further includes:

[0021] The YARN node management component is used to monitor the execution status of the task executor.

[0022] Optionally, the method further includes:

[0023] After the task executor completes executing the data processing task, the first pod is released.

[0024] Optionally, the method further includes:

[0025] After the task executor completes executing the data processing task and before the first pod is released, an execution log of the task executor is obtained and the execution log is stored in a file system.

[0026] In a second aspect, the present application provides a YARN-based data processing device, which is applied to a server on which YARN technology is deployed, and the device includes:

[0027] A receiving unit, used for receiving data processing tasks from a big data processing platform;

[0028] An application unit, configured to apply for processing resources in a first cluster for the data processing task by using a YARN node management component, wherein the first cluster is a cluster to which k8s is applied;

[0029] A first deployment unit, configured to deploy a first pod in a first cluster based on the target resources applied for the data processing task by the YARN node management component, and to deploy a task executor in the first pod;

[0030] An execution unit, configured to execute the data processing task by using the task executor in the first pod.

[0031] Optionally, the resources managed by the server only include clusters to which k8s is applied, the clusters to which k8s is applied include the first cluster, and the YARN node management component is deployed in a second pod included in the first cluster, and the second pod is different from the first pod.

[0032] Optionally, the resources managed by the server also include a second cluster to which an elastic computing service is applied, and the YARN node management component is deployed in the second cluster.

[0033] Optionally, the device further comprises:

[0034] The second deployment unit is configured to deploy the YARN node management component in the second cluster when available resources of the second cluster are insufficient.

[0035] Optionally, the first deployment unit is used to:

[0036] An initialization container and a task execution container are deployed in the first pod, the initialization container is used to perform a resource localization operation, the resource localization operation is used to complete the environment preparation for executing the data processing task, and the task execution container is used to execute the data processing task based on the environment prepared by the initialization container.

[0037] Optionally, the resource localization operation includes:

[0038] Download and store user-submitted applications, configuration items of the big data processing platform, execution scripts, and execution environments;

[0039] Based on the stored application, configuration items, execution script and execution environment, the environment preparation for executing the data processing task is completed.

[0040] Optionally, the device further comprises:

[0041] A status monitoring unit is used to monitor the execution status of the task executor using the YARN node management component.

[0042] Optionally, the device further comprises:

[0043] A releasing unit is used to release the first pod after the task executor completes executing the data processing task.

[0044] Optionally, the device further comprises:

[0045] A log processing unit is used to obtain the execution log of the task executor after the task executor completes the data processing task and before the first pod is released, and store the execution log in a file system.

[0046] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising a processor and a memory;

[0047] The processor is configured to execute instructions stored in the memory, so that the device performs the method as described in any one of the first aspects above.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, wherein the instructions instruct a device to execute a method as described in any one of the above first aspects.

[0049] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect above.

[0050] Compared with the prior art, the embodiments of the present application have the following advantages:

[0051] The present application provides a data processing method based on YARN, which is applied to a server that has deployed YARN technology. The server can receive data processing tasks from a big data processing platform, and use the YARN node manager (nodemanager, NM) component to apply for processing resources in the first cluster for the data processing task, and the first cluster is a cluster that applies k8s. The NM component can apply for target resources for the data processing task. Furthermore, a first pod can be deployed in the first cluster based on the target resources, and a task executor can be deployed in the first pod. Further, the data processing task can be executed using the task executor in the first pod. It can be seen that using this solution, assuming that the NM component is in the pod, this solution is not to directly deploy the task executor in the pod where the NM component is located, but to deploy the first pod based on the target resources applied by the NM component for the data processing task, and deploy the task executor in the first pod. That is: in this application, the task executor is no longer bound to the NM component in a pod. The task executor can be deployed in a first pod independent of the pod where the NM component is located, so that the resources occupied by the first pod can be released according to the life cycle of the task executor, which is beneficial to improve the resource utilization efficiency of the first cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1a A schematic diagram of managing k8s resources based on YARM is shown;

[0054] Figure 1b A schematic diagram of managing ECS ​​resources and k8s resources based on YARN is shown;

[0055] Figure 2 A flowchart of a YARN-based data processing method provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of a process for deploying a task executor provided in an embodiment of the present application;

[0057] Figure 4a A schematic diagram of a scenario of a YARN-based data processing method provided in an embodiment of the present application;

[0058] Figure 4bA schematic diagram of another YARN-based data processing method provided in an embodiment of the present application;

[0059] Figure 5 A schematic diagram of the structure of a YARN-based data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0061] For the convenience of description, the following terms related to YARN are explained first.

[0062] YARN resource manager (RM) component: It is used to process requests from clients (such as clients corresponding to the big data processing platform), coordinate the resources of the entire cluster, manage NM components, etc. The YARNRM component can be referred to as the RM component.

[0063] YARN NM component: It is used to manage the resources of the nodes to which it belongs and process the instructions from the RM component and the AM component. The YARN NM component can be referred to as the NM component for short.

[0064] Application master (AM) component: It is responsible for allocating tasks and monitoring the execution of tasks.

[0065] Container: It is an abstraction of resources managed by YARN. It can encapsulate multi-dimensional resources, for example, it can encapsulate multi-dimensional resources such as memory, central processing unit (CPU), disk, and network.

[0066] The inventor of this application has found through research that, currently, most of the resources managed by YARN are Elastic Compute Service (ECS) resources. In other words, YARN can manage the resources of clusters that use ECS, but it is difficult for YARN to manage the resources of clusters that use k8s. For the convenience of description, in the following description, clusters that use k8s are referred to as "k8s clusters", and clusters that use ECS are referred to as "ECS clusters".

[0067] In an example, if you want to implement YARN management of k8s cluster resources, you can deploy YARN-related components such as NM into pods and keep them resident. Among them: pod is the smallest deployable computing unit that can be created and managed in k8s. Pod consists of a set of closely related containers that run on the same node and share the same computing resources such as network namespace and storage volume, maximizing the advantages of resource sharing through cluster management.

[0068] For reference Figure 1a To understand, Figure 1a A schematic diagram of managing k8s resources based on YARM is shown.

[0069] like Figure 1a As shown, the k8s cluster includes at least one pod. Figure 1b Pod 1 and pod 2 are shown. RM components are deployed in pod 1, and NM components are deployed in pod 2. When a data processing task is received from a client, a task executor for processing the task is deployed in pod 2, and the task executor processes the data processing task.

[0070] In addition, the k8s cluster is also connected to the management and control system deployed outside the k8s cluster. The management and control system controls the k8s cluster to run elastic expansion / contraction instances, thereby expanding or shrinking the resources of the k8s cluster.

[0071] See also Figure 1b To understand, Figure 1b A schematic diagram of managing ECS ​​resources and k8s resources based on YARN is shown.

[0072] like Figure 1b As shown, the ECS cluster includes several nodes ( Figure 1b 3 nodes are shown), one of which is deployed with an RM component and a corresponding agent, wherein the agent corresponding to the RM component is used to pull up the RM component. Accordingly, the RM component manages multiple NM components, and one NM component corresponds to one node. Accordingly, each NM component also corresponds to an agent, and the agent corresponding to the NM component is used to pull up the NM component.

[0073] The k8s cluster includes at least one pod. Figure 1b Pod1 and pod2 are shown. NM components are deployed in both pod1 and pod2.

[0074] The data processing task submitted by the client is received by the RM component. In one example, the RM component can assign the data processing task to the NM component on a node (eg, node 2) in the ECS cluster, and the NM component on the node processes the data processing task.

[0075] In another example, the RM component can assign the data processing task to the k8s cluster for processing. The RM component can interact with the management and control system to assign the data processing task to the k8s cluster for processing. For example, if the data processing task is assigned to pod1 and / or pod2 for processing, a task executor for processing the task can be deployed in pod1 and / or pod2, and the task executor processes the data processing task.

[0076] However, the above method has the problem of low resource utilization of the k8s cluster. Because in this way, the NM component and the task executor that processes the data processing tasks submitted by the client are deployed in the same pod. Accordingly, when the resources of the k8s cluster are expanded or reduced, they are processed according to the NM granularity, and the NM resides in the pod, which makes it impossible to flexibly release the resources occupied by the NM, resulting in low resource utilization of the k8s cluster. Moreover, in this way, it is also necessary to deploy an external management and control system to expand or reduce the resources of the k8s cluster, and the solution deployment is also relatively complicated.

[0077] In order to solve the above problems, the embodiments of the present application provide a data processing method and device based on YARN.

[0078] Various non-limiting implementations of the present application are described in detail below in conjunction with the accompanying drawings.

[0079] Exemplary Methods

[0080] See also Figure 2 , which is a flow chart of a YARN-based data processing method provided in an embodiment of the present application. Figure 2 The method shown can be applied to a server deployed with the YARN technology. The method can include the following steps S101-S104, for example.

[0081] S101: receiving a data processing task from a big data processing platform.

[0082] In the present application, receiving a data processing task from a big data processing platform may be receiving a data processing task from a client corresponding to the big data processing platform. The present application embodiment does not specifically limit the data processing task, and the data processing task may be related to a business scenario corresponding to the big data processing platform, which is not limited here.

[0083] The embodiment of the present application does not specifically limit the big data processing platform, and the big data processing platform may be a currently existing, under development, or future big data processing platform. As an example, the big data processing platform may be hive or spark.

[0084] In one example, the data processing task may be received by a YARN RM component.

[0085] In an example, if the server manages resources of both the ECS cluster and the k8s cluster, the YARN RM component can be deployed on a node of the ECS cluster. For example, the ECS cluster includes a second cluster, and the YARN RM component is deployed on a first node on the ECS cluster.

[0086] In another example, if the resources managed by the server only include a k8s cluster, the YARN RM component can be deployed in a pod (e.g., a third pod) of the k8s cluster, wherein the k8s cluster includes at least the first cluster.

[0087] S102: Using the YARN NM component to apply for processing resources in the first cluster for the data processing task.

[0088] After receiving the data processing task, processing resources may be applied for the data processing task first. In the present application, YARN NM components may be used to apply for processing resources in the first cluster for the data processing task.

[0089] In one example, if the resources managed by the server only include the k8s cluster, the YARN NM component is deployed in the second pod included in the first cluster. As a specific example, the YARN NM component may be deployed in the second pod when the first cluster starts running. The second pod may be deployed with the YARN NM component and the agent corresponding to the YARN NM component, and the agent corresponding to the YARN NM component is used to pull up the YARN NM component.

[0090] In another example, if the resources managed by the server include both k8s clusters and ECS clusters, for example, the resources managed by the server include a first cluster and a second cluster to which ECS is applied, then in one example, after the aforementioned YARN RM component receives a data processing task, it can preferentially utilize the resources of the second cluster to process the data processing task, and only utilize the resources of the first cluster to process the data processing task when the resources of the second cluster are insufficient. In this scenario, the aforementioned YARN NM component can be deployed in the second cluster. As an example, the YARNNM component can be deployed on the second node in the second cluster.

[0091] In a scenario where the YARN NM component can be deployed on the second cluster, the YARNRM component deployed in the second cluster can deploy the YARN NM component in the second cluster when available resources of the second cluster are insufficient. The available resources of the second cluster are insufficient, which may be that the second cluster does not have enough resources to process the data processing task.

[0092] As a specific example, after receiving the data processing task, the YARN RM component can determine whether the second cluster has sufficient resources to process the data processing task based on the resource usage of the second cluster. If the second cluster has sufficient resources to process the data processing task, the resources of the second cluster can be directly used to process the data processing task. If the second cluster does not have sufficient resources to process the data processing task, the YARN RM component can create a second node in the second cluster and deploy the YARN RM component in the second node. In one example, the YARN RM component can be different from other ordinary YARN RM components deployed in the second cluster. The YARN RM component can be a YARN RM component with a plug-in, and the YARN RM component with a plug-in is used to schedule and manage the resources of the first cluster.

[0093] As another specific example, the second node may also be created when the second cluster starts running, and when the second cluster does not have sufficient resources to process the data processing task, the YARN RM component uses the YARN RM component to apply for processing resources in the first cluster for the data processing task. When the second cluster has sufficient resources to process the data processing task, the YARN RM component directly uses the resources of the second cluster to process the data processing task, instead of using the YARN RM component to apply for processing resources in the first cluster for the data processing task.

[0094] In one example, the first cluster includes a resource scheduler, and the YARN RM component can apply for resources for the data processing task from the resource scheduler. As a specific example, the first cluster also provides an application programming interface (API) service for applying for resources from the resource scheduler, and the YARN RM component can apply for resources for the data processing task from the resource scheduler through the API service.

[0095] S103: Based on the target resources applied for the data processing task by the YARN NM component, a first pod is deployed in a first cluster, and a task executor is deployed in the first pod, where the first cluster is a cluster to which k8s is applied.

[0096] After the YARN NM component applies for processing resources in the first cluster for the data processing task, it can deploy a first pod in the first cluster based on the target processing resources applied for the data processing task, and deploy a task executor in the first pod. The embodiment of the present application does not specifically limit the task executor, and the task executor may be related to the data processing task. As a specific example, the task executor may be related to a big data processing platform that sends the data processing task. For example, if the data processing task comes from hive, the task executor may be a hive task executor. For another example, if the data processing task comes from spark, the task executor may be a spark task executor.

[0097] In one example, when S103 is implemented, reference may be made to Figure 3 To understand. Figure 3 A schematic diagram of a process for deploying a task executor provided in an embodiment of the present application. Figure 3 The deployment of the task executor may include the deployment of an initialization container and the deployment of a task execution container, wherein the initialization container is used to perform resource localization operations, and the task execution container is used to perform the aforementioned data processing tasks.

[0098] In the present application, the resource localization operation is used to complete the environmental preparation for executing the data processing task. In other words, the resource localization operation is used to download the resources required for executing the data processing task to the k8s cluster, thereby using the downloaded resources to complete the environmental preparation for executing the data processing task. Accordingly, the task execution container executes the data processing task based on the environment prepared by the initialization container. The environmental preparation for completing the execution of the data processing task mentioned here can, for example, be the generation of relevant dependency packages and configuration files required to execute the data processing task.

[0099] The embodiments of the present application do not specifically limit the resources required for executing the data processing task. The resources required for executing the data processing task may include other resources that are unrelated to the user-defined image when executing the data processing task. In this way, other resources unrelated to the user-defined image can be handed over to the initialization container for processing, instead of handing over both the other resources unrelated to the user-defined image and the user-defined image to the task execution container for processing. The resource localization operation can be decoupled from the processing logic of the user-defined image, so that the resource localization operation can be applied to a variety of different user-defined images.

[0100] As a specific example, the resources required to execute the data processing task include, but are not limited to: user-submitted applications, configuration items of the big data processing platform, execution scripts, and execution environments. Among them, the user-submitted application can be understood as software code written by the user. The configuration items of the big data processing platform can be configuration files related to the big data processing platform. For example, for a big data processing platform that adopts a map reduce algorithm, the configuration items of the big data processing platform can include configuration files related to map reduce. The execution script may include the script on which the execution of the aforementioned user-submitted application depends, and the execution script may be a script from the big data processing platform. The execution environment refers to the environment in which the aforementioned execution script and the user-submitted application run.

[0101] In an embodiment of the present application, before the initialization container and the task execution container are deployed in the first pod, an initialization operation and a startup operation may also be performed. Among them, the initialization operation is used to initialize the environment, for example, to set the image and initialize the execution time of the container in the first pod. The startup operation is used to start a container (such as an initialization container). After starting the container, the aforementioned initialization container is further run to perform a resource localization operation. In one example, if the initialization container fails to start, the replay container can be run to restore the aforementioned initialization container.

[0102] S104: Execute the data processing task in the first pod using the task executor.

[0103] After the task executor is deployed in the first pod, the data processing task can be executed by using the task executor in the first pod. As a specific example, the task execution container can be run in the first pod to execute the data processing task based on the environment prepared by the initialization container.

[0104] In one example, after the task executor completes the data processing task, the first pod can be released, wherein releasing the first pod refers to releasing the target resources occupied by the first pod, so that the target resources can be used to perform other operations, thereby improving the utilization of k8s resources. That is: in this application, the first pod can be managed according to the life cycle of the task executor. Using the solution of the embodiment of the present application, the task executor is no longer bound to the NM component in a pod, and the task executor can be deployed in a first pod independent of the pod where the NM component is located, so that the resources occupied by the first pod can be released according to the life cycle of the task executor, which is beneficial to improve the resource utilization efficiency of the first cluster. Using this solution, there is no need to deploy a management and control system independent of the k8s cluster to expand or shrink the resources of the k8s cluster, and the solution deployment is also relatively simple.

[0105] In one example, considering that the first pod will be released after the task executor completes the data processing task, the data (such as execution log) generated by the task executor in the first pod will be cleared accordingly. In other words, after the first pod is released, the execution log of the task executor can no longer be obtained.

[0106] In view of this, in one example, after the task executor completes the data processing task and before the first pod is released, the execution log of the task executor can be obtained and stored in the file system. For example, the storage path for storing the aforementioned execution log can be first determined, and the execution log can be read from the path, so as to further store the read execution log in the file system. In this way, if the processing process of the aforementioned data processing task needs to be backtracked later, the execution log can be read from the file system to backtrack the processing process of the aforementioned data processing task.

[0107] The embodiment of the present application does not specifically limit the file system, and the file system may be any system that can be used to store data.

[0108] In one example, when the task executor executes the data processing task, in order to facilitate determining the execution status of the task executor, the aforementioned YARN NM component can be used to monitor the execution status of the task executor. Among them, the status of the task executor may include: normal execution or abnormal execution. The status of the task executor may also include the execution progress of the data processing task, or include other detailed information of the data processing task, which is not specifically limited in the embodiments of the present application.

[0109] In a specific example, the aforementioned YARN NM component can integrate the functions of the k8s client. Therefore, the aforementioned YARN NM component can use the listen / watch functions of the k8s client to monitor the execution status of the task executor.

[0110] The above introduces the YARN-based data processing method provided by the present application. Next, the application of the present application in two scenarios is introduced in conjunction with the accompanying drawings.

[0111] See also Figure 4a , which is a scenario diagram of a YARN-based data processing method provided in an embodiment of the present application.

[0112] like Figure 4a As shown, pod1 and pod2 are deployed in the k8s cluster, RM components are deployed in pod1, and NM components are deployed in pod2. The k8s cluster also provides an API service and a resource scheduler. When the RM component receives a data processing task from the big data processing platform, it can assign the data processing task to the NM component deployed in pod2. The NM component applies for resources for the data processing task from the resource scheduler through the API service. And using the applied target resources, pod3 (corresponding to the aforementioned first pod) is deployed in the k8s cluster, and the task executor is run in pod3 to perform at least the aforementioned data processing task.

[0113] See also Figure 4b , this figure is a scenario schematic diagram of another YARN-based data processing method provided in an embodiment of the present application.

[0114] like Figure 4b As shown, the ECS cluster includes several nodes ( Figure 4b 3 nodes are shown), where RM components and corresponding agents are deployed on node 1. Correspondingly, NM components and corresponding agents are deployed on the other two nodes. Among them, the NM component deployed on node 2 is a common component used to schedule ECS resources. The NM component deployed on node 3 is a component with a plug-in used to schedule k8s resources.

[0115] The k8s cluster provides API service and resource scheduler.

[0116] The data processing task submitted by the client is received by the RM component. When the RM component determines that the resources of the ECS cluster are insufficient, it deploys the NM component with a plug-in on node 3. The NM component with a plug-in applies for resources for the data processing task from the resource scheduler through the API service. And using the applied target resources, pod4 (corresponding to the aforementioned first pod) is deployed in the k8s cluster, and the task executor is run in pod4 to at least perform the aforementioned data processing task.

[0117] Exemplary Devices

[0118] Based on the method provided in the above embodiment, the embodiment of the present application further provides a device, which is described below in conjunction with the accompanying drawings.

[0119] See also Figure 5 , Figure 5 A schematic diagram of the structure of a YARN-based data processing device provided in an embodiment of the present application. Figure 5 The device 500 shown can be applied to a server deployed with YARN technology to execute the YARN-based data processing method provided by the above method embodiment. The device 500 can specifically include: a receiving unit 501, an application unit 502, a first deployment unit 503 and an execution unit 504.

[0120] A receiving unit 501 is used to receive a data processing task from a big data processing platform;

[0121] An application unit 502 is used to apply for processing resources in a first cluster for the data processing task by using a YARN node management component, where the first cluster is a cluster to which k8s is applied;

[0122] A first deployment unit 503 is used to deploy a first pod in a first cluster based on the target resources applied for the data processing task by the YARN node management component, and to deploy a task executor in the first pod;

[0123] The execution unit 504 is used to execute the data processing task by using the task executor in the first pod.

[0124] Optionally, the resources managed by the server only include clusters to which k8s is applied, the clusters to which k8s is applied include the first cluster, and the YARN node management component is deployed in a second pod included in the first cluster, and the second pod is different from the first pod.

[0125] Optionally, the resources managed by the server also include a second cluster to which an elastic computing service is applied, and the YARN node management component is deployed in the second cluster.

[0126] Optionally, the device further comprises:

[0127] The second deployment unit is configured to deploy the YARN node management component in the second cluster when available resources of the second cluster are insufficient.

[0128] Optionally, the first deployment unit 503 is configured to:

[0129] An initialization container and a task execution container are deployed in the first pod, the initialization container is used to perform a resource localization operation, the resource localization operation is used to complete the environment preparation for executing the data processing task, and the task execution container is used to execute the data processing task based on the environment prepared by the initialization container.

[0130] Optionally, the resource localization operation includes:

[0131] Download and store user-submitted applications, configuration items of the big data processing platform, execution scripts, and execution environments;

[0132] Based on the stored application, configuration items, execution script and execution environment, the environment preparation for executing the data processing task is completed.

[0133] Optionally, the device further comprises:

[0134] A status monitoring unit is used to monitor the execution status of the task executor using the YARN node management component.

[0135] Optionally, the device further comprises:

[0136] A releasing unit is used to release the first pod after the task executor completes executing the data processing task.

[0137] Optionally, the device further comprises:

[0138] A log processing unit is used to obtain the execution log of the task executor after the task executor completes the data processing task and before the first pod is released, and store the execution log in a file system.

[0139] Since the device 500 is a device corresponding to the YARN-based data processing method provided by the above method embodiment, the specific implementation of each unit of the device 500 is the same concept as the above method embodiment. Therefore, for the specific implementation of each unit of the device 500, reference can be made to the relevant description part of the above method embodiment, which will not be repeated here.

[0140] The embodiment of the present application also provides an electronic device, the electronic device comprising a processor and a memory;

[0141] The processor is used to execute the instructions stored in the memory, so that the device executes the YARN-based data processing method provided by the above method embodiment.

[0142] An embodiment of the present application provides a computer-readable storage medium, including instructions, wherein the instructions instruct a device to execute the YARN-based data processing method provided in the above method embodiment.

[0143] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the YARN-based data processing method provided by the above method embodiment.

[0144] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0145] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0146] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A data processing method based on YARN, characterized in that: Applied to a server on which YARN technology is deployed, the method includes: Receive data processing tasks from the big data processing platform; Using a YARN node management component to apply for processing resources in a first cluster for the data processing task, where the first cluster is a cluster to which k8s is applied; Deploy a first pod in a first cluster based on the target resources applied for the data processing task by the YARN node management component, and deploy a task executor in the first pod; The data processing task is executed in the first pod using the task executor.

2. The method according to claim 1, characterized in that: The resources managed by the server only include clusters to which k8s is applied, the clusters to which k8s is applied include the first cluster, the YARN node management component is deployed in a second pod included in the first cluster, and the second pod is different from the first pod.

3. The method according to claim 1, characterized in that: The resources managed by the server also include a second cluster to which the elastic computing service is applied, and the YARN node management component is deployed in the second cluster.

4. The method according to claim 3, characterized in that The method further comprises: When available resources of the second cluster are insufficient, the YARN node management component is deployed in the second cluster.

5. The method according to claim 1, characterized in that The deploying the task executor in the first pod includes: An initialization container and a task execution container are deployed in the first pod, the initialization container is used to perform a resource localization operation, the resource localization operation is used to complete the environment preparation for executing the data processing task, and the task execution container is used to execute the data processing task based on the environment prepared by the initialization container.

6. The method according to claim 5, characterized in that The resource localization operation includes: Download and store user-submitted applications, configuration items of the big data processing platform, execution scripts, and execution environments; Based on the stored application, configuration items, execution script and execution environment, the environment preparation for executing the data processing task is completed.

7. The method according to claim 1, characterized in that The method further comprises: The YARN node management component is used to monitor the execution status of the task executor.

8. The method according to claim 1, characterized in that The method further comprises: After the task executor completes executing the data processing task, the first pod is released.

9. The method according to claim 8, characterized in that The method further comprises: After the task executor completes executing the data processing task and before the first pod is released, an execution log of the task executor is obtained and the execution log is stored in a file system.

10. A data processing device based on YARN, characterized in that: Applied to a server deployed with YARN technology, the device includes: A receiving unit, used for receiving data processing tasks from a big data processing platform; An application unit, configured to apply for processing resources in a first cluster for the data processing task by using a YARN node management component, wherein the first cluster is a cluster to which k8s is applied; A first deployment unit, configured to deploy a first pod in a first cluster based on the target resources applied for the data processing task by the YARN node management component, and to deploy a task executor in the first pod; An execution unit, configured to execute the data processing task by using the task executor in the first pod.

11. An electronic device, characterized in that: The device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory so that the device executes the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The method comprises instructions, wherein the instructions instruct the device to execute the method according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Resource scheduling method and device based on kubernetes, equipment and storage medium

    CN112199194A

  • Task processing method and device

    CN113312165A

  • Cluster capacity expansion method and device, electronic equipment and computer readable storage medium

    CN114327846A

  • Spark cluster deployment method and device

    CN115964130A