A task processing method and device based on YARN

Abstract

The present application discloses a task processing method based on YARN (another resource coordinator), including: receiving data processing tasks from a big data processing platform, scheduling the resources of a k8s cluster, and utilizing the task computing container in a second pod deployed in the k8s cluster to perform the data processing tasks. A main application container is also deployed in the first pod in the k8s cluster. Based on the data processing task, the main application container sends a task allocation message including the address of the first pod and the address of the second pod to the task computing container in the k8s cluster. The task allocation message is used to allocate computing tasks to the task computing container. The task computing container is used to execute the computing tasks allocated by the main application container to the task computing container, thereby realizing the execution of the aforementioned data processing tasks. When this solution utilizes YARN to schedule the resources of the k8s cluster, the resources are utilized to execute data processing tasks from the big data processing platform.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a YARN-based task processing method and device. Background Art

[0002] Yet Another Resource Negotiator (YARN) is a general resource management and scheduling platform that can provide unified resource management and scheduling services for upper-layer applications.

[0003] YARN is usually combined with a big data processing platform. As a specific example, various computing clusters can be integrated together, and YARN can receive data processing tasks from the big data processing platform and allocate resources to the task from the aforementioned integrated cluster, so as to utilize the allocated resources to execute the data processing task.

[0004] At present, when YARN is used to schedule the resources of a cluster that uses k8s (full name: kubernetes), how to use the resources to execute tasks from a big data processing platform is a problem that has yet to be solved. Among them, the cluster that uses k8s can be referred to as a k8s cluster. Summary of the invention

[0005] In order to solve or at least partially solve the above technical problems, the present application provides a task processing method and device based on YARN.

[0006] In a first aspect, the present application provides a YARN-based task processing method, which is applied to a server on which YARN technology is deployed. The method includes:

[0007] Receive data processing tasks from the big data processing platform;

[0008] The resources of the k8s cluster are scheduled to utilize the task computing container deployed in the k8s cluster to execute the data processing task, wherein a main application container is also deployed in the k8s cluster, the main application container is deployed in a first pod in the k8s cluster, and the task computing container is deployed in a second pod in the k8s cluster. The main application container is used to send a task allocation message to the task computing container in the k8s cluster based on the data processing task, and the task allocation message is used to allocate computing tasks to the task computing container. The task computing container is used to execute the computing tasks allocated by the main application container to the task computing container, and the task allocation message includes the address of the first pod and the address of the second pod.

[0009] Optionally, the method further includes:

[0010] Apply for resources in the k8s cluster for the data processing task based on the YARN node management component;

[0011] The first pod and the second pod are deployed in the k8s cluster using the target resources applied for the data processing task by the YARN node management component.

[0012] Optionally, the resources managed by the server only include resources to which k8s is applied, and the resources to which k8s is applied include the k8s cluster. The YARN node management component is pre-deployed in a third pod included in the k8s cluster, and the third pod is different from the first pod and the second pod.

[0013] Optionally, the resources managed by the server also include a cluster to which an elastic computing service is applied, and the YARN node management component is deployed in the cluster to which the elastic computing service is applied.

[0014] Optional,

[0015] The task computing container is further used to send a status reporting message to the main application container, where the status reporting message includes the address of the first pod, the address of the second pod, and execution information of the computing task.

[0016] Optionally, the task computing container includes:

[0017] A map task computing container and a reduce task computing container, wherein the map task computing container and the reduce task computing container are deployed in two different second pods.

[0018] In a second aspect, the present application provides a YARN-based task processing device, which is applied to a server deployed with YARN technology, and the device includes:

[0019] A receiving unit, used for receiving data processing tasks from a big data processing platform;

[0020] A scheduling unit schedules the resources of the k8s cluster to utilize the task computing container deployed in the k8s cluster to execute the data processing task, wherein a main application container is also deployed in the k8s cluster, the main application container is deployed in a first pod in the k8s cluster, and the task computing container is deployed in a second pod in the k8s cluster. The main application container is used to send a task allocation message to the task computing container in the k8s cluster based on the data processing task, and the task allocation message is used to allocate computing tasks to the task computing container. The task computing container is used to execute the computing tasks allocated by the main application container to the task computing container, and the task allocation message includes the address of the first pod and the address of the second pod.

[0021] Optionally, the device further comprises:

[0022] An application unit, used for applying for resources in the k8s cluster for the data processing task based on a YARN node management component;

[0023] A deployment unit is used to deploy the first pod and the second pod in the k8s cluster by using the target resources applied for the data processing task by the YARN node management component.

[0024] Optionally, the resources managed by the server only include resources to which k8s is applied, and the resources to which k8s is applied include the k8s cluster. The YARN node management component is pre-deployed in a third pod included in the k8s cluster, and the third pod is different from the first pod and the second pod.

[0025] Optionally, the resources managed by the server also include a cluster to which an elastic computing service is applied, and the YARN node management component is deployed in the cluster to which the elastic computing service is applied.

[0026] Optional,

[0027] The task computing container is further used to send a status reporting message to the main application container, where the status reporting message includes the address of the first pod, the address of the second pod, and execution information of the computing task.

[0028] Optionally, the task computing container includes:

[0029] A map task computing container and a reduce task computing container, wherein the map task computing container and the reduce task computing container are deployed in two different second pods.

[0030] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising a processor and a memory;

[0031] The processor is configured to execute instructions stored in the memory, so that the device performs the method as described in any one of the first aspects above.

[0032] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, wherein the instructions instruct a device to execute a method as described in any one of the above first aspects.

[0033] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect above.

[0034] Compared with the prior art, the embodiments of the present application have the following advantages:

[0035] The present application provides a task processing method based on YARN, which can be applied to a server deployed with YARN technology. The method includes: receiving a data processing task from a big data processing platform, and further scheduling the resources of the k8s cluster to perform the data processing task using the task computing container deployed in the k8s cluster. In the present application, a main application container is also deployed in the k8s cluster, the main application container is deployed in the first pod in the k8s cluster, and the task computing container is deployed in the second pod in the k8s cluster. The main application container and the task computing container can communicate using the address of the first pod and the address of the second pod. Specifically, the main application container can send a task assignment message to the task computing container in the k8s cluster based on the data processing task, and the task assignment message includes the address of the first pod and the address of the second pod. The task assignment message is used to assign computing tasks to the task computing container. Correspondingly, the task computing container is used to execute the computing tasks assigned by the main application container to the task computing container, thereby realizing the execution of the aforementioned data processing task. It can be seen that by using the solution provided in this application, when using YARN to schedule the resources of the k8s cluster, the resources can be used to execute data processing tasks from the big data processing platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1aA schematic diagram of managing k8s resources based on YARM is shown;

[0038] Figure 1b A schematic diagram of managing ECS ​​resources and k8s resources based on YARN is shown;

[0039] Figure 2 A flowchart of a YARN-based task processing method provided in an embodiment of the present application;

[0040] Figure 3a A schematic diagram of a YARN-based task processing method provided in an embodiment of the present application;

[0041] Figure 3b A schematic diagram of another YARN-based task processing method provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the structure of a YARN-based task processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] For the convenience of description, the following terms related to YARN are explained first.

[0045] YARN resource manager (RM) component: It is used to process requests from clients (such as clients corresponding to the big data processing platform), coordinate the resources of the entire cluster, manage NM components, etc. The YARNRM component can be referred to as the RM component.

[0046] YARN NM component: It is used to manage the resources of the nodes to which it belongs and process the instructions from the RM component and the AM component. The YARN NM component can be referred to as the NM component for short.

[0047] Application master (AM) component: It is responsible for allocating tasks and monitoring the execution of tasks.

[0048] Container: It is an abstraction of resources managed by YARN. It can encapsulate multi-dimensional resources, for example, it can encapsulate multi-dimensional resources such as memory, central processing unit (CPU), disk, and network.

[0049] The inventor of this application has found through research that, currently, most of the resources managed by YARN are Elastic Compute Service (ECS) resources. In other words, YARN can manage the resources of clusters that use ECS, but it is difficult for YARN to manage the resources of clusters that use k8s. For the convenience of description, in the following description, clusters that use k8s are referred to as "k8s clusters", and clusters that use ECS are referred to as "ECS clusters".

[0050] In an example, if you want to implement YARN management of k8s cluster resources, you can deploy YARN-related components such as NM into pods and keep them resident. Among them: pod is the smallest deployable computing unit that can be created and managed in k8s. Pod consists of a set of closely related containers that run on the same node and share the same computing resources such as network namespace and storage volume, maximizing the advantages of resource sharing through cluster management.

[0051] For reference Figure 1a To understand, Figure 1a A schematic diagram of managing k8s resources based on YARM is shown.

[0052] like Figure 1a As shown, the k8s cluster includes at least one pod. Figure 1b Pod 1 and pod 2 are shown. RM components are deployed in pod 1, and NM components are deployed in pod 2. When a data processing task is received from a client, a task executor for processing the task is deployed in pod 2, and the task executor processes the data processing task.

[0053] In addition, the k8s cluster is also connected to the management and control system deployed outside the k8s cluster. The management and control system controls the k8s cluster to run elastic expansion / contraction instances, thereby expanding or shrinking the resources of the k8s cluster.

[0054] See also Figure 1b To understand, Figure 1b A schematic diagram of managing ECS ​​resources and k8s resources based on YARN is shown.

[0055] like Figure 1b As shown, the ECS cluster includes several nodes ( Figure 1b 3 nodes are shown), one of which is deployed with an RM component and a corresponding agent, wherein the agent corresponding to the RM component is used to pull up the RM component. Accordingly, the RM component manages multiple NM components, and one NM component corresponds to one node. Accordingly, each NM component also corresponds to an agent, and the agent corresponding to the NM component is used to pull up the NM component.

[0056] The k8s cluster includes at least one pod. Figure 1b Pod1 and pod2 are shown. NM components are deployed in both pod1 and pod2.

[0057] The data processing task submitted by the client is received by the RM component. In one example, the RM component can assign the data processing task to the NM component on a node (eg, node 2) in the ECS cluster, and the NM component on the node processes the data processing task.

[0058] In another example, the RM component can assign the data processing task to the k8s cluster for processing. The RM component can interact with the management and control system to assign the data processing task to the k8s cluster for processing. For example, if the data processing task is assigned to pod1 and / or pod2 for processing, a task executor for processing the task can be deployed in pod1 and / or pod2, and the task executor processes the data processing task.

[0059] However, the above method has the problem of low resource utilization of the k8s cluster. Because in this way, the NM component and the task executor that processes the data processing tasks submitted by the client are deployed in the same pod. Accordingly, when the resources of the k8s cluster are expanded or reduced, they are processed according to the NM granularity, and the NM resides in the pod, which makes it impossible to flexibly release the resources occupied by the NM, resulting in low resource utilization of the k8s cluster. Moreover, in this way, it is also necessary to deploy an external management and control system to expand or reduce the resources of the k8s cluster, and the solution deployment is also relatively complicated.

[0060] In order to solve the above problems, the inventors of the present application discovered that the task executor and the NM component no longer need to be bound to one pod. The task executor can be deployed in one or more pods independent of the pod where the NM component is located, so that the resources occupied by the task executor can be released according to the life cycle of the task executor, which is beneficial to improving the resource utilization efficiency of the k8s cluster.

[0061] Next, the YARN-based resource processing method provided by this application is introduced.

[0062] Various non-limiting implementations of the present application are described in detail below in conjunction with the accompanying drawings.

[0063] Exemplary Methods

[0064] See also Figure 2 , which is a flow chart of a YARN-based task processing method provided in an embodiment of the present application. Figure 2 The method shown can be applied to a server deployed with YARN technology. In this embodiment, the method can include the following steps: S101-S102.

[0065] S101: receiving a data processing task from a big data processing platform.

[0066] In the present application, receiving a data processing task from a big data processing platform may be receiving a data processing task from a client corresponding to the big data processing platform. The present application embodiment does not specifically limit the data processing task, and the data processing task may be related to a business scenario corresponding to the big data processing platform, which is not limited here.

[0067] The embodiment of the present application does not specifically limit the big data processing platform, and the big data processing platform may be a currently existing, under development, or future big data processing platform. As an example, the big data processing platform may be hive or spark.

[0068] In one example, the data processing task may be received by a YARN RM component.

[0069] In an example, if the server manages resources of both the ECS cluster and the k8s cluster, the YARN RM component may be deployed on a node of the ECS cluster, for example, the YARN RM component is deployed on the first node of the ECS cluster.

[0070] In another example, if the resources managed by the server only include a k8s cluster, the YARN RM component can be deployed in a pod (eg, a third pod) of the k8s cluster.

[0071] S102: Scheduling the resources of the k8s cluster to utilize the task computing container deployed in the k8s cluster to execute the data processing task, wherein a main application container is also deployed in the k8s cluster, the main application container is deployed in a first pod in the k8s cluster, and the task computing container is deployed in a second pod in the k8s cluster. The main application container is used to send a task allocation message to the task computing container in the k8s cluster based on the data processing task, and the task allocation message is used to allocate computing tasks to the task computing container. The task computing container is used to execute the computing tasks allocated by the main application container to the task computing container, and the task allocation message includes the address of the first pod and the address of the second pod.

[0072] After receiving the data processing task, the resources of the k8s cluster can be scheduled to apply for processing resources for the data processing task. In one example, the YARN NM component can be used to apply for processing resources in the k8s cluster for the data processing task.

[0073] In one example, if the resources managed by the server only include a k8s cluster, the YARN NM component is deployed in a fourth pod included in the k8s cluster. As a specific example, the YARN NM component may be deployed in the fourth pod when the k8s cluster starts running. The fourth pod may be deployed with a YARN NM component and an agent corresponding to the YARN NM component, and the agent corresponding to the YARN NM component is used to pull up the YARN NM component.

[0074] In another example, if the resources managed by the server include both k8s clusters and ECS clusters, then in one example, after the aforementioned YARN RM component receives the data processing task, it can preferentially use the resources of the ECS cluster to process the data processing task. When the resources of the ECS cluster are insufficient, the resources of the k8s cluster are used to process the data processing task. In this scenario, the aforementioned YARN NM component can be deployed in the ECS cluster. As an example, the YARN NM component can be deployed on the second node in the ECS cluster.

[0075] In a scenario where the YARN NM component is deployed on an ECS cluster, the YARN RM component deployed in the ECS cluster may deploy the YARN NM component in the ECS cluster when the available resources of the ECS cluster are insufficient. The insufficient available resources of the ECS cluster may be that the ECS cluster does not have enough resources to process the data processing task.

[0076] As a specific example, after receiving the data processing task, the YARN RM component can determine whether the ECS cluster has sufficient resources to process the data processing task based on the resource usage of the ECS cluster. If the ECS cluster has sufficient resources to process the data processing task, the resources of the ECS cluster can be directly used to process the data processing task. If the ECS cluster does not have sufficient resources to process the data processing task, the YARN RM component can create a second node in the ECS cluster and deploy the YARN RM component in the second node. In one example, the YARN RM component can be different from other ordinary YARN RM components deployed in the second cluster. The YARN RM component can be a YARN RM component with a plug-in, which is used to schedule and manage the resources of the k8s cluster.

[0077] As another specific example, the second node may also be created when the ECS cluster starts running. When the ECS cluster does not have sufficient resources to process the data processing task, the YARN RM component uses the YARN RM component to apply for processing resources in the k8s cluster for the data processing task. When the ECS cluster has sufficient resources to process the data processing task, the YARN RM component directly uses the resources of the ECS cluster to process the data processing task, instead of using the YARN RM component to apply for processing resources in the k8s cluster for the data processing task.

[0078] In one example, the k8s cluster includes a resource scheduler, and the YARN RM component can apply for resources from the resource scheduler for the data processing task. As a specific example, the k8s cluster also provides an application programming interface (API) service for applying for resources from the resource scheduler, and the YARN RM component can apply for resources from the resource scheduler for the data processing task through the API service.

[0079] After applying for target resources for the data processing task, at least one pod can be deployed in the k8s cluster based on the target resources, and a task executor is deployed in the at least one pod, and the task executor is used to execute the data processing task.

[0080] In a specific example, the task executor is deployed in the main application container and the task computing container. In other words, the main application container and the task computing container can be deployed in the k8s cluster. The main application container and the task computing container can be deployed in different pods. In the present application, the main application container can be deployed in the first pod, and the task computing container can be deployed in the second pod. Among them, in the present application, one or more second pods can be deployed in the k8s cluster, and each second pod can be deployed with one or more containers for executing corresponding computing tasks.

[0081] In one example, multiple second pods may be deployed in the k8s cluster, and each second pod may be deployed with a container for executing a certain type of computing task. As a specific example, assuming that the computing task includes a map task and a reduce task, the aforementioned task computing container may include a map task computing container and a reduce task computing container. Accordingly, two second pods may be deployed in the k8s cluster, one of which may be deployed with the map task computing container, and the other may be deployed with the reduce task computing container.

[0082] In the present application, the main application container is used to send a task assignment message to the task computing container in the k8s cluster based on the data processing task, and the task assignment message is used to assign computing tasks to the task computing container. Correspondingly, the task computing container is used to execute the computing tasks assigned by the main application container to the task computing container. In the present application, since the main application container and the task computing container are in different pods, therefore, in the present application, when the main application container and the task computing container communicate, they can use the address of the first pod where the main application container is located and the address of the second pod where the task computing container is located to communicate. In other words, the aforementioned task assignment message may include the address of the first pod and the address of the second pod, the address of the first pod as the address corresponding to the sender of the task assignment message, and the address of the second pod as the address corresponding to the receiver of the task assignment message.

[0083] In one example, the address of the first pod may be a communication address of the first pod, such as an Internet Protocol (IP) address of the first pod. Correspondingly, the address of the second pod may be a communication address of the second pod, such as an IP address of the second pod.

[0084] From the above description, it can be seen that using the solution provided by the present application, in the scenario where the main application container is deployed in the first pod and the task computing container is deployed in the second pod, the main application container and the task computing container can communicate based on the address of the first pod and the address of the second pod, so that the task computing container can perform the aforementioned data processing tasks, that is: using the solution provided by the present application, when using YARN to schedule the resources of the k8s cluster, the resources can be used to execute data processing tasks from the big data processing platform.

[0085] In one example, in order to facilitate the main application container to determine the task execution status of the task computing container, so that the main application container can better allocate computing tasks. The task computing container can send a status reporting message to the main container application. The status reporting message is used to report the task execution status of the task computing container. The task execution status mentioned here may include, for example: being executed, completed, etc. The task execution status may also include the execution progress of the task, which is not specifically limited in the embodiment of the present application. As mentioned above, when the main application container and the task computing container communicate, they can use the address of the first pod where the main application container is located and the address of the second pod where the task computing container is located to communicate. Therefore, the aforementioned status reporting message may include the address of the first pod and the address of the second pod, the address of the first pod as the address corresponding to the receiver of the status reporting message, and the address of the second pod as the address corresponding to the sender of the status reporting message.

[0086] The above introduces the YARN-based task processing method provided by the present application. Next, the application of the present application in two scenarios is introduced in conjunction with the accompanying drawings.

[0087] See also Figure 3a , which is a scenario diagram of a YARN-based task processing method provided in an embodiment of the present application.

[0088] like Figure 3aAs shown, pod1 and pod2 are deployed in the k8s cluster, RM components are deployed in pod1, and NM components are deployed in pod2. The k8s cluster also provides an API service and a resource scheduler. When the RM component receives a data processing task from the big data processing platform, it can assign the data processing task to the NM component deployed in pod2. The NM component applies for resources for the data processing task from the resource scheduler through the API service. And using the applied target resources, pod3 (corresponding to the aforementioned first pod) and pod4 (corresponding to the aforementioned second pod) are deployed in the k8s cluster, and the main application container is deployed in pod3, and the task computing container is deployed in pod4. The main application container and the task computing container communicate with each other through the address of pod3 and the address of pod4.

[0089] See also Figure 3b , this figure is a scenario schematic diagram of another YARN-based task processing method provided in an embodiment of the present application.

[0090] like Figure 3b As shown, the ECS cluster includes several nodes ( Figure 3b 3 nodes are shown), where RM components and corresponding agents are deployed on node 1. Correspondingly, NM components and corresponding agents are deployed on the other two nodes. Among them, the NM component deployed on node 2 is a common component used to schedule ECS resources. The NM component deployed on node 3 is a component with a plug-in used to schedule k8s resources.

[0091] The k8s cluster provides API service and resource scheduler.

[0092] The data processing task submitted by the client is received by the RM component. When the RM component determines that the resources of the ECS cluster are insufficient, it deploys the NM component with a plug-in on node 3. The NM component with a plug-in applies for resources for the data processing task from the resource scheduler through the API service. And using the applied target resources, pod5 (corresponding to the aforementioned first pod) and pod6 (corresponding to the aforementioned second pod) are deployed in the k8s cluster, and the main application container is deployed in pod5, and the task computing container is deployed in pod6. The main application container and the task computing container communicate with each other through the address of pod5 and the address of pod6.

[0093] Exemplary Devices

[0094] Based on the method provided in the above embodiment, the embodiment of the present application further provides a device, which is described below in conjunction with the accompanying drawings.

[0095] See also Figure 4, Figure 4 A schematic diagram of the structure of a YARN-based task processing device provided in an embodiment of the present application. Figure 4 The device 400 shown can be applied to a server deployed with YARN technology to execute the YARN-based task processing method provided by the above method embodiment. The device 400 can specifically include: a receiving unit 401 and a scheduling unit 402.

[0096] A receiving unit 401 is used to receive a data processing task from a big data processing platform;

[0097] The scheduling unit 402 schedules the resources of the k8s cluster to utilize the task computing container deployed in the k8s cluster to execute the data processing task, wherein a main application container is also deployed in the k8s cluster, the main application container is deployed in a first pod in the k8s cluster, and the task computing container is deployed in a second pod in the k8s cluster. The main application container is used to send a task allocation message to the task computing container in the k8s cluster based on the data processing task, and the task allocation message is used to allocate computing tasks to the task computing container. The task computing container is used to execute the computing tasks allocated by the main application container to the task computing container, and the task allocation message includes the address of the first pod and the address of the second pod.

[0098] Optionally, the device further comprises:

[0099] An application unit, used for applying for resources in the k8s cluster for the data processing task based on a YARN node management component;

[0100] A deployment unit is used to deploy the first pod and the second pod in the k8s cluster by using the target resources applied for the data processing task by the YARN node management component.

[0101] Optionally, the resources managed by the server only include resources to which k8s is applied, and the resources to which k8s is applied include the k8s cluster. The YARN node management component is pre-deployed in a third pod included in the k8s cluster, and the third pod is different from the first pod and the second pod.

[0102] Optionally, the resources managed by the server also include a cluster to which an elastic computing service is applied, and the YARN node management component is deployed in the cluster to which the elastic computing service is applied.

[0103] Optional,

[0104] The task computing container is further used to send a status reporting message to the main application container, where the status reporting message includes the address of the first pod, the address of the second pod, and execution information of the computing task.

[0105] Optionally, the task computing container includes:

[0106] A map task computing container and a reduce task computing container, wherein the map task computing container and the reduce task computing container are deployed in two different second pods.

[0107] Since the device 400 is a device corresponding to the YARN-based task processing method provided by the above method embodiment, the specific implementation of each unit of the device 400 is the same concept as the above method embodiment. Therefore, for the specific implementation of each unit of the device 400, reference can be made to the relevant description part of the above method embodiment, which will not be repeated here.

[0108] The embodiment of the present application also provides an electronic device, the electronic device comprising a processor and a memory;

[0109] The processor is used to execute the instructions stored in the memory, so that the device executes the YARN-based task processing method provided by the above method embodiment.

[0110] An embodiment of the present application provides a computer-readable storage medium, including instructions, wherein the instructions instruct a device to execute the YARN-based task processing method provided in the above method embodiment.

[0111] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the YARN-based task processing method provided by the above method embodiment.

[0112] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0113] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0114] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A task processing method based on YARN, characterized in that: Applied to a server on which YARN technology is deployed, the method includes: Receive data processing tasks from the big data processing platform; The resources of the k8s cluster are scheduled to utilize the task computing container deployed in the k8s cluster to execute the data processing task, wherein a main application container is also deployed in the k8s cluster, the main application container is deployed in a first pod in the k8s cluster, and the task computing container is deployed in a second pod in the k8s cluster. The main application container is used to send a task allocation message to the task computing container in the k8s cluster based on the data processing task, and the task allocation message is used to allocate computing tasks to the task computing container. The task computing container is used to execute the computing tasks allocated by the main application container to the task computing container, and the task allocation message includes the address of the first pod and the address of the second pod.

2. The method according to claim 1, characterized in that The method further comprises: Apply for resources in the k8s cluster for the data processing task based on the YARN node management component; The first pod and the second pod are deployed in the k8s cluster using the target resources applied for the data processing task by the YARN node management component.

3. The method according to claim 2, characterized in that The resources managed by the server only include resources to which k8s is applied, and the resources to which k8s is applied include the k8s cluster. The YARN node management component is pre-deployed in a third pod included in the k8s cluster, and the third pod is different from the first pod and the second pod.

4. The method according to claim 2, wherein the characteristic resource, The resources managed by the server also include a cluster to which the elastic computing service is applied, and the YARN node management component is deployed in the cluster to which the elastic computing service is applied.

5. The method according to claim 1, characterized in that The task computing container is further used to send a status reporting message to the main application container, where the status reporting message includes the address of the first pod, the address of the second pod, and execution information of the computing task.

6. The method according to claim 1, characterized in that The task computing container includes: A map task computing container and a reduce task computing container, wherein the map task computing container and the reduce task computing container are deployed in two different second pods.

7. A task processing device based on YARN, characterized in that: Applied to a server deployed with YARN technology, the device includes: A receiving unit, used for receiving data processing tasks from a big data processing platform; A scheduling unit schedules the resources of the k8s cluster to utilize the task computing container deployed in the k8s cluster to execute the data processing task, wherein a main application container is also deployed in the k8s cluster, the main application container is deployed in a first pod in the k8s cluster, and the task computing container is deployed in a second pod in the k8s cluster. The main application container is used to send a task allocation message to the task computing container in the k8s cluster based on the data processing task, and the task allocation message is used to allocate computing tasks to the task computing container. The task computing container is used to execute the computing tasks allocated by the main application container to the task computing container, and the task allocation message includes the address of the first pod and the address of the second pod.

8. The device according to claim 7, characterized in that The device also includes: An application unit, used for applying for resources in the k8s cluster for the data processing task based on a YARN node management component; A deployment unit is used to deploy the first pod and the second pod in the k8s cluster by using the target resources applied for the data processing task by the YARN node management component.

9. An electronic device, characterized in that: The device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory so that the device executes the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The method comprises instructions, wherein the instructions instruct the device to execute the method according to any one of claims 1 to 6.

Images