Task state monitoring method and device, electronic equipment and storage medium

Abstract

The invention relates to a task state monitoring method and device, electronic equipment and a storage medium, and the method comprises the steps: creating a custom resource CRD on a target application program, the CRD being used for monitoring the execution state of a task with a target label identifier on the target application program; monitoring the execution state of the task with the target label identifier on the target application program based on the CRD to obtain a monitoring result; wherein the monitoring result comprises the task state and the task information; and updating the monitoring result to a task list. According to the task state monitoring method and device, real-time monitoring of the task can be carried out through CRD, so that the real-time performance of task state monitoring is guaranteed, and the problem that the real-time performance of the task state cannot be guaranteed due to the fact that the state information of the current running task is screened out at set intervals in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a method and device for monitoring task status, an electronic device, and a storage medium. Background Art

[0002] There is currently no unified job list for cloud-native Spark jobs to display job status, and it is impossible to view and manage all running jobs in a centralized location. At the same time, due to container network restrictions, it is impossible to directly access and monitor the status of jobs through the Web UI (Website User Interface).

[0003] In the prior art, by setting a timed task, the status information of currently running Spark jobs is filtered out in Kubernetes (k8s) every once in a while and summarized into a job list; in addition, by mapping container ports, it is possible to access the Web UI outside the container. However, in the process of obtaining Spark job status information regularly in the prior art, the timed task consumes additional resources to filter out the Spark job list, and the real-time nature of the Spark job list and status cannot be guaranteed. In addition, multiple ports need to be mapped to access the Web UI, increasing the complexity of accessing the Web UI. Summary of the Invention

[0004] This application provides a method and device for monitoring task status, an electronic device, and a storage medium to solve the problem in the prior art that the status information of currently running tasks is filtered out every once in a while, resulting in the inability to guarantee the real-time nature of the task status.

[0005] In a first aspect, this application provides a method for monitoring task status, including: creating a Custom Resource Definition (CRD) on a target application, where the CRD is used to monitor the execution status of tasks with a target label identifier on the target application; monitoring the execution status of tasks with the target label identifier on the target application based on the CRD to obtain a monitoring result, where the monitoring result includes the task status and the task information; and updating the monitoring result to a task list.

[0006] In a second aspect, the present application provides a monitoring device for task status, including: a first creation module configured to create a Custom Resource Definition (CRD) on a target application, where the CRD is used to monitor the execution status of a task with a target label identifier on the target application; a monitoring module configured to monitor the execution status of the task with the target label identifier on the target application based on the CRD to obtain a monitoring result, where the monitoring result includes the task status and the task information; and an update module configured to update the monitoring result to a task list.

[0007] In a third aspect, the present application provides an electronic device, including: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, where the processor is configured to execute the monitoring method for task status according to the first aspect of the present application above.

[0008] In a fourth aspect, the present application further provides a computer storage medium storing computer-executable instructions for executing the monitoring method for task status according to the first aspect of the present application above.

[0009] The above technical solutions provided by the embodiments of the present application have the following advantages compared with the prior art: In the embodiments of the present application, by customizing the CRD in the target application, the task with the label identifier can be monitored through the CRD, and the monitoring result is updated to the task list. That is to say, in the embodiments of the present application, the task can be monitored in real time through the CRD, thereby ensuring the real-time nature of the task status monitoring and solving the problem in the prior art that the status information of the currently running tasks is screened out every once in a while, resulting in the inability to ensure the real-time nature of the task status. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings here are incorporated into the specification and form a part of the specification, showing embodiments consistent with the present invention and used together with the specification to explain the principles of the present invention.

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0012] One or more embodiments are exemplarily illustrated by the pictures in the corresponding drawings. These exemplary illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings represent similar elements. Unless otherwise stated, the drawings in the figures do not constitute a scale limitation.

[0013] Figure 1 This is a flowchart of a method for monitoring the task status provided by an embodiment of the present application;

[0014] Figure 2 This is an alternative flowchart of a method for monitoring the task status provided by an embodiment of the present application;

[0015] Figure 3 This is a flowchart of a method for automatically monitoring cloud-native Spark jobs and automatically proxying the Web UI based on CRD provided by an embodiment of the present application;

[0016] Figure 4 This is a schematic structural diagram of a device for monitoring the task status provided by an embodiment of the present application;

[0017] Figure 5 This is a flowchart of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0019] The following disclosure provides many different embodiments or examples for implementing different structures of the present invention. To simplify the disclosure of the present invention, the components and settings of specific examples are described below. Of course, they are only examples and are not intended to limit the present invention. In addition, the present invention may repeat reference numerals and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0020] First, the relevant terminologies in the embodiments of the present application are explained;

[0021] Kubernetes, abbreviated as K8s, is an abbreviation formed by replacing the eight characters "ubernete" in the middle of the name with 8. It is an open-source tool for managing containerized applications on multiple hosts in a cloud platform. The goal of Kubernetes is to make it simple and efficient to deploy containerized applications. Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance.

[0022] Spark is a fast and general-purpose cluster computing system. It provides API programming interfaces in four programming languages: Java, Scala, Python, and R, as well as an optimization engine based on DAG graph execution.

[0023] Figure 1 The following is a flowchart of a method for monitoring the task status provided by an embodiment of the present application. As Figure 1 shown, the steps of the method include:

[0024] Step 101: Create a Custom Resource Define (CRD) on the target application, where the CRD is used to monitor the execution status of tasks with a target label identifier on the target application;

[0025] In a specific example of the present application, the target application involved in the present application may refer to K8s, that is, a CRD can be created on K8s, and the CRD can monitor tasks with target labels.

[0026] It should be noted that CRD (a built-in resource type in Kubernetes, that is, the definition of custom resources, used to describe what user-defined resources look like. Explanation of CRD:

[0027] 1) CRD is a mechanism added in v1.7+ to extend the Kubernetes API without changing the code, used to manage custom objects.

[0028] 2) From the perspective of Kubernetes users, everything is called a resource (Resource), which is the content of the field Kind in Yaml (markup language), such as Service, Deployment, etc.

[0029] 3) In addition to common built-in resources, Kubernetes allows users to customize resources Custom Resource, and CRD represents the definition of custom resources.

[0030] 4) When you create a new CRD, the Kubernetes API server will generate a new RESTful resource path for each version you specify.

[0031] 5) The custom resources created based on the CRD object can be namespace-scoped or cluster-scoped, depending on the setting of the spec.scope field of the CRD object.

[0032] 6) Defining the operation of the CRD object will create a new custom resource using the name and schema definition you set, and the Kubernetes API is responsible for providing storage and access services for the custom resource.

[0033] Step 102, monitor the execution status of the tasks with the target label identifier on the target application based on the CRD to obtain the monitoring result; wherein, the monitoring result includes the task status and task information.

[0034] Step 103, update the monitoring result to the task list.

[0035] It can be seen from the above steps 101 to 103 that in the embodiment of the present application, by customizing the CRD in the target application, the tasks with label identifiers can be monitored through the CRD, and the monitoring result can be updated to the task list. That is to say, in the embodiment of the present application, the tasks can be monitored in real time through the CRD, thus ensuring the real-time monitoring of the task status and solving the problem in the prior art that the status information of the currently running tasks is filtered out every once in a while, resulting in the inability to ensure the real-time nature of the task status.

[0036] In an alternative implementation manner of the embodiment of the present application, for the manner of monitoring the execution status of the tasks with the target label identifier on the target application based on the CRD in step 102 above to obtain the monitoring result, it can further include:

[0037] Step 11, when the Spark job task is submitted to the target application, determine whether the Spark job task has the target label identifier.

[0038] Step 12, when the Spark job task has the target label identifier, monitor the creation operation of the Spark job task based on the CRD, and obtain the monitoring result based on the creation result of the Spark job task and the task information of the Spark job task.

[0039] In a specific example of the embodiment of the present application, the CRD can be for all Spark job tasks. However, after a Spark job task with a target label identifier is submitted on k8s, the custom CRD resource will monitor the creation operation, and based on the task information of the Spark job task and the creation information, obtain a monitoring result. Finally, the monitoring result needs to be updated to the task list so that the task information and its status of each Spark job task can be obtained in real time through the task list.

[0040] In an alternative implementation manner of the embodiment of the present application, for the manner of monitoring the execution status of a task with a target label identifier on a target application based on the CRD in step 102 above to obtain a monitoring result, it can further include:

[0041] Step 21, when modifying the task information of a Spark job task in the task list, monitor the modification operation of the Spark job task based on the CRD, and obtain a monitoring result based on the modification result.

[0042] In a specific example, after a new Spark job task is submitted on k8s, the task information monitored by the CRD can include the number of PODs (a Pod is the smallest unit that can be created and managed in the k8s system, the smallest resource object model created or deployed by a user in the resource object model, and also the resource object for running containerized applications on k8s) of the current job task, status information, spark UI, etc. Therefore, in the subsequent process, the above task information can be modified, such as modifying the number of PODs, modifying the status information, etc. However, each time a modification is made, the CRD can monitor it and update the monitoring result to the task list, that is, through the method of the embodiment of the present application, the dynamics of the job task can be monitored in real time and updated to the task list for viewing.

[0043] In an alternative implementation manner of the embodiment of the present application, for the manner of monitoring the execution status of a task with a target label identifier on a target application based on the CRD in step 102 above to obtain a monitoring result, it can further include:

[0044] Step 31, when deleting the task information of a Spark job task in the task list, monitor the deletion operation of the Spark job task based on the CRD, and obtain a monitoring result based on the deletion result.

[0045] In a specific example, when a deletion operation needs to be performed, the current task needs to be deleted, that is, all information related to the current task in the task list is deleted, such as the number of PODs, status, Spark UI, etc. of the current task, and the current task dynamics are updated in real time to the task list, that is, the current task in the task list has been deleted.

[0046] In an alternative implementation manner of the embodiment of the present application, as Figure 2 shown, the method of the embodiment of the present application may further include:

[0047] Step 104, create a network proxy when monitoring the execution status of a task with a target label identifier on a target application based on a CRD; wherein, the network proxy is used to access the UI interface from the outside;

[0048] Step 105, access the UI interface based on the network proxy, wherein the UI interface is used to display the execution status of the task.

[0049] In a specific example, when the target application is K8s, the created network proxy can be used to access the UI interface (Web UI) from outside the K8s container without accessing the Web UI by mapping the container port, that is, the Web UI can be directly accessed from outside the container through the network proxy. Through the visual interface provided by the Web UI, users can better analyze the execution process of the spark job. It can be seen that the way of accessing the Web UI is optimized through the present application.

[0050] Further, in an alternative implementation manner of the embodiment of the present application, for the method of creating a network proxy involved in the above step 104, it may further include: creating an ingress resource; wherein, the ingress resource includes an ingress controller, and the ingress controller is used to access the UI interface based on a user request;

[0051] For the method of accessing the UI interface based on the network proxy involved in the above step 105, it may further include: after receiving a user request, access the UI interface based on the ingress controller.

[0052] It should be noted that Ingress is equivalent to a 7-layer load balancer, which can be understood as an abstraction of reverse proxy by K8s, and its working principle is similar to that of Nginx. Specifically, many mapping rules are established in Ingress, and the Ingress controller listens to these configuration rules and converts them into Nginx reverse proxy configurations, and then provides services to the outside. It can be seen that the Ingress controller is a program used to specifically implement reverse proxy and load balancing, parses the rules defined by Ingress, and realizes request forwarding according to the configured rules. The implementation methods include Nginx, Contour, Haproxy, etc.

[0053] In an alternative implementation manner of the embodiment of the present application, the method of the embodiment of the present application may further include:

[0054] Step 106, when a deletion operation on the Spark job task is detected based on the CRD, delete the network proxy.

[0055] It can be seen that in the embodiment of the present application, when deleting the current task, its corresponding network proxy also needs to be deleted accordingly. Because after the current task is deleted, it is not necessary to access the current task, so the corresponding network proxy needs to be deleted.

[0056] The following explains the present application in combination with the specific implementation manner of the embodiment of the present application. The specific implementation manner provides a method for automatically monitoring cloud-native Spark jobs and automatically proxying the Web UI based on CRD, as Figure 3 shown. The steps of the method include:

[0057] Step 301, customize the CRD resource and create a new Spark job task. Among them, the CRD resource is used to detect operations such as the creation, modification, and deletion of tasks;

[0058] Among them, the CRD resource is used to listen to the spark jobs with label identifiers. After the spark job is submitted on k8s, the customized CRD resource will detect the creation operation, create a network proxy at the same time, and add the current task information to the task list.

[0059] Step 302, determine whether the task needs to be listened to according to whether the Spark job task has a label identifier;

[0060] Step 303, when the Spark job task has a label identifier, record the current status to the task list and create a corresponding network proxy;

[0061] Step 304, modify the current status of the task in the task list;

[0062] Step 305: Delete the current task in the task list and delete the corresponding network proxy.

[0063] Step 306: Aggregate all monitored task lists and the Web UI.

[0064] Through the above steps 301 to 306, by creating custom CRD resources in k8s, according to the label identification of the Spark job, the statuses such as creation, modification, and deletion of the job can be monitored and updated to the task list, enabling real-time access to the status and list of each job. At the same time, the corresponding network proxy is created to achieve external access to the container. By means of the embodiments of the present application, no additional resources need to be consumed, and there is no need to worry about port mapping issues. As long as the CRD resources are deployed to k8s, the statuses such as creation, modification, and deletion of the job can be monitored, and the corresponding network proxy is created at the same time to achieve external access to the container without mapping multiple ports to access the Web UI.

[0065] Corresponding to the above Figure 1 , the embodiments of the present application also provide a monitoring device for task status, as Figure 4 shown, the device includes:

[0066] The first creation module 402 is used to create a custom resource CRD on the target application, where the CRD is used to monitor the execution status of tasks with a target label identification on the target application;

[0067] The monitoring module 404 is used to monitor the execution status of tasks with a target label identification on the target application based on the CRD to obtain a monitoring result; where the monitoring result includes task status and task information;

[0068] The update module 406 is used to update the monitoring result to the task list.

[0069] In the embodiments of the present application, by customizing the CRD in the target application, the tasks with label identification can be monitored through the CRD, and the monitoring result is updated to the task list. That is to say, in the embodiments of the present application, real-time monitoring of tasks can be performed through the CRD, thus ensuring the real-time nature of the task status monitoring and solving the problem in the prior art that the status information of the currently running tasks is screened out every once in a while, resulting in the inability to ensure the real-time nature of the task status.

[0070] In an alternative implementation manner of the embodiment of the present application, the monitoring module 404 in the embodiment of the present application may further include: a determination unit, configured to determine whether a target label identifier exists for a Spark job task when the Spark job task is submitted to a target application; a first processing unit, configured to, when the Spark job task has a target label identifier, monitor a creation operation on the Spark job task based on the CRD, and obtain a monitoring result based on the creation result of the Spark job task and the task information of the Spark job task.

[0071] In a specific example of the embodiment of the present application, the CRD may be for all Spark job tasks. However, after a Spark job task with a target label identifier is submitted on k8s, the custom CRD resource will monitor the creation operation, and obtain a monitoring result based on the task information and creation information of the Spark job task. Finally, the monitoring result needs to be updated to the task list so that the task information and its status of each Spark job task can be obtained in real time through the task list.

[0072] In an alternative implementation manner of the embodiment of the present application, the monitoring module 404 in the embodiment of the present application may further include: a second processing unit, configured to, when the task information of a Spark job task in the task list is modified, monitor a modification operation on the Spark job task based on the CRD, and obtain a monitoring result based on the modification result.

[0073] In a specific example, after a new Spark job task is submitted on k8s, the task information monitored by the CRD may include the number of PODs of the current job task (a Pod is the smallest unit that can be created and managed in the k8s system, the smallest resource object model created or deployed by a user in the resource object model, and also the resource object for running containerized applications on k8s), status information, spark UI, etc. Therefore, the above task information can be modified in the subsequent process, such as modifying the number of PODs, modifying the status information, etc. However, each modification can be monitored by the CRD, and the monitoring result is updated to the task list, that is, the dynamic of the job task can be monitored in real time and updated to the task list through the method of the embodiment of the present application for viewing.

[0074] In an alternative implementation manner of the embodiment of the present application, the monitoring module 404 in the embodiment of the present application may further include: a third processing unit, configured to, when the task information of a Spark job task in the task list is deleted, monitor a deletion operation on the Spark job task based on the CRD, and obtain a monitoring result based on the deletion result.

[0075] In a specific example, when a deletion operation needs to be performed, the current task needs to be deleted, that is, all information related to the current task in the task list is deleted, such as the number of PODs, status, Spark UI, etc. of the current task, and the current task dynamics are updated in real time to the task list, that is, the current task in the task list has been deleted.

[0076] In an alternative embodiment of the embodiment of the present application, the device of the embodiment of the present application may further include: a second creation module, configured to create a network proxy when monitoring the execution status of a task with a target label identifier on a target application based on a CRD; wherein, the network proxy is used to access the UI interface from the outside; an access module, configured to access the UI interface based on the network proxy, wherein the UI interface is used to display the execution status of the task.

[0077] In a specific example, when the target application is K8s, the created network proxy can be used to access the UI interface (Web UI) from outside the K8s container, without the need to access the Web UI by mapping the container port, that is, the Web UI can be directly accessed from outside the container through the network proxy. Through the visual interface provided by the Web UI, it can help users better analyze the execution process of the spark job. It can be seen that the way of accessing the Web UI is optimized through the present application.

[0078] In an alternative embodiment of the embodiment of the present application, the second creation module includes: a creation unit, configured to create an ingress resource; wherein, the ingress resource includes an ingress controller, and the ingress controller is used to access based on a user request; the access module includes: an access unit, configured to access the UI interface based on the ingress controller after receiving a user request.

[0079] In an alternative embodiment of the embodiment of the present application, the device of the embodiment of the present application may further include: a deletion module, configured to delete the network proxy when monitoring a deletion operation on a Spark job task based on a CRD.

[0080] It can be seen that in the embodiment of the present application, when the current task is deleted, its corresponding network proxy also needs to be deleted accordingly. Because after the current task is deleted, it is not necessary to access the current task, so the corresponding network proxy needs to be deleted.

[0081] As Figure 5 shown, the embodiment of the present application provides an electronic device, including a processor 511, a communication interface 512, a memory 513, and a communication bus 514. Among them, the processor 511, the communication interface 512, and the memory 513 communicate with each other through the communication bus 514.

[0082] A memory 513 for storing a computer program;

[0083] In an embodiment of the present application, when the processor 511 executes the program stored in the memory 513, it implements the method for monitoring the task state provided in any of the foregoing method embodiments. The role it plays is similar and will not be elaborated here.

[0084] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the method for monitoring the task state provided in any of the foregoing method embodiments.

[0085] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0086] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution or the part that contributes to the related technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0087] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" as used herein may also include the plural forms. The terms "include", "comprise", "contain", and "have" are inclusive and thus specify the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or their combinations. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be executed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that alternative or additional steps can be used.

[0088] The above are only specific embodiments of the present invention, enabling those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for monitoring task status, It is characterized in that include: Creating a custom resource CRD on the target application, wherein the CRD is used to monitor the execution status of tasks identified with a target tag on the target application; Based on the CRD, the execution status of the task with the target tag identification on the target application is monitored to obtain a monitoring result; wherein the monitoring result includes task status and task information; The monitoring results are updated in the task list.

2. The method according to claim 1, It is characterized in that Based on the CRD, the execution status of the task with the target tag on the target application is monitored, and the monitoring results obtained include: When a Spark job task is submitted to the target application, determining whether the Spark job task has the target tag identifier; In the case that the Spark job task has the target tag identifier, the creation operation of the Spark job task is monitored based on the CRD, and the monitoring result is obtained based on the creation result of the Spark job task and the task information of the Spark job task.

3. The method according to claim 2, It is characterized in that Based on the CRD, the execution status of the task with the target tag on the target application is monitored, and the monitoring results obtained include: In the case where the task information of the Spark job task in the task list is modified, the modification operation on the Spark job task is monitored based on the CRD, and the monitoring result is obtained based on the modification result.

4. The method according to claim 2, It is characterized in that Based on the CRD, the execution status of the task with the target tag on the target application is monitored, and the monitoring results obtained include: In the case where the task information of the Spark job task in the task list is deleted, the deletion operation of the Spark job task is monitored based on the CRD, and the monitoring result is obtained based on the deletion result.

5. The method according to claim 4, It is characterized in that The method further comprises: Creating a network proxy when monitoring the execution status of the task with the target tag on the target application based on the CRD; wherein the network proxy is used to access the UI interface from the outside; The UI interface is accessed based on the network agent, wherein the UI interface is used to display the execution status of the task.

6. The method according to claim 5, It is characterized in that Creating the network proxy includes: creating an ingress resource; wherein the ingress resource includes an ingress controller, and the ingress controller is used to access the UI interface based on a user request; Accessing the UI interface based on the network proxy includes: after receiving a user request, accessing the UI interface based on the ingress controller.

7. The method according to claim 5, It is characterized in that The method further comprises: When a deletion operation on the Spark job task is detected based on the CRD, the network proxy is deleted.

8. A task status monitoring device, It is characterized in that include: A first creation module is used to create a custom resource CRD on a target application, wherein the CRD is used to monitor the execution status of a task with a target tag on the target application; A monitoring module, configured to monitor the execution status of the task with the target tag identification on the target application based on the CRD to obtain a monitoring result; wherein the monitoring result includes the task status and task information; The updating module is used to update the monitoring result into the task list.

9. An electronic device, include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor coupled to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to execute the task status monitoring method described in any one of claims 1 to 7 of the present application.

10. A computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute the task status monitoring method according to any one of claims 1 to 7 of the present application.

Images