Methods, devices, servers, equipment, and media for monitoring distributed tasks
By working together with the monitoring server and the status monitoring client, the problem of not being able to access the field after a task has ended in Kubernetes distributed task monitoring has been solved. This enables status monitoring of distributed task processes and rapid fault location, improving the efficiency of fault diagnosis and resolution.
Patent Information
- Application Number
- CN202210137833.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-15
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-02-15
AI Technical Summary
Existing distributed task monitoring methods based on Kubernetes automatically exit the unit container environment when the task ends, making it impossible to enter the task context for troubleshooting when the task fails or encounters errors, and failing to accurately reflect the status of multiple internal processes.
By monitoring the collaborative work of the monitoring server and the status monitoring client, the status of distributed task processes is monitored, and the exit timing of the container environment is controlled according to the global task status and unit retention settings. Optional retention of the unit container environment is supported, enabling status monitoring and problem localization for each distributed task process.
It enables the preservation of the unit container environment when a distributed task fails or encounters an error, allowing operations and maintenance personnel to quickly access the task site to handle the problem and improve the efficiency of troubleshooting and resolution.
Smart Images

Figure CN116643833B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the fields of cloud computing, distributed computing and artificial intelligence technologies, and in particular to a method, apparatus, server, device and medium for monitoring distributed tasks. Background Technology
[0002] With the development of distributed computing and cloud computing technologies, in various complex and computationally intensive scenarios, such as business processing, operation and maintenance, and algorithm model development, global tasks can be divided into distributed tasks and processed collaboratively on various dispersed remote computing devices.
[0003] Kubernetes (K8S) is a promising system for automated application deployment, containerized management, and automatic scaling, with excellent prospects in distributed computing. Taking algorithm model development as an example, K8S can currently monitor the state of distributed training tasks. For instance, Kubeflow (an open-source machine learning library) can create and monitor the state of deep learning distributed tasks using tools like TensorFlow (Google's second-generation AI learning system) and PyTorch (an open-source Python machine learning library). It boasts advantages such as simple deployment, high portability, and good scalability, making it a commonly used framework for implementing distributed training tasks based on K8S.
[0004] However, the aforementioned Kubernetes-based status monitoring method has the following technical problems: It automatically exits the unit container environment when a task is detected as finished, resulting in the task execution context not being preserved. Therefore, when a task fails or an error occurs, it's impossible to access the task execution context for troubleshooting and debugging. Furthermore, the current framework uses the pod (smallest scheduling / deployment unit) status to determine the distributed task status. For scenarios where multiple distributed task processes run within a single unit, this logic cannot accurately reflect the status of each internal process. Summary of the Invention
[0005] To address or at least partially address the aforementioned technical problems, embodiments of this disclosure provide a method, apparatus, server, device, and medium for monitoring distributed tasks.
[0006] In a first aspect, embodiments of this disclosure provide a method for monitoring distributed tasks. The method is applied to nodes in a service cluster. For each node, the method includes: responding to a task processing request, starting a target distributed task process in a global task, the target distributed task process being located in the smallest scheduling unit of the current node; monitoring the distributed task status of the target distributed task process and reporting it to a monitoring server; sending a status acquisition request to the monitoring server to obtain the global task status based on preset time interval information; and controlling the target distributed task process and the unit container environment of the current node to exit based on the global task status fed back by the monitoring server and the unit retention status setting information for the current node.
[0007] According to embodiments of this disclosure, controlling the exit of the target distributed task process and the unit container environment of the current node based on the global task status fed back by the monitoring server and the unit retention status setting information for the current node includes: determining whether the global task is in an execution completion state (wherein the execution completion state includes a success state or a failure state) based on the global task status; if it is determined that the global task is in an execution completion state, determining the exit timing information of the target distributed task process and the unit container environment of the current node based on the unit retention status setting information for the current node; and controlling the exit of the target distributed task process and the unit container environment of the current node based on the exit timing information.
[0008] According to embodiments of this disclosure, determining the exit timing information of the target distributed task process and the unit container environment of the current node based on the aforementioned unit retention status setting information for the current node includes: when the unit retention status setting information is set to not retain, determining the exit timing information of the target distributed task process and the unit container environment of the current node is: when it is determined that the global task changes from a running state to an execution-completed state, exiting the target distributed task process and the unit container environment of the current node; when the unit retention status setting information is set to retain and a retention duration is set, determining the exit timing information of the target distributed task process and the unit container environment of the current node is: when it is determined that the global task changes from a running state to an execution-completed state, controlling the target distributed task process and the unit container environment of the current node to continue maintaining the aforementioned retention duration before exiting.
[0009] According to embodiments of this disclosure, when the minimum scheduling unit includes one or more target distributed task processes, monitoring the distributed task status of the target distributed task processes and reporting it to the monitoring server includes: monitoring the startup and running status of each target distributed task process in the minimum scheduling unit; and sending the distributed task status corresponding to each target distributed task process to the monitoring server based on the startup and running status.
[0010] According to embodiments of this disclosure, based on the aforementioned startup and running states, the distributed task status corresponding to each of the aforementioned target distributed task processes is sent to the monitoring server, including: when the startup status of the current target distributed task process is detected as successful startup, sending the distributed task status of the current target distributed task process to the task status queue of the monitoring server as: running state; when the running status of the current target distributed task process is detected as successful running, sending the distributed task status of the current target distributed task process to the task status queue of the monitoring server as: execution successful state; when the startup status of the current target distributed task process is detected as startup failure or running status as running failure, sending the distributed task status of the current target distributed task process to the task status queue of the monitoring server as: failure state, and carrying corresponding failure reason information.
[0011] According to embodiments of this disclosure, the method further includes: monitoring the unit status of the smallest scheduling unit in the current node, the unit status including the unit abnormal exit status; and sending the monitored unit status to the monitoring server in the form of an event.
[0012] Secondly, embodiments of this disclosure provide a method for monitoring distributed tasks. The method is applied to a monitoring server and includes: receiving distributed task status information about their respective target distributed task processes from various nodes in a service cluster that execute global tasks, wherein the target distributed task processes are located in the smallest scheduling unit of each node; updating and summarizing the task status of distributed task processes in a database based on the received distributed task status information of the target distributed task processes from the various nodes, obtaining a global task status and storing it; receiving a status acquisition request from one or more of the nodes for obtaining the global task status; and, in response to receiving the status acquisition request, storing the time of receiving the status acquisition request and sending the global task status information to the one or more nodes.
[0013] According to embodiments of this disclosure, the method further includes: determining the distributed task process whose task state is in the running state and corresponding to the waiting distributed task process; performing communication timeout detection on the waiting nodes corresponding to the waiting distributed task process based on the receiving time; if a communication timeout is detected in the target waiting node among the waiting nodes, setting the task state of the distributed task process corresponding to the target waiting node to a failed state, updating the corresponding global task state, and storing the target waiting node and timeout information as failure reason information.
[0014] According to an embodiment of this disclosure, the above-mentioned communication timeout detection for the waiting nodes corresponding to the waiting distributed task process based on the above-mentioned reception time includes: for each waiting node, calculating the difference between the current detection time and the latest reception time corresponding to the current waiting node to obtain the waiting time; determining whether the above-mentioned waiting time exceeds the waiting time threshold, the above-mentioned waiting time threshold being related to the preset time interval information of the current waiting node sending the status acquisition request; and determining that the current waiting node has a communication timeout if the above-mentioned waiting time exceeds the above-mentioned waiting time threshold.
[0015] According to embodiments of this disclosure, the method further includes: receiving the unit status of the smallest scheduling unit sent by each of the nodes, wherein the unit status of one or more specific nodes includes a unit abnormal exit status; obtaining from the database the task status and global task status of the distributed task process in the specific unit corresponding to the unit abnormal exit status; and updating the task status and global task status of the distributed task process in the specific unit to both the initialization and running states when the task status and global task status of the distributed task process are in the initialization or running state.
[0016] Thirdly, embodiments of this disclosure provide an apparatus for monitoring distributed tasks. The apparatus includes: a status monitoring client, configured to be located in the smallest scheduling unit of a node in a service cluster. The status monitoring client is used to control the startup and exit of each distributed task process, and to monitor the status of each distributed task process in the current unit. The status monitoring client includes: a task startup control module, a status monitoring and reporting module, a request sending module, and a task exit control module. The task startup control module is used to start a target distributed task process in the global task in response to a task processing request. The target distributed task process is located in the smallest scheduling unit of the current node. The status monitoring and reporting module is used to monitor the distributed task status of the target distributed task process and report it to a monitoring server. The request sending module is used to send a status acquisition request to the monitoring server based on a preset time interval. The task exit control module is used to control the exit of the target distributed task process and the unit container environment of the current node based on the global task status fed back by the monitoring server and the unit retention status setting information for the current node.
[0017] According to embodiments of this disclosure, the apparatus further includes an event monitoring and reporting module. This module monitors the unit status of the smallest scheduling unit in the current node and sends the monitored unit status as an event to the monitoring server. The unit status includes a unit abnormal exit status.
[0018] Fourthly, embodiments of this disclosure provide a monitoring server. The monitoring server includes: a status receiving module, a status updating and summarizing module, a request receiving module, and a status sending module. The status receiving module receives distributed task status information about their respective target distributed task processes from each node in the service cluster that executes the global task. The target distributed task processes are located in the smallest scheduling unit of each node. The status updating and summarizing module updates and summarizes the task status of the distributed task processes in the database based on the received distributed task status information of the target distributed task processes from each node, obtaining and storing the global task status. The request receiving module receives status acquisition requests from one or more nodes to obtain the global task status. In response to receiving the status acquisition request, the status sending module stores the time of receipt of the status acquisition request and sends the global task status information to the one or more nodes.
[0019] Fifthly, embodiments of this disclosure provide an electronic device. The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus; the memory stores computer programs; and the processor, when executing the program stored in the memory, implements the method for monitoring distributed tasks as described above.
[0020] Sixthly, embodiments of this disclosure provide a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method for monitoring distributed tasks as described above.
[0021] The technical solutions provided in the embodiments of this disclosure have at least some or all of the following advantages:
[0022] By monitoring and reporting the distributed task status of the target distributed task process in the smallest scheduling unit (e.g., a pod in a Kubernetes cluster) located on the current node, it can support task status monitoring scenarios where the smallest scheduling unit contains one or more containers. This enables status monitoring of each distributed task process, which helps the monitoring server record specific distributed task processes that have encountered problems within the unit. Simultaneously, by periodically obtaining the global task status from the monitoring server, and based on the global task status and the unit retention status settings for the current node, it can control the exit of the target distributed task process and the current unit container environment. The exit timing can be controlled based on the optional retention settings of the unit container environment. When a global task fails or an error occurs, if a unit retention status is set, the retention of the unit container environment allows relevant personnel (e.g., operations and maintenance personnel) to access the task execution site to handle the problem. Furthermore, based on the reported distributed task status, rapid problem localization can be achieved, serving as reference information for relevant personnel and enabling efficient troubleshooting and resolution. Attached Figure Description
[0023] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0024] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.
[0025] Figure 1 The system architecture of the method and apparatus for monitoring distributed tasks applicable to embodiments of this disclosure is illustrated schematically;
[0026] Figure 2 A flowchart illustrating a method for monitoring distributed tasks according to an embodiment of the present disclosure is shown schematically.
[0027] Figure 3 A detailed implementation flowchart of operation S202 according to an embodiment of the present disclosure is illustrated schematically;
[0028] Figure 4 A detailed implementation flowchart of operation S204 according to an embodiment of the present disclosure is illustrated schematically;
[0029] Figure 5 The schematic diagram illustrates an implementation flowchart of a control exit unit container environment according to an embodiment of the present disclosure;
[0030] Figure 6 A flowchart illustrating a method for monitoring distributed tasks according to another embodiment of this disclosure is shown schematically;
[0031] Figure 7 A flowchart illustrating a method for monitoring distributed tasks according to yet another embodiment of this disclosure is shown.
[0032] Figure 8A The illustration schematically shows a timing diagram of a method for monitoring a distributed task in the states of successful startup, running, and successful execution of a distributed task according to an embodiment of the present disclosure;
[0033] Figure 8B The illustration schematically shows a timing diagram of a method for monitoring a distributed task in a distributed task startup failure state according to an embodiment of the present disclosure;
[0034] Figure 8C A timing diagram illustrating a method for monitoring a distributed task in a distributed task execution failure state according to an embodiment of the present disclosure is shown.
[0035] Figure 9 A schematic flowchart illustrating the handling of anomalies in a status monitoring client according to an embodiment of the present disclosure is shown.
[0036] Figure 10 This schematically illustrates a timing diagram of the process when the smallest scheduling unit (pod) exits abnormally according to an embodiment of the present disclosure;
[0037] Figure 11 This schematic diagram illustrates a structural block diagram of an apparatus for monitoring distributed tasks according to an embodiment of the present disclosure;
[0038] Figure 12 A schematic diagram illustrating the structure of a monitoring server according to an embodiment of this disclosure is shown; and
[0039] Figure 13 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure is shown. Detailed Implementation
[0040] The embodiments of this disclosure provide a method, apparatus, server, device, and medium for monitoring distributed tasks. These methods support task status monitoring scenarios where the smallest scheduling unit contains one or more containers, enabling status monitoring of each distributed task process. This facilitates the recording of a specific distributed task process experiencing a problem within the unit on the monitoring server. Furthermore, the exit timing can be controlled based on the optional retention setting of the unit container environment. When a global task fails or an error occurs, with the unit retention setting enabled, the retention of the unit container environment allows relevant personnel (e.g., operations and maintenance personnel) to access the task execution site for problem handling. Moreover, based on the reported distributed task status, problem location can be quickly achieved, serving as reference information for relevant personnel and enabling efficient troubleshooting and resolution.
[0041] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.
[0042] Figure 1 The schematic illustration shows the system architecture of the method and apparatus for monitoring distributed tasks applicable to embodiments of this disclosure.
[0043] Reference Figure 1 As shown, the system architecture 100 of the method and apparatus for monitoring distributed tasks applicable to embodiments of this disclosure includes: a service request terminal 110, a service cluster 120, and a monitoring server 130. The service request terminal 110 and the service cluster 120, and the service cluster 120 and the monitoring server 130 are connected via a network.
[0044] The aforementioned network is a medium for providing communication links and can include various connection types, such as wired or wireless communication links or fiber optic cables, etc.
[0045] The aforementioned service request side 110 can be terminal devices, application servers, etc., that require the service cluster 120 to provide various services such as Software as a Service (SaaS), Platform as a Service (PAAS), or computing services. For example, terminal devices can include, but are not limited to, smartphones, tablets, laptops, desktop computers, smartwatches, smart robots, etc.
[0046] Service cluster 120 may be a cluster including at least one service node (hereinafter referred to as a node) for providing data and / or computing services to service requester 110. Service cluster 120 may be a cloud computing service cluster, such as a Kubernetes cluster. The service cluster 120 is used to schedule computing resources and perform data processing based on those resources. In one embodiment, distributed tasks in a global task are distributed to at least one node in the cluster for processing.
[0047] Users (such as algorithm developers) can submit task processing requests after performing relevant operations on the operation interface of service request side 110. These operations may include: creating a model name, dividing a model training task (an example of a global task) into multiple sub-training tasks (an example of a distributed task), and then submitting the task. For example, the submitted task processing request carries the identifier of the global task, the identifiers of each distributed task, and the execution instructions for processing the corresponding distributed tasks.
[0048] The aforementioned service cluster 120 will create nodes for executing the aforementioned global tasks based on the relevant information submitted by the user, or allocate nodes for executing the aforementioned global tasks from existing service nodes. It is understood that when the smallest scheduling unit in the aforementioned nodes includes multiple containers, the multiple distributed tasks in the aforementioned global tasks can be processed in multiple containers on the same node, or the multiple distributed tasks in the aforementioned global tasks can be distributed to multiple nodes for processing.
[0049] Understandably, nodes in a service cluster are electronic devices capable of providing storage and / or computing resources.
[0050] In the embodiments of this disclosure, a new functional module is proposed in the service cluster, described as a status monitoring client. This status monitoring client is set in the smallest scheduling unit of each node in the service cluster. The status monitoring client is used to control the start and stop of each distributed task process in its unit (pod), and to monitor the status of each distributed task process in its unit, referring to... Figure 1 The dashed boxes in the middle illustrate the status monitoring clients located in pod1 on node 1 and pod2 on node 2, respectively.
[0051] Based on the above, a system architecture for a distributed task status monitoring system (hereinafter referred to as the monitoring system) is also provided. This system architecture includes, in addition to the aforementioned status monitoring client, the aforementioned monitoring server 130, as described above. Figure 1 As shown.
[0052] The aforementioned status monitoring client is one of the core components of the distributed task status monitoring system provided in this embodiment of the disclosure. It is used to collect, summarize and report the status of distributed tasks, as well as to control the start and exit of distributed tasks.
[0053] The aforementioned monitoring server 130 is another core component of the distributed task status monitoring system. It is used to process, summarize, and store the status of distributed tasks, thereby managing the status of distributed tasks, and to interact with the status monitoring client.
[0054] As an example of specific functionalities in the architecture (the functional modules of monitoring server 130 can be merged or split, as described in the third embodiment below, and are not limited to the architecture shown here), refer to... Figure 1 As shown, the monitoring server 130 includes a control unit (Monitor), which includes a control module (Controller component) and a status processing module (Train_status component).
[0055] The controller module obtains the unit status (or pod status) of pods in Kubernetes through events (informers). These events (informers) are a monitoring mechanism in Kubernetes, responsible for monitoring and reporting the creation, modification, and deletion of pod status in an event-driven manner.
[0056] The aforementioned status monitoring client sends changes in the distributed state to the task status queue as messages. The control unit (Monitor) consumes and processes the task status. Specifically, the message status processing module (Train_status component) consumes messages from the task status queue to obtain the status of the distributed tasks from the status monitoring client. In addition, the status of the distributed tasks obtained from the task status queue is integrated with the pod status obtained from the control module (Controller component) to obtain the complete task status, which is then written to a database (e.g., Redis) for storage.
[0057] The aforementioned database is used to store basic information about distributed tasks and their status, for querying and display.
[0058] It should be understood that, Figure 1 The number of service demand side 110, the number of nodes in service cluster 120, and the number of monitoring service side 130 are merely illustrative.
[0059] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.
[0060] This disclosure provides a method for monitoring distributed tasks. The method provided in this embodiment can be applied to... Figure 1 Each node in the example service cluster 120 is used to perform global tasks.
[0061] Figure 2 A flowchart illustrating a method for monitoring distributed tasks according to an embodiment of the present disclosure is shown schematically.
[0062] Reference Figure 2 As shown, the method for monitoring distributed tasks provided in this embodiment includes the following operations: S201, S202, S203 and S204.
[0063] In operation S201, in response to the task processing request, the target distributed task process in the global task is started. The target distributed task process is located in the smallest scheduling unit of the current node.
[0064] exist Figure 1 The service cluster 120 is a Kubernetes cluster, which includes two nodes: node 1 and node 2. For example, each node has a pod containing one or more containers, and each container can run a distributed task process.
[0065] The aforementioned target distributed task process is the task processing process of one or more containers located in the smallest scheduling unit (e.g., a pod) of the current node.
[0066] Combination Figure 1 and Figure 2 As can be seen, taking task A for model training as an example of a global task, global task A is divided into 3 subtasks A1 and A2. 11 A 12 and A 13 The smallest scheduling unit on node 1 is container V in pod1. 11 and container V 12 Each handles a subtask, corresponding to A respectively. 11 and A 12 The smallest scheduling unit on node 2 is container V in pod2. 21 Process subtask A 13 .
[0067] For node 1, the target distributed task process is the subtask A in pod 1. 11 and A 12 Two containers V 11 and V 12 The task processing process.
[0068] For node 2, the target distributed task process is the subtask A in pod2. 13 Container V 21 The task processing process.
[0069] In operation S202, monitor the distributed task status of the aforementioned target distributed task process and report it to the monitoring server.
[0070] After each node reports the distributed task status of the target distributed task process running in its respective pod to the monitoring server, the monitoring server updates and summarizes the task status of the distributed task process in the database based on the received distributed task status of the target distributed task process from each node, obtains the global task status, and stores it.
[0071] In this way, based on the reported distributed task status, the distributed task status stored on the monitoring server side and the global task status obtained after aggregation can help relevant personnel to quickly locate task processing problems, and serve as reference information for relevant personnel to achieve efficient troubleshooting and resolution of faults.
[0072] In operation S203, based on preset time interval information, a status acquisition request for obtaining the global task status is sent to the aforementioned monitoring server.
[0073] In operation S204, based on the global task status feedback from the monitoring server and the unit retention status settings for the current node, the target distributed task process and the unit container environment of the current node are controlled to exit.
[0074] The above operations S201 to S204 can be performed by the status monitoring client in the node.
[0075] For example, in one exemplary embodiment, the entry command for creating the smallest scheduling unit (pod) is pre-configured as the startup command for the aforementioned status monitoring client. When a node is created or configured, the pod is created, and the status monitoring client located within that pod is simultaneously started. Once started, the status monitoring client autonomously controls the startup and status monitoring of distributed tasks.
[0076] Based on the above operations S201-S204, by monitoring and reporting the distributed task status of the target distributed task process in the smallest scheduling unit (e.g., a pod in a Kubernetes cluster) located at the current node, it can support task status monitoring scenarios where the smallest scheduling unit contains one or more containers, realizing status monitoring of each distributed task process. This helps the monitoring server record a specific distributed task process in the unit that has a problem. At the same time, by periodically obtaining the global task status from the monitoring server, and based on the global task status and the unit retention status setting information for the current node, it controls the exit of the target distributed task process and the current unit container environment. It can control the exit timing based on the optional retention setting of the unit container environment. When the global task fails or an error occurs, if the unit retention status is set, the retention of the unit container environment can facilitate relevant personnel (e.g., operations and maintenance personnel) to enter the task execution site to handle the problem. Furthermore, based on the reported distributed task status, it can achieve rapid problem location, serving as reference information for relevant personnel, and achieving efficient troubleshooting and resolution.
[0077] The following refers to... Figure 3 and Figure 4 The above operations S202 and S204 will be described in detail.
[0078] Figure 3 A detailed implementation flowchart of operation S202 according to an embodiment of the present disclosure is illustrated.
[0079] According to an embodiment of this disclosure, when the minimum scheduling unit includes one or more target distributed task processes, the operation S202, which monitors the distributed task status of the target distributed task processes and reports it to the monitoring server, includes the following operations: S301 and S302.
[0080] In operation S301, monitor the startup and running status of each target distributed task process in the aforementioned minimum scheduling unit.
[0081] In operation S302, based on the above startup and running status, the distributed task status corresponding to each of the above target distributed task processes is sent to the monitoring server.
[0082] The above operation S302 specifically includes one or more of the following situations (a) to (c):
[0083] (a) If the startup status of the current target distributed task process is detected as successful, send the above-mentioned distributed task status of the current target distributed task process to the task status queue of the monitoring server as: running state;
[0084] (b) If the current target distributed task process is detected to be running successfully, the distributed task status of the current target distributed task process is sent to the task status queue of the monitoring server as: execution successful.
[0085] (c) If the current target distributed task process is detected to have failed to start or failed to run, the task status of the current target distributed task process is sent to the task status queue of the monitoring server as: failed, along with the corresponding failure reason information.
[0086] By monitoring each target distributed process in the smallest scheduling unit (pods on each node in a Kubernetes cluster), it is possible to support task status monitoring scenarios where the smallest scheduling unit contains one or more containers, enabling status monitoring of each distributed task process. This helps to identify a specific distributed task process that has encountered a problem in the monitoring server's record unit.
[0087] Figure 4 A detailed implementation flowchart of operation S204 according to an embodiment of the present disclosure is illustrated.
[0088] According to embodiments of this disclosure, referring to Figure 4 As shown, in the above operation S204, based on the global task status fed back by the monitoring server and the unit retention status setting information for the current node, the target distributed task process and the unit container environment of the current node are controlled to exit, including the following operations: S401, S402 and S403.
[0089] In operation S401, based on the aforementioned global task status, it is determined whether the aforementioned global task is in the execution completed state.
[0090] The execution completion status mentioned above includes either a success status or a failure status.
[0091] In operation S402, if it is determined that the above-mentioned global task is in the execution completed state, the exit timing information of the above-mentioned target distributed task process and the current node's unit container environment is determined according to the above-mentioned unit retention state setting information for the current node.
[0092] Figure 5 The schematic diagram illustrates an implementation flowchart of a control exit unit container environment according to an embodiment of the present disclosure.
[0093] Reference Figure 5As shown, the specific implementation process of the above operation S402 may include the following logic: when the above unit retention state setting information is not to retain, the exit timing information of the above target distributed task process and the unit container environment of the current node is determined as follows: when it is determined that the global task changes from the running state to the execution end state, the above target distributed task process and the unit container environment of the current node exit.
[0094] The specific implementation process of the above operation S402 may also include the following logic: when the above unit retention state setting information is to retain and a retention duration is set, the exit timing information of the above target distributed task process and the unit container environment of the current node is determined as follows: when it is determined that the global task changes from the running state to the execution end state, the above target distributed task process and the unit container environment of the current node are controlled to continue to maintain the above retention duration before exiting.
[0095] The aforementioned unit retention status settings can be set based on the expected execution effect of the global task (generally unlikely to fail, likely to fail, etc.): do not retain (e.g., for an expected execution effect that is generally unlikely to fail) or retain (e.g., for an expected execution effect that is likely to fail). The retention duration settings can be pre-configured by operations / development personnel, etc., according to actual needs when setting the pod status.
[0096] In operation S403, based on the aforementioned exit timing information, the target distributed task process and the current node's unit container environment are controlled to exit.
[0097] By controlling the exit of the target distributed task process and the current unit container environment based on the global task status and the unit retention status settings for the current node, the exit timing can be controlled according to the optional retention settings of the unit container environment. When the global task fails or an error occurs, if the unit retention status is set, the retention of the unit container environment can facilitate relevant personnel (such as operation and maintenance personnel) to enter the task execution site to handle the problem.
[0098] Figure 6 A flowchart illustrating a method for monitoring distributed tasks according to another embodiment of this disclosure is shown schematically.
[0099] In another embodiment, reference is made to Figure 6 As shown, the method for monitoring distributed tasks includes not only operations S201 to S204 as described above, but also the following operations: S601 and S602. Operations S601 and S602 are executed after operation S201, and operations S601 and S202 are parallel operation steps.
[0100] In operation S601, the status of the smallest scheduling unit in the current node is monitored, including the abnormal exit status of the unit.
[0101] When operating S602, the monitored unit status is sent to the aforementioned monitoring server in the form of an event.
[0102] Reference Figure 1 As shown, the above operations S601 and S602 can be implemented by the monitoring function module set up inside the node in the Kubernetes cluster. This monitoring function module monitors the pod status (creating, modifying, deleting, abnormally exiting, etc.) and reports it to the monitoring server through event-based (informer) methods.
[0103] In one embodiment, the monitoring function module and the status monitoring client are two different monitoring modules; or, in other embodiments, the functions of the monitoring function module and the status monitoring client can be integrated into one monitoring module.
[0104] In embodiments including steps S201-S204, S601, and S602, the unit status of the smallest scheduling unit (e.g., a pod) of the current node and the distributed task status of the target distributed task process in the smallest scheduling unit are monitored and reported simultaneously. This allows the monitoring server to obtain the overall status of the pod and its internal processes, achieving comprehensive and accurate status monitoring. This supports task status monitoring scenarios where the smallest scheduling unit contains one or more containers.
[0105] This disclosure provides a method for monitoring distributed tasks. The method provided in this embodiment can be applied to a monitoring server, the exemplary architecture of which can be referred to... Figure 1 The description of the system architecture in the text will not be repeated here.
[0106] Figure 7 A flowchart illustrating a method for monitoring distributed tasks according to yet another embodiment of this disclosure is shown.
[0107] Reference Figure 7 As shown, the method for monitoring distributed tasks provided in this disclosure includes the following operations: S701, S702, S703 and S704.
[0108] In operation S701, the distributed task status of each target distributed task process is sent by each node in the service cluster that is used to execute the global task. The target distributed task process is located in the smallest scheduling unit of each node.
[0109] Reference Figure 1As shown, monitoring server 130 receives data from node 1 regarding subtask A. 11 and A 12 Two containers V 11 and V 12 The task status of the task processing process; receiving the message from node 2 regarding subtask A. 13 Container V 21 The task status of the task processing process.
[0110] During operation S702, based on the distributed task status of the target distributed task processes of the aforementioned nodes, the task status of the distributed task processes in the database is updated and summarized to obtain the global task status and store it.
[0111] According to embodiments of this disclosure, the above operation S602 may specifically include one or more of the following cases (a) to (d):
[0112] (a) If the received distributed task status is a failure status and the failure reason information is startup failure, update the task status of the distributed task process in the database from initialization to failure status, and update the global task status corresponding to the above global task to failure status.
[0113] (b) If the received distributed task status is in the running state, update the task status of the distributed task process in the database from initialization to running state, and if the task status of all distributed task processes under the above global task is in the running state, update the global task status corresponding to the above global task to running state.
[0114] (c) If the received distributed task status is successful, update the task status of the distributed task process in the database from running to successful. If the task status of all distributed task processes under the global task is successful, update the global task status corresponding to the global task to successful.
[0115] (d) If the received distributed task status is a failure status and the failure reason information is running failure, update the task status of the distributed task process in the database from running status to failure status, and update the global task status corresponding to the above global task to failure status.
[0116] The aforementioned failure or success states both belong to the execution completion state.
[0117] In operation S703, a status acquisition request for obtaining the global task status is received from one or more of the aforementioned nodes.
[0118] In operation S704, in response to receiving the aforementioned status acquisition request, the receiving time of receiving the aforementioned status acquisition request is stored, and global task status information is sent to one or more of the aforementioned nodes.
[0119] Based on operations S701 to S704, by receiving the monitoring results of the distributed task status of the target distributed task process in the smallest scheduling unit (e.g., a pod in a Kubernetes cluster) of each node, and updating and summarizing the task status of the distributed task process in the database according to the monitoring results, the global task status is obtained and stored. This enables real-time recording / storage of the status of each distributed task process, and records a specific distributed task process in the smallest scheduling unit (pod) where a problem occurs, facilitating rapid problem localization and serving as reference information for relevant personnel, thus achieving efficient troubleshooting and resolution.
[0120] The following is combined Figures 8A to 8C This example illustrates the interaction process between the status monitoring client and the monitoring server in a monitoring system.
[0121] Figure 8A The illustration schematically shows a timing diagram of a method for monitoring a distributed task in the states of successful startup, running, and successful execution of a distributed task according to an embodiment of the present disclosure; Figure 8B The illustration schematically shows a timing diagram of a method for monitoring a distributed task in a distributed task startup failure state according to an embodiment of the present disclosure; Figure 8C The illustration schematically shows a timing diagram of a method for monitoring a distributed task in a distributed task execution failure state according to an embodiment of the present disclosure.
[0122] In one embodiment, the status monitoring client in the smallest scheduling unit (e.g., a pod) of each node in the service cluster executes operations S201 and S202. In operation S202, the distributed task status of the target distributed task process is monitored and reported to the monitoring server, referring to... Figure 8A and Figure 8C The example reported the status as "Running", or refer to... Figure 8B The reported status in the example is "failure". Accordingly, the monitoring server executes operation S701, receiving the distributed task status of their respective target distributed task processes from the aforementioned nodes; then, the monitoring server executes operation S702, updating and summarizing the task status of the distributed task processes in the database based on the received distributed task statuses from the aforementioned nodes, obtaining the global task status and storing it, for example... Figures 8A to 8CThe example task status (including the global task status and the distributed task status of each target distributed task process) is written to Redis.
[0123] Subsequently, the status monitoring client executes operation S203, sending a status acquisition request to the monitoring server based on a preset time interval. Correspondingly, the monitoring server executes operation S701, receiving the status acquisition request, and then executes S702, sending the global task status information to the node. Correspondingly, the status monitoring client executes operation S204, corresponding to... Figures 8A to 8C The exit logic is handled within the program.
[0124] Figure 9 A schematic flowchart illustrating the handling of anomalies in a status monitoring client according to an embodiment of the present disclosure is shown.
[0125] In some situations, the status monitoring client may fail to report its status correctly. For example, if the task starts normally but the status monitoring client exits abnormally, or if network problems prevent it from connecting to the control unit (Monitor), the Monitor may be unable to obtain the status of the distributed task. Refer to the aforementioned embodiments regarding normal communication states and... Figures 8A to 8C As shown, the status monitoring client periodically pings (used to test the reachability of hosts on the Internet Protocol network) the Monitor on the monitoring server and records the ping time in Redis (an example of a database). In the Monitor, tasks with a status of "running" (or described as running) are checked periodically. If a "running" task is found to have a waiting time exceeding the waiting time threshold without being pinged, the task status is set to failed.
[0126] According to another embodiment of this disclosure, the method applied to the monitoring server, in addition to the operations S701 to S704 described above, also includes the following operations: S901, S902, and S903. (See also...) Figure 9 As shown, for the sake of simplicity, only the specific implementation process of operations S901 to S903 is illustrated.
[0127] In operation S901, the task status of the distributed task process is determined to be the waiting distributed task process corresponding to the running state. Figure 9 The example uses a control unit as the execution subject for operations S901 to S903.
[0128] In operation S902, based on the aforementioned reception time, communication timeout detection is performed on the waiting nodes corresponding to the aforementioned waiting distributed task process.
[0129] According to embodiments of this disclosure, referring to Figure 9As shown, the above-mentioned communication timeout detection for the waiting nodes corresponding to the above-mentioned distributed task process based on the above-mentioned reception time includes: S902a, for each waiting node, calculating the difference between the current detection time and the latest reception time corresponding to the current waiting node to obtain the waiting time; S902b, determining whether the above-mentioned waiting time exceeds the waiting time threshold, the above-mentioned waiting time threshold is related to the preset time interval information of the current waiting node sending the status acquisition request; S902c, if the above-mentioned waiting time exceeds the above-mentioned waiting time threshold, determining that the current waiting node has a communication timeout.
[0130] In operation S903, if a communication timeout is detected in the target node among the above-mentioned waiting nodes, the task status of the distributed task process corresponding to the target node is set to a failure status, the corresponding global task status is updated, and the target node and timeout information are stored as failure reason information.
[0131] Figure 10 The schematic diagram illustrates the processing timing when the smallest scheduling unit (pod) exits abnormally according to an embodiment of the present disclosure.
[0132] According to another embodiment of this disclosure, based on the foregoing embodiments, in addition to including the operations S701-S704, or in addition to including operations S701-S704 and S901-S903, the above method may also include the following operations: S1001, S1002, and S1003. Figure 10 For the sake of simplicity, only the specific implementation sequence of operations S1001 to S1003 is shown in the illustration.
[0133] In operation S1001, the unit status of the minimum scheduling unit sent by each of the above nodes is received, wherein the unit status of one or more specific nodes includes the unit abnormal exit status.
[0134] In operation S1002, the task status and global task status of the distributed task process in the specific unit corresponding to the abnormal exit status of the above unit are obtained from the database.
[0135] In operation S1003, if the task status of the aforementioned distributed task process and the global task status are either initialized or running, updating the task status of the distributed task process and the global task status in the aforementioned specific unit will both fail.
[0136] The above method also includes: when the task status of the above distributed task process and the global task status are in the execution completed state (including success state and failure state), there is no need to update the task status of the distributed task process and the global task status in the above specific unit, and the abnormal exit status of the above unit can be stored as the running log.
[0137] Embodiments of this disclosure also provide an apparatus for monitoring distributed tasks.
[0138] Figure 11 A schematic block diagram of an apparatus for monitoring distributed tasks according to an embodiment of the present disclosure is shown.
[0139] Reference Figure 11 As shown in the solid line box, the aforementioned device 1100 for monitoring distributed tasks includes: a status monitoring client 1110, which is configured in the smallest scheduling unit of the nodes in the service cluster. The status monitoring client is used to control the start and exit of each distributed task process, and to monitor the status of each distributed task process in the current unit.
[0140] The aforementioned status monitoring client 1110 includes: a task start control module 1111, a status monitoring and reporting module 1112, a request sending module 1113, and a task exit control module 1114.
[0141] The task initiation control module 1111 mentioned above is used to initiate the target distributed task process in the global task in response to a task processing request. The target distributed task process is located in the smallest scheduling unit of the current node.
[0142] The aforementioned status monitoring and reporting module 1112 is used to monitor the distributed task status of the aforementioned target distributed task process and report it to the monitoring server.
[0143] The aforementioned request sending module 1113 is used to send a status acquisition request to the aforementioned monitoring server to obtain the global task status based on a preset time interval.
[0144] The task exit control module 1114 is used to control the exit of the target distributed task process and the unit container environment of the current node based on the global task status fed back by the monitoring server and the unit retention status setting information for the current node.
[0145] According to embodiments of this disclosure, referring to Figure 11As shown in the dashed box, the aforementioned device 1100 may further include an event monitoring and reporting module 1120. The event monitoring and reporting module 1120 is used to monitor the unit status of the smallest scheduling unit in the current node and send the monitored unit status to the monitoring server in the form of events. The unit status includes the unit's abnormal exit status.
[0146] Embodiments of this disclosure also provide a monitoring server.
[0147] Figure 12 A schematic block diagram of a monitoring server according to an embodiment of the present disclosure is shown.
[0148] Reference Figure 12 As shown, the monitoring server 1200 provided in this embodiment includes: a status receiving module 1201, a status update and summary module 1202, a request receiving module 1203, and a status sending module 1204.
[0149] The aforementioned status receiving module 1201 is used to receive the distributed task status of each target distributed task process sent by each node in the service cluster that is used to execute the global task. The target distributed task process is located in the smallest scheduling unit of each node.
[0150] The aforementioned status update and summary module 1202 is used to update and summarize the task status of the distributed task processes in the database based on the received distributed task status of the target distributed task processes of the aforementioned nodes, so as to obtain the global task status and store it.
[0151] The aforementioned request receiving module 1203 is used to receive status acquisition requests sent by one or more of the aforementioned nodes for obtaining the global task status.
[0152] The aforementioned status sending module 1204 is used to respond to receiving the aforementioned status acquisition request, store the receiving time of receiving the aforementioned status acquisition request, and send global task status information to the aforementioned one or more nodes.
[0153] According to embodiments of this disclosure, the monitoring server 1200 may further include: a determination module, a timeout detection module, and a status setting module; further, it may also include: a unit status receiving module and a task status acquisition module.
[0154] The aforementioned determining module is used to identify the distributed task process whose task status is in the running state, corresponding to the waiting distributed task process. Figure 9 The example uses a control unit as the execution subject for operations S901 to S903.
[0155] The aforementioned detection module is used to perform communication timeout detection on the waiting nodes corresponding to the aforementioned waiting distributed task processes, based on the aforementioned reception time.
[0156] The aforementioned status setting module is used to set the task status of the distributed task process corresponding to the aforementioned target node to a failed state when a communication timeout is detected in the target node, update the corresponding global task status, and store the aforementioned target node and timeout information as failure reason information.
[0157] The above-mentioned unit status receiving module (e.g., is) Figure 1 The example controller component is used to receive the unit status of the minimum scheduling unit sent by each of the above nodes, where the unit status of one or more specific nodes includes the unit abnormal exit status.
[0158] The above task status acquisition module (e.g., for example) Figure 1 The example Train-status component is used to retrieve the task status of the distributed task process and the global task status of the specific unit corresponding to the abnormal exit status of the above unit from the database.
[0159] The aforementioned status update and summary module 1202 is also used to update the task status of the distributed task process and the global task status in the aforementioned specific unit to a failed state when the task status and global task status of the aforementioned distributed task process are initialized or in a running state.
[0160] Any plurality of the modules exemplified in apparatus 1100 and apparatus 1200 described above can be combined into a single module, or any one of the modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in a single module. At least one of the modules exemplified in apparatus 1100 and apparatus 1200 described above can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the modules exemplified in apparatus 1100 and apparatus 1200 described above can be at least partially implemented as a computer program module, which, when run, can perform a corresponding function.
[0161] Embodiments of this disclosure also provide an electronic device.
[0162] Figure 13 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure is shown.
[0163] Reference Figure 13 As shown, the electronic device 1300 provided in this embodiment includes a processor 1301, a communication interface 1302, a memory 1303, and a communication bus 1304. The processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304. The memory 1303 is used to store computer programs. When the processor 1301 executes the program stored in the memory, it implements the method for monitoring distributed tasks as described above.
[0164] Embodiments of this disclosure also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method for monitoring distributed tasks as described above.
[0165] The computer-readable storage medium may be included in the device / apparatus described in the above embodiments; or it may exist independently and not assembled into the device / apparatus. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.
[0166] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0167] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0168] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A method of monitoring a distributed task, characterized by, Applied to nodes in a service cluster, the method includes, for each node: In response to a task processing request, a target distributed task process in the global task is started. The target distributed task process is located in the minimum scheduling unit of the current node. The minimum scheduling unit includes one or more containers, each of which runs a distributed task process. Monitor the distributed task status of the target distributed task process and report it to the monitoring server; Based on a preset time interval, a status acquisition request for obtaining the global task status is sent to the monitoring server. Based on the global task status fed back by the monitoring server and the unit retention status setting information for the current node, the target distributed task process and the unit container environment of the current node are controlled to exit. Specifically, when the unit retention status setting is set to retain with a retention duration, the exit timing information for the target distributed task process and the unit container environment of the current node is determined as follows: when it is determined that the global task has changed from a running state to an execution-completed state, the target distributed task process and the unit container environment of the current node are controlled to maintain the retention duration before exiting.
2. The method of claim 1, wherein, The step of controlling the exit of the target distributed task process and the unit container environment of the current node based on the global task status fed back by the monitoring server and the unit retention status setting information for the current node includes: Based on the global task status, determine whether the global task is in the execution completed state; If it is determined that the global task is in an execution completed state, the exit timing information of the target distributed task process and the current node's unit container environment is determined based on the unit retention state setting information for the current node; and Based on the exit timing information, the target distributed task process and the current node's unit container environment are controlled to exit.
3. The method of claim 2, wherein, The step of determining the exit timing information of the target distributed task process and the current node's unit container environment based on the unit retention status setting information for the current node includes: When the unit retention status setting information is set to not retain, the exit timing information for the target distributed task process and the unit container environment of the current node is determined as follows: when it is determined that the global task changes from the running state to the execution end state, the target distributed task process and the unit container environment of the current node exit.
4. The method of claim 1, wherein, When the minimum scheduling unit includes one or more target distributed task processes, the step of monitoring the distributed task status of the target distributed task processes and reporting it to the monitoring server includes: Monitor the startup and running status of each target distributed task process in the minimum scheduling unit; Based on the startup and running status, the distributed task status corresponding to each target distributed task process is sent to the monitoring server.
5. The method according to claim 4, characterized in that, The step of sending the distributed task status corresponding to each target distributed task process to the monitoring server based on the startup and running status includes: If the startup status of the current target distributed task process is detected as successful, the distributed task status of the current target distributed task process is sent to the task status queue of the monitoring server as: in running state; If the current target distributed task process is detected to be running successfully, the distributed task status of the current target distributed task process is sent to the task status queue of the monitoring server as: execution successful. If the current target distributed task process is detected to have either a startup failure or a running failure status, the system sends the distributed task status of the current target distributed task process as "failed" to the task status queue of the monitoring server, along with the corresponding failure reason information.
6. The method of claim 1, wherein, Also includes: Monitor the status of the smallest scheduling unit in the current node, including the abnormal exit status of the unit; as well as The monitored unit status is sent to the monitoring server in the form of an event.
7. A method of monitoring a distributed task, characterized by, Applied to a monitoring server, the method includes: The system receives distributed task status information about the respective target distributed task processes sent by each node in the service cluster that is used to execute global tasks. The target distributed task processes are located in the smallest scheduling unit of each node. The smallest scheduling unit includes one or more containers, each of which runs a distributed task process. Based on the received distributed task status of the target distributed task process of each node, the task status of the distributed task process in the database is updated and summarized to obtain the global task status and store it. Receive a status acquisition request from one or more of the nodes to obtain the global task status; and In response to receiving the status acquisition request, the receiving time of the status acquisition request is stored, and global task status information is sent to the one or more nodes.
8. The method of claim 7, wherein, Also includes: The task status of the distributed task process is determined to be the waiting distributed task process that is in the running state; Based on the receiving time, communication timeout detection is performed on the waiting nodes corresponding to the waiting distributed task process; If a communication timeout is detected in the target waiting node among the waiting nodes, the task status of the distributed task process corresponding to the target waiting node is set to a failed state, the corresponding global task status is updated, and the target waiting node and timeout information are stored as failure reason information.
9. The method of claim 8, wherein, The step of performing communication timeout detection on the waiting nodes corresponding to the waiting distributed task process based on the receiving time includes: For each waiting node, the waiting time is calculated by subtracting the current detection time from the latest reception time corresponding to the current waiting node. Determine whether the waiting time exceeds a waiting time threshold, wherein the waiting time threshold is related to a preset time interval information for the current waiting node to send a status acquisition request; If the waiting time exceeds the waiting time threshold, it is determined that the current waiting node has a communication timeout.
10. The method of claim 7, wherein, Also includes: Receive the unit status of the minimum scheduling unit sent by each node, wherein the unit status of one or more specific nodes includes a unit abnormal exit status; Retrieve the task status and global task status of the distributed task process in the specific unit corresponding to the abnormal exit status of the unit from the database; If the task state of the distributed task process and the global task state are in the initialization or running state, updating the task state of the distributed task process and the global task state in the specific unit will both fail.
11. An apparatus for monitoring a distributed task, the apparatus comprising: The device includes: a status monitoring client, configured to be installed in the smallest scheduling unit of a node in the service cluster, the status monitoring client comprising: The task startup control module is used to respond to a task processing request and start a target distributed task process in the global task. The target distributed task process is located in the minimum scheduling unit of the current node. The minimum scheduling unit includes one or more containers, and each container runs a distributed task process. The status monitoring and reporting module is used to monitor the distributed task status of the target distributed task process and report it to the monitoring server. The request sending module is used to send a status acquisition request to the monitoring server to obtain the global task status based on a preset time interval. The task exit control module is used to control the exit of the target distributed task process and the unit container environment of the current node based on the global task status fed back by the monitoring server and the unit retention status setting information for the current node. Specifically, when the unit retention status setting information indicates retention with a specified retention duration, the exit timing information for the target distributed task process and the unit container environment of the current node is determined as follows: when it is determined that the global task has changed from a running state to an execution-completed state, the module controls the target distributed task process and the unit container environment of the current node to maintain the specified retention duration before exiting.
12. The apparatus of claim 11, wherein, Also includes: The event monitoring and reporting module is used to monitor the unit status of the smallest scheduling unit in the current node. The unit status includes the abnormal exit status of the unit, and the monitored unit status is sent to the monitoring server in the form of an event.
13. A monitoring server, characterized by include: The status receiving module is used to receive the distributed task status of each target distributed task process sent by each node in the service cluster that is used to execute the global task. The target distributed task process is located in the minimum scheduling unit of each node. The minimum scheduling unit includes one or more containers, and each container runs a distributed task process. The status update and aggregation module is used to update and aggregate the task status of the distributed task processes in the database based on the received distributed task status of the target distributed task processes of each node, so as to obtain the global task status and store it. The request receiving module is configured to receive status acquisition requests for obtaining the global task status sent by one or more of the nodes; and The status sending module is used to respond to receiving the status acquisition request, store the time of receiving the status acquisition request, and send global task status information to the one or more nodes.
14. An electronic device, comprising: It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-10.
15. A computer readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-10.
Citation Information
Patent Citations
Fault positioning method and device, electronic equipment and readable storage medium
CN114024834A
