A method for synchronizing task status in a cluster

Abstract

The present invention belongs to the field of computer software, and proposes a method for synchronizing task status in a cluster. Through the status reporting interface, the executor reports the second task status of the working node, and the scheduler sends the container that needs to be cleaned, thereby realizing two-way state synchronization between the executor and the scheduler. The executor periodically sends a request to query the task status of the container to the container API in the working node, thereby avoiding the failure to timely discover the container with abnormal status; by merging the first task status, error code or health check result of the container, the second task status is obtained, which effectively makes up for the limitation of relying solely on the feedback status of the container itself; reporting in units of working nodes reduces the number of network communications and has better performance than reporting the task status of each container separately; by establishing a long connection in units of working nodes to achieve task status synchronization, the total number of connections is greatly reduced compared to the method of establishing a long connection for each container separately, and the memory usage of the server is effectively reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer software, and in particular to a method for synchronizing task states in a cluster. Background Art

[0002] Linux container technology is a lightweight virtualization technology that originates from the cgroups and namespaces features of the Linux kernel, developed under the leadership of Linus Torvalds. It allows developers to package applications and their dependencies into a lightweight, portable container.

[0003] In a cluster, the scheduler distributes tasks to each executor, which then executes them. However, current technologies fail to achieve bidirectional synchronization of task status between the scheduler and executors. This flaw causes a large number of tasks that should have been deleted to remain in the cluster, continuously occupying runtime memory resources, resulting in resource waste and degraded cluster performance. Furthermore, because the scheduler cannot monitor the specific execution status of tasks in real time, it struggles to quickly and accurately perform necessary operations such as terminating or restarting tasks when faced with task exceptions or timeouts, severely impacting the efficiency and flexibility of cluster task management. Summary of the Invention

[0004] In order to solve the technical problem in the prior art that task states between a scheduler and an executor in a cluster cannot be bidirectionally synchronized, the present invention proposes a method for synchronizing task states in a cluster to solve the above problem.

[0005] The present invention provides a method for synchronizing task status in a cluster, the method comprising the following steps:

[0006] A method for synchronizing task status in a cluster, including the following steps:

[0007] S1: Query and obtain the first task status of the container: The task runs in the form of a container on each working node. The executor in each working node periodically sends a container status query request to the container API; the container API feeds back the first task status of all containers in the corresponding working node to the executor; the first task status of all containers in the working node at the current moment is aggregated to obtain a first task status set;

[0008] S2: Construct a second task state set: The executor traverses the first task state of each container in the first task state set obtained by the corresponding working node in S1; determines whether to obtain an error code and perform a health check based on the first task state, and merges the error code and health check result with the first task state to obtain a second task state; after the traversal is completed, the work log of each container in the first task state set, the first task state, and the second task state are aggregated into a second task state set;

[0009] S3: Reporting the second task status set: the executor reports the second task status set of the working node corresponding to the executor to the scheduler through the status reporting interface;

[0010] S4: Comparing and determining containers to be cleaned: The scheduler compares the second task state set with a third task state set pre-stored in a database for the working node corresponding to the second task state set, and obtains a set of containers to be cleaned of the working node according to the comparison result;

[0011] S5: Returning the set of containers to be cleaned: The scheduler calls the status reporting interface and sends the set of containers to be cleaned to the executor in the working node corresponding to the set of containers to be cleaned;

[0012] S6: Container cleaning: The executor cleans containers according to the set of containers to be cleaned.

[0013] Preferably, the method for obtaining the first task status set of the container in step S1 is: the executor in the working node periodically sends a container status query request to the container API in the working node; the container API feeds back the first task status of all containers in the working node to the executor based on the container status query request; and the first task status of all containers in the working node at the current moment is summarized to obtain the first task status set of the working node at the current moment.

[0014] Preferably, the step of constructing the second task state set in step S2 is:

[0015] A1: Determine whether all containers of the first task state set have been traversed. If not, jump to A2. Otherwise, jump to A4.

[0016] A2: Determine the first task status of the current traversal container; if the first task status of the current traversal container is "exited", obtain the error code of the current traversal container; combine the first task status of the current traversal container and the error code to obtain the second task status of the current traversal container related to the business; collect the full log of the current traversal container; add the full log and the second task status related to the business to the first task status set to form the second task status set; after the addition is completed, jump to A1; if the first task status is not "exited", jump to A3;

[0017] A3: Collect the work log of the current traversal container; determine whether there is a health check configuration for the task corresponding to the current traversal container; if so, perform a health check; merge the health check result with the first task state of the current traversal container to obtain the second task state of the current traversal container related to the business; if not, set the second task state of the current traversal container to "running"; add the work log and the second task state of the current traversal container to the first task state set to form the second task state set, and jump to A1;

[0018] A4: Traverse all containers recorded in the first task status set and obtain a second task status set.

[0019] Preferably, the second task status set includes: task information corresponding to all containers on the working node; the task information includes: the globally unique ID corresponding to each container, the first task status, the second task status, the work log, the task start time and the task end time.

[0020] Preferably, the health check method is:

[0021] B1: The executor requests the HTTP service port in the current traversal container and determines the status code of the HTTP interface;

[0022] B2: Determine whether the status code is within the set status code range; if it is, the health check is considered successful; if it is not within the range or the request fails, the health check is considered failed.

[0023] Preferably, when there is not only one instance of the scheduler in the cluster, the task request from the executor is forwarded to any scheduler through the load balancer.

[0024] Preferably, the method for reporting the second task status set in step S3 is: each executor calls the status reporting interface of the scheduler through the load balancer; and reports the second task status set to the scheduler through the status reporting interface.

[0025] Preferably, the method for comparing and determining the container to be cleaned in step S4 is:

[0026] S41: querying the third task status set pre-stored in the database of the working node corresponding to the second task status set and recording it as a set to be synchronized;

[0027] S42: Record the second task status set as the reporting set; compare the set to be synchronized with the reporting set to obtain a comparison result; update the task status in the set to be synchronized based on the comparison result, record the containers that only exist in the reporting set, and add the containers to the set of containers to be cleaned;

[0028] S43: When the second task state in the set to be synchronized is in the non-executing state, update the end time of the container corresponding to the second task state in the non-executing state; and add the containers in the updated set to be synchronized whose end time is greater than the set cleaning period from the current comparison time to the set of containers to be cleaned.

[0029] Preferably, the method for comparing the set to be synchronized and the reporting set in S42 is: for containers that only exist in the reporting set, add them to the set of containers to be cleaned; for containers that exist in both the reporting set and the set to be synchronized, perform a status overwriting operation; the status overwriting operation is: modifying the task status of the container in the set to be synchronized to the task status of the corresponding container in the reporting set; for containers that only exist in the set to be synchronized, modifying the second task status of the container to failure.

[0030] Preferably, the set to be synchronized is all task status information of the working node stored in the database; the task status information includes a second task status field; the second task status field is represented as a state other than "running" and is set to "non-executing state".

[0031] Preferably, the database is connected to each scheduler and stores the second task status of the container corresponding to the task assigned to the working node by each scheduler.

[0032] Preferably, the task cleaning method comprises: extracting a globally unique ID corresponding to a container in the set of containers to be cleaned; finding a corresponding container in the working node according to the globally unique ID, and clearing the container.

[0033] Beneficial effects:

[0034] The present invention proposes a method for synchronizing task status in a cluster. The executor reports the task status of all containers in the working node to the scheduler through the status reporting interface. The scheduler sends the containers that need to be cleaned to the executor through the status reporting interface, thereby realizing two-way synchronization between the executor and the scheduler. Based on this, the scheduler can obtain the real-time running status of the task and detailed log output from the executor, so that the user can clearly understand the execution progress of the submitted task, and can also use the log output to accurately diagnose problems that arise during the task operation. On the other hand, the executor can obtain the task information that needs to be deleted from the scheduler, and perform the deletion operation accordingly, thereby ensuring that the actual running task status is highly consistent with the expected status of the scheduler.

[0035] By periodically sending a request to query the status of the container to the container API in the working node, the executor can obtain the first task status of all containers in the working node in real time, avoiding the failure to discover containers with abnormal status in time; by merging the first task status of the container with the error code or health check result, the second task status corresponding to the container is obtained, and the actual running status of the task corresponding to the container can be accurately obtained, and at the same time, it can effectively make up for the limitations of relying solely on the feedback status of the container itself; through the status reporting interface, the second task status set of the working node can be reported, and the set of containers to be cleaned can be sent, avoiding the design of multiple independent interfaces, reducing interface complexity, and at the same time reducing the number of network requests and improving response speed; by batch reporting based on working nodes, compared with reporting the task status of each container individually, the number of network communications is reduced and the performance is better; by establishing a long connection based on the working node to synchronize the task status, the total number of connections is greatly reduced compared to establishing a long connection for each container separately, and the memory usage of the server is effectively reduced, which significantly improves the utilization and stability of system resources.

[0036] The scheduler compares the reported task status collection with the task status collection of the working node in the database, generates a difference update and saves it to the database, thereby updating the task status of the working node in the database; because the scheduler itself has no status, the scheduler saves the task status by connecting to the database, and synchronizes the task status of the scheduler and the executor by updating the task status in the database; the set of containers to be cleaned is obtained by comparison, and the containers are cleaned according to the set of containers to be cleaned to avoid wasting resources; only full logs are collected for containers in non-working state, saving network communication data and reducing the load pressure of each component; for the exited container, the first task status and error code of the container are merged to obtain the second task status of the container corresponding to the current business of the container, thereby effectively distinguishing between normal exit and abnormal termination of containers, and avoiding the mistaken cleaning of tasks that exit due to normal completion of business logic (such as user-initiated termination, natural end of task, etc.), ensuring that cleaning operations are only performed on tasks that are actually exited due to exceptions or failures. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a method for synchronizing task status in a cluster.

[0038] Figure 2 This is an example diagram of container cluster deployment.

[0039] Figure 3 A timing diagram of the core processing that synchronizes the task status of a container.

[0040] Figure 4 A flowchart for periodically obtaining the task status and container logs of a container for the executor.

[0041] Figure 5 A flowchart for comparing and updating the task status of containers.

[0042] Figure 6 Flowchart of task status determination for the model training container.

[0043] Figure 7 Flowchart for determining the operating status of the inference service container. DETAILED DESCRIPTION

[0044] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] Example 1:

[0046] like Figure 1 As shown in the figure, a method for synchronizing task status in a cluster includes the following steps:

[0047] S1: Query and obtain the first task status of the container: The task runs in the form of a container on each working node. The executor in each working node periodically sends a container status query request to the container API; the container API feeds back the first task status of all containers in the corresponding working node to the executor; the first task status of all containers in the working node at the current moment is aggregated to obtain a first task status set;

[0048] S2: Construct a second task state set: The executor traverses the first task state of each container in the first task state set obtained by the corresponding working node in S1; determines whether to obtain an error code and perform a health check based on the first task state, and merges the error code and health check result with the first task state to obtain a second task state; after the traversal is completed, the work logs of all containers in the first task state set, the first task state, and the second task state are aggregated into a second task state set;

[0049] S3: Reporting the second task status set: the executor reports the second task status set of the working node corresponding to the executor to the scheduler through the status reporting interface;

[0050] S4: Comparing and determining containers to be cleaned: The scheduler compares the second task state set with a third task state set pre-stored in a database for the working node corresponding to the second task state set, and obtains a set of containers to be cleaned of the working node according to the comparison result;

[0051] S5: Returning the set of containers to be cleaned: The scheduler calls the status reporting interface and sends the set of containers to be cleaned to the executor in the working node corresponding to the set of containers to be cleaned;

[0052] S6: Container cleaning: The executor cleans containers according to the set of containers to be cleaned.

[0053] Example 2:

[0054] There are multiple worker nodes in the cluster, and tasks are run on each worker node in the form of containers. Here, one container corresponds to only one task. The executor in the worker node accesses the scheduler through the load balancer. The scheduler itself cannot save the task status, and needs to use the connected database to save the status of the tasks assigned by the scheduler. The deployment of the task cluster is as follows Figure 2 As shown in the figure, the role of the load balancer is: when there are multiple scheduler instances in the system, the load balancer can forward requests from the executor to any scheduler.

[0055] The overall process of state synchronization, such as Figure 3As shown: the executor needs to periodically call the scheduler's status reporting interface to report the task status and work log; after updating the task status, the scheduler returns cleanable containers to the executor; after receiving them, the executor deletes these containers.

[0056] The following steps are involved:

[0057] 1-2: The executor requests the container API in the working node to obtain the task status set of the working node, that is, the second task status set;

[0058] Specifically, such as Figure 4 As shown, the following process is executed at the actuator timing:

[0059] Get the first task status of all containers on the current node; determine whether the first task status of the container is running. For containers whose first task status is running, collect their recent logs. If there is a health check configuration for the task corresponding to the container, perform a health check and set its second task status to whether it is ready based on the check result. For non-running containers, get the error code of the container, merge the first task status and error code of the container to obtain the second task status of the container, and collect all logs. After the traversal is completed, the set of the second task status and logs corresponding to all containers of the current node is obtained, that is, the second task status set.

[0060] The following describes the “business-related second task state” with an example.

[0061] For example, for a deep learning training platform, there are two common task types: model training tasks and inference service tasks.

[0062] For model training tasks, the running status includes "Running," "Task Successful," and "Task Failed." The success or failure of a task is determined by the error code returned when the container terminates. According to the Linux process error code convention, a return value of 0 indicates success, while a non-zero value indicates failure.

[0063] For the inference service task, the second task status of its container is "Loading", "Ready", "Stopped", and "Abnormal Stop". "Loading" means that the container has been started, but cannot provide external services because the loading is not complete. Whether the loading is successful can be determined by trying to access the health check interface of the service. "Ready" means that the container has been started and can provide external services. On the basis that the first task status of the container is "Running", it passes the health check and enters the "Ready" state. The similarity between "Stop" and "Abnormal Stop" is that the first task status of the container is Stop. The difference is that the error code of the "Abnormal Stop" container is non-zero, indicating that an error has occurred in the container; the error code of the "Stop" container is 0, indicating that the container has stopped normally without an error.

[0064] The following describes how to determine the two "business-related second task states": "model training task state" and "inference service task state".

[0065] like Figure 6 The figure shows a flowchart of the model training task status judgment.

[0066] The model training task status judgment criteria are described as follows. The number before the colon is the second task status of the model training task, and the number after the colon is the judgment criteria for this status. The status mentioned after the colon refers to the first task status of the container returned by the container API.

[0067] Running: The first task status of the container is "Running";

[0068] Task success: The first task status of the container is "exited" and the error code is 0;

[0069] Task failure: The first task status of the container is "Exited" and the error code is a non-zero value.

[0070] like Figure 7 The figure shows a flowchart of the inference service task status judgment.

[0071] The description of the judgment criteria for the inference service task status is as follows. The number before the colon is the second task status of the inference service task, and the number after the colon is the judgment criteria for this status. The status mentioned after the colon refers to the first task status of the container returned by the container API.

[0072] Loading: The first task of the container is in the running state, but the health check fails;

[0073] Ready: The first task status of the container is running and the health check is successful;

[0074] Stop: The first task status of the container is stopped, and the error code is 0;

[0075] Abnormal stop: The first task status of the container is stopped, and the error code is a non-zero value.

[0076] Regarding the implementation of health checks, the executor attempts to request an HTTP service port in the container and determines whether the health check is successful by judging the HTTP status code. A status code of 2xx (status code >= 200 and status code < 300) is considered a health check success. Other status codes or request failure are considered health check failures.

[0077] 3: The executor calls the scheduler's status reporting interface through the load balancer, and reports the second task status set containing the task status and work logs of all containers in the working node to the scheduler through the status reporting interface;

[0078] 4: The scheduler compares and updates the task status;

[0079] Specifically, the task status update process is implemented in the scheduler. The status reporting interface is in the form of an HTTP interface for the executor to call. The input of this interface is the task status set of a node reported by the executor, and the output is the container set to be cleaned. The processing flow is as follows Figure 5 As shown, the specific steps are:

[0080] 1) Query the third task status set of the working node recorded in the database;

[0081] 2) Record the reported second task status set as the "reporting set" and the third task status set of the node currently recorded in the database as the "pending synchronization set." Containers that appear only in the "reporting set" are added to the cleanable container set. Containers that appear in both the "reporting set" and the "pending synchronization set" have their status in the "pending synchronization set" overwritten with their status in the "reporting set." Containers that appear only in the "pending synchronization set" have their status changed to failed.

[0082] 3) Update the end time of all non-executing containers in the "to be synchronized set";

[0083] 4) Add the containers in the "to be synchronized set" whose end time is more than one week away from the current comparison time to the set of containers to be cleaned up;

[0084] Specifically, the cleanup strategy in step 4 cleans up containers that have been running for a week. This is because intermediate files generated during container operation are not automatically cleaned up immediately after the container ends to facilitate troubleshooting. This still occupies some disk space on the corresponding node. Therefore, in addition to manual cleanup by the user, an automatic cleanup strategy is also necessary.

[0085] 5) Output the container set to be cleaned;

[0086] 5: The scheduler returns the set of containers to be cleaned to the executor through the status reporting interface;

[0087] 6: The executor cleans up the containers in the working nodes corresponding to the container set to be cleaned according to the container set to be cleaned.

[0088] It should be noted that the specific embodiments described above can enable those skilled in the art to more fully understand the present invention, but do not limit the present invention in any way. Therefore, although this specification has described the present invention in detail with reference to the drawings and embodiments, those skilled in the art should understand that the present invention can still be modified or replaced with equivalents. In short, all technical solutions and improvements that do not depart from the spirit and scope of the present invention should be included in the scope of protection of the patent for the present invention.

Claims

1. A method for synchronizing task status in a cluster, characterized in that: S1: Query and obtain the first task status of the container: The task runs in the form of a container on each working node. The executor in each working node periodically sends a container status query request to the container API; the container API feeds back the first task status of all containers in the corresponding working node to the executor; Summarize the first task states of all containers in the working node at the current moment to obtain a first task state set; S2: Construct a second task state set: The executor traverses the first task state of each container in the first task state set obtained by the corresponding working node in S1; Determine whether to obtain an error code and perform a health check based on the first task status, and combine the error code, health check result, and the first task status to obtain a second task status; After the traversal is completed, the work logs, the first task status, and the second task status of all containers in the first task status set are aggregated into a second task status set; S3: Reporting the second task status set: the executor reports the second task status set of the working node corresponding to the executor to the scheduler through the status reporting interface; S4: Comparing and determining containers to be cleaned: The scheduler compares the second task state set with a third task state set pre-stored in a database for the working node corresponding to the second task state set, and obtains a set of containers to be cleaned of the working node according to the comparison result; S5: Returning the set of containers to be cleaned: The scheduler calls the status reporting interface and sends the set of containers to be cleaned to the executor in the working node corresponding to the set of containers to be cleaned; S6: Container cleaning: The executor cleans containers according to the set of containers to be cleaned.

2. A method for synchronizing task status in a cluster according to claim 1, characterized in that: The method for obtaining the first task status set of the container in step S1 is as follows: the executor in the working node periodically sends a container status query request to the container API in the working node; the container API feeds back the first task status of all containers in the working node to the executor according to the container status query request; Summarize the first task states of all containers in the working node at the current moment to obtain a first task state set of the working node at the current moment.

3. A method for synchronizing task status in a cluster according to claim 1, characterized in that: The steps of constructing the second task state set in step S2 are: A1: Determine whether all containers in the first task status set have been traversed. If not, jump to A2. Otherwise, jump to A4. A2: Determine the first task status of the current traversal container; if the first task status of the current traversal container is "exited", obtain the error code of the current traversal container; combine the first task status of the current traversal container and the error code to obtain the second task status of the current traversal container related to the business; Collecting the full log of the currently traversed container; adding the full log and a second task state related to the business to the first task state set to form a second task state set; After adding, jump to A1; If the first task status is not "exited", jump to A3; A3: Collect the work log of the current traversal container; determine whether there is a health check configuration for the task corresponding to the current traversal container; if so, perform a health check; merge the health check result with the first task status of the current traversal container to obtain the second task status of the current traversal container related to the business; If it does not exist, set the second task state of the current traversal container to "running"; add the work log and the second task state of the current traversal container to the first task state set to form a second task state set, and jump to A1; A4: Traverse all containers recorded in the first task status set and obtain a second task status set.

4. A method for synchronizing task status in a cluster according to claim 1 or 3, characterized in that: The second task status set includes: task information corresponding to all containers on the working node; the task information includes: the globally unique ID corresponding to each container, the first task status, the second task status, the work log, the task start time and the task end time.

5. A method for synchronizing task status in a cluster according to claim 3, characterized in that: The health check method is: B1: The executor requests the HTTP service port in the current traversal container and determines the status code of the HTTP interface; B2: Determine whether the status code is within the set status code range; if it is, the health check is considered successful; if it is not within the range or the request fails, the health check is considered failed.

6. A method for synchronizing task status in a cluster according to claim 1, characterized in that: When there is more than one scheduler instance in the cluster, the task request from the executor is forwarded to any scheduler through the load balancer.

7. A method for synchronizing task status in a cluster according to claim 1, characterized in that: The method for reporting the second task status set in step S3 is: each executor calls the status reporting interface of the scheduler through the load balancer; and reports the second task status set to the scheduler through the status reporting interface.

8. A method for synchronizing task status in a cluster according to claim 1, characterized in that: The method for comparing and determining the container to be cleaned in step S4 is as follows: S41: querying the third task status set pre-stored in the database of the working node corresponding to the second task status set and recording it as a set to be synchronized; S42: Record the second task status set as the reporting set; compare the set to be synchronized with the reporting set to obtain a comparison result; update the task status in the set to be synchronized based on the comparison result, record the containers that only exist in the reporting set, and add the containers to the set of containers to be cleaned; S43: When the second task state in the set to be synchronized is in the non-executing state, update the end time of the container corresponding to the second task state in the non-executing state; and add the containers in the updated set to be synchronized whose end time is greater than the set cleaning period from the current comparison time to the set of containers to be cleaned.

9. A method for synchronizing task status in a cluster according to claim 8, characterized in that: The method for comparing the set to be synchronized with the reported set in S42 is as follows: for containers that exist only in the reported set, add them to the set of containers to be cleaned; for containers that exist in both the reported set and the set to be synchronized, perform a state overwriting operation; the state overwriting operation is to modify the task state of the container in the set to be synchronized to the task state of the corresponding container in the reported set; For a container that only has to be synchronized, the second task status of the container is modified to failure.

10. A method for synchronizing task status in a cluster according to claim 8, characterized in that: The set to be synchronized is all task status information of the working node stored in the database; the task status information includes a second task status field; the second task status field is represented as a state other than "running" and is set to "non-executing state".

11. A method for synchronizing task status in a cluster according to claim 1, characterized in that: The database is connected to each scheduler and stores the second task status of the container corresponding to the task assigned to the working node by each scheduler.

12. A method for synchronizing task status in a cluster according to claim 1, characterized in that: The task cleaning method comprises: extracting a globally unique ID corresponding to a container in a set of containers to be cleaned; finding a corresponding container in the working node according to the globally unique ID, and clearing the container.

Images