Method for synchronizing task states in cluster

Abstract

The invention belongs to the field of computer software, and provides a method for synchronizing task states in a cluster. And through a state reporting interface, the executor reports a second task state of the working node, and the scheduler issues a container needing to be cleaned, so that bidirectional state synchronization of the executor and the scheduler is realized. The actuator regularly sends a request for querying the task state of the container to the container API in the working node, so that the situation that the container in an abnormal state cannot be found in time is avoided; the second task state is obtained by combining the first task state, the error code or the health examination result of the container, so that the limitation that only the feedback state of the container is depended on is effectively made up; the task state is reported by taking the working node as a unit, so that the network communication frequency is reduced and the performance is better than that of independently reporting the task state by each container; the task state synchronization is realized by establishing the long connection by taking the working node as the unit, compared with a mode of independently establishing the long connection for each container, the total number of connections is greatly reduced, and the memory occupation of the server is effectively reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer software, and more particularly to a method for synchronizing task states in a cluster. Background Art

[0002] Linux container technology is a lightweight virtualization technology that originated from the cgroups and namespaces features of the Linux kernel developed under the leadership of Linus Torvalds. It allows developers to package an application and its dependencies into a lightweight and portable container.

[0003] In a cluster, a scheduler distributes tasks to each executor, and the executor executes these tasks. However, the current existing technology fails to achieve two-way synchronization of task states between the scheduler and the executor. This defect has led to a large number of tasks that should have been deleted still remaining in the cluster, continuously occupying running memory resources, causing resource waste and a decline in cluster performance. On the other hand, since the scheduler cannot grasp the specific task execution status in real time, in the face of task anomalies, timeouts, etc., it is difficult to quickly and accurately perform necessary operations such as terminating and restarting tasks, seriously affecting the efficiency and flexibility of cluster task management. Summary of the Invention

[0004] In order to solve the technical problem that the task states between the scheduler and the executor in the cluster cannot be two-way synchronized in the prior art, the present invention proposes a method for synchronizing task states in a cluster, which solves the above problem.

[0005] The present invention proposes a method for synchronizing task states in a cluster, and the method includes the following steps: A method for synchronizing task states in a cluster, and the specific steps are as follows: S1: Query and obtain the first task state of the container: The tasks run in the form of containers on each worker node, and the executor in each worker node periodically sends a request to query the container status to the container API; the container API feeds back the first task state of all containers in the corresponding worker node to the executor; summarize the first task states of all containers in the worker node at the current moment to obtain a first task state set; S2: Construct a second task state set: The executor traverses the first task state of each container in the first task state set obtained in S1 for the corresponding worker node; determines whether to obtain an error code and perform a health check according to the first task state, and merges the error code, the health check result and the first task state to obtain a second task state; after the traversal is completed, summarize the working logs, the first task state and the second task state of each container in the first task state set into a second task state set; S3: Report the second task status set: The executor reports the second task status set of the working node corresponding to the executor to the scheduler through the status reporting interface; S4: Compare and determine the containers to be cleaned: The scheduler compares the second task status set with the third task status set pre-stored in the database for the working node corresponding to the second task status set, and obtains the set of containers to be cleaned for the working node according to the comparison result; S5: Return the set of containers to be cleaned: The scheduler calls the status reporting interface and sends the set of containers to be cleaned to the executor in the working node corresponding to the set of containers to be cleaned; S6: Container cleaning: The executor cleans the containers according to the set of containers to be cleaned.

[0006] Preferably, the method for obtaining the first task status set of containers in step S1 is: The executor in the working node periodically sends a request to query the container status to the container API in the working node; the container API feeds back the first task status of all containers in the working node according to the request to query the container status; summarize the first task status of all containers in the working node at the current moment to obtain the first task status set of the working node at the current moment.

[0007] Preferably, the steps for constructing the second task status set in step S2 are: A1: Determine whether all containers in the first task status set have been traversed; if not, jump to A2; otherwise, jump to A4; A2: Determine the first task status of the currently traversed container; if the first task status of the currently traversed container is "exited", obtain the error code of the currently traversed container; merge the first task status of the currently traversed container and the error code to obtain the second task status related to the business of the currently traversed container; collect the full amount of logs of the currently traversed container; add the full amount of logs and the second task status related to the business to the first task status set to form the second task status set; after adding, jump to A1; if the first task status is not "exited", then jump to A3; A3: Collect the working day logs of the currently traversed container; determine whether there is a health check configuration for the task corresponding to the currently traversed container; if so, perform a health check; merge the health check result and the first task status of the currently traversed container to obtain the second task status related to the business of the currently traversed container; if not, set the second task status of the currently traversed container to "running"; add the working day logs and the second task status of the currently traversed container to the first task status set to form the second task status set, and jump to A1; A4: Traverse all containers recorded in the first task status set and summarize to obtain the second task status set.

[0008] Preferably, the second task status set includes: task information corresponding to all containers on the worker node; the task information includes: the globally unique ID corresponding to each container, the first task status, the second task status, the work log, the task start time, and the task end time.

[0009] Preferably, the method of health check is: B1: The executor requests the HTTP service port in the currently traversed container and judges the status code of the HTTP interface; B2: Judge whether the status code is within the set status code range; if it is within the range, it is judged that the health check is successful; if it is not within the range or the request fails, it is judged that the health check fails.

[0010] Preferably, when the instances of the scheduler in the cluster are not unique, the task requests from the executor are forwarded to any scheduler through the load balancer.

[0011] Preferably, the method for the S3 step to report the second task status set is: each executor calls the status reporting interface of the scheduler through the load balancer; the second task status set is reported to the scheduler through the status reporting interface.

[0012] Preferably, the method for the S4 step to compare and determine the containers to be cleaned is: S41: Query the third task status set pre-stored in the database corresponding to the worker node of the second task status set and record it as the set to be synchronized; S42: Record the second task status set as the reported set; compare the set to be synchronized with the reported set to obtain the comparison result; update the task status in the set to be synchronized according to the comparison result, and record the containers that only exist in the reported set, and add the containers to the set of containers to be cleaned; S43: When the second task status in the set to be synchronized is a non-executing state, update the end time of the container corresponding to the non-executing state; add the containers in the updated set to be synchronized whose end time of the container is greater than the set cleaning period from the current comparison time to the set of containers to be cleaned.

[0013] Preferably, the method for comparing the set to be synchronized with the reported set in S42 is: for the containers that only exist in the reported set, add them to the set of containers to be cleaned; for the containers that exist in both the reported set and the set to be synchronized, perform a status overwrite operation; the status overwrite operation is: modify the task status of the container in the set to be synchronized to the task status of the corresponding container in the reported set; for the containers that only exist in the set to be synchronized, modify the second task status of the container to failed.

[0014] Preferably, the set of tasks to be synchronized is all task status information of the working nodes stored in the database; the task status information includes a second task status field; when the second task status field represents a status other than "running", it is set to "non-executing status".

[0015] Preferably, the database is connected to each scheduler and stores the second task status of the containers corresponding to the tasks assigned by each scheduler to the working nodes.

[0016] Preferably, the method for task cleaning is: extracting the globally unique ID corresponding to the container in the set of containers to be cleaned; finding the corresponding container in the working node according to the globally unique ID, and clearing the container.

[0017] Beneficial effects: The present invention proposes a method for synchronizing task status in a cluster. The executor reports the task status of all containers in the working node to the scheduler through the status reporting interface, and the scheduler issues the containers to be cleaned to the executor through the status reporting interface, realizing two-way synchronization between the executor and the scheduler. Based on this, the scheduler can obtain the real-time running status of the task and the detailed log output from the executor, enabling the user to clearly understand the execution progress of the submitted task and accurately diagnose problems occurring during the task running process with the help of the log output. On the other hand, the executor can obtain the task information to be deleted from the scheduler and perform the deletion operation accordingly, thus ensuring that the actual running task status is highly consistent with the expected status of the scheduler.

[0018] By the executor sending a request to query the status of the container to the container API in the working node at regular intervals, it can obtain the first task status of all containers in the working node in real time, avoiding containers with abnormal status that cannot be detected in time; by merging the first task status of the container with the error code or the health check result, the second task status corresponding to the container can be accurately obtained, enabling the actual running status of the task corresponding to the container to be obtained precisely, and at the same time effectively making up for the limitations of relying only on the status feedback of the container itself; through the status reporting interface, the reporting of the second task status set of the working node and the issuance of the set of containers to be cleaned can be completed, avoiding the design of multiple independent interfaces, reducing the interface complexity, and at the same time reducing the number of network requests and improving the response speed; by reporting in batches with the working node as the unit, compared with reporting the task status of each container separately, the number of network communications is reduced, and the performance is better; by establishing a long connection with the working node as the unit to realize task status synchronization, compared with the method of establishing a long connection for each container separately, the total number of connections is significantly reduced, and the memory occupancy of the server is effectively reduced, significantly improving the system resource utilization rate and stability.

[0019] The scheduler generates differential updates by comparing the reported set of task statuses with the set of task statuses of the worker nodes in the database, and saves them to the database, thereby updating the task statuses of the worker nodes in the database. Since the scheduler itself has no status, the scheduler saves the task status by connecting to the database and synchronizes the task statuses of the scheduler and the executor by updating the task statuses in the database. The set of containers to be cleaned is obtained by comparison, and the containers are cleaned according to the set of containers to be cleaned, avoiding waste of resources. Only full logs of non-working containers are collected, saving network communication data volume and reducing the load pressure on each component. For containers that have exited, the first task status and error code of the container are merged to obtain the second task status of the container corresponding to the current business of the container, thereby effectively distinguishing between normally exited and abnormally terminated containers, and avoiding mis-cleaning tasks that have exited due to normal completion of business logic (such as user-initiated termination, natural end of tasks, etc.), ensuring that only tasks that have exited due to exceptions or failures are cleaned up. Description of the Drawings

[0020] Figure 1 A method for synchronizing task statuses in a cluster.

[0021] Figure 2 A deployment example diagram of a container cluster.

[0022] Figure 3 A core processing timing diagram for task status synchronization of containers.

[0023] Figure 4 A flowchart for an executor to periodically obtain the task status and container logs of a container.

[0024] Figure 5 A flowchart for comparing and updating the task status of a container.

[0025] Figure 6 A flowchart for judging the task status of a model training container.

[0026] Figure 7 A flowchart for judging the running status of an inference service container. Detailed Implementation Modes

[0027] The present invention will be further described in detail below with reference to the drawings and specific implementation modes.

[0028] Example 1: As Figure 1 shown, a method for synchronizing task statuses in a cluster, the specific steps are as follows: S1: Query and obtain the first task status of the container: The tasks run in the form of containers on each worker node, and the executor in each worker node periodically sends a request to query the container status to the container API; the container API feeds back the first task status of all containers in the corresponding worker node to the executor; summarize the first task status of all containers in the worker node at the current moment to obtain the first task status set. S2: Construct the second task status set: The executor traverses the first task status of each container in the first task status set obtained in S1 for the corresponding worker node; determines whether to obtain an error code and perform a health check based on the first task status, and merges the error code, health check result with the first task status to obtain the second task status; after the traversal is completed, summarize the working logs, the first task status and the second task status of all containers in the first task status set into the second task status set. S3: Report the second task status set: The executor reports the second task status set of the corresponding worker node of the executor to the scheduler through the status reporting interface. S4: Compare and determine the containers to be cleaned: The scheduler compares the second task status set with the third task status set pre-stored in the database for the corresponding worker node of the second task status set, and obtains the set of containers to be cleaned for the worker node according to the comparison result. S5: Return the set of containers to be cleaned: The scheduler calls the status reporting interface to send the set of containers to be cleaned to the executor in the corresponding worker node of the set of containers to be cleaned. S6: Container cleaning: The executor cleans the containers according to the set of containers to be cleaned.

[0029] Embodiment 2: There are multiple worker nodes in the cluster, and the tasks run in the form of containers on each worker node. Here, only one task corresponds to one container. The executor in the worker node accesses the scheduler through the load balancer. The scheduler itself cannot save the task status and needs to save the status corresponding to the tasks assigned by the scheduler through the connected database. The deployment of the task cluster is as Figure 2 shown. Among them, the role of the load balancer: When there are multiple scheduler instances in the system, the load balancer can forward requests from the executor to any scheduler.

[0030] The overall process of status synchronization is as Figure 3 shown: The executor needs to periodically call the status reporting interface of the scheduler to report the task status and working logs; after the scheduler updates the task status, it returns the containers that can be cleaned to the executor; after receiving them, the executor deletes these containers.

[0031] It includes the following steps: 1 - 2: The executor requests the container API within the worker node to obtain the set of task statuses of the current worker node, i.e., the second task status set. Specifically, as Figure 4 shown, the executor periodically executes the following process: Obtain the first task status of all containers on the current node; determine whether the first task status of the container is "running". For containers with a first task status of "running", collect their recent logs. If the task corresponding to the container has a health check configuration, perform a health check and set its second task status to "ready" or not according to the check result. For non-running containers, obtain the error code of the container, combine the first task status and the error code of the container to obtain the second task status of the container, and collect all logs. After completing the traversal, obtain the set of the second task statuses and logs corresponding to all containers on the current node, i.e., the second task status set.

[0032] The following gives an example description of the "second task status related to the business".

[0033] For example, for a deep learning training platform, there are two common task types: model training tasks and inference service tasks.

[0034] For model training tasks, its running statuses are "running", "task successful", and "task failed". Whether the task is successful or failed needs to be judged according to the error code when the container ends. According to the convention of Linux process error codes, returning 0 represents success, and non-0 represents failure.

[0035] For inference service tasks, the second task statuses of its containers are "loading", "ready", "stopped", and "abnormally stopped". Among them, "loading" is the status where the container has started but cannot provide services externally because the loading is not yet complete. Whether the loading is successful can be determined by attempting to access the health check interface of the service. And "ready" is the status where the container has started and can provide services externally. On the basis that the first task status of the container is "running", if the health check passes, it enters the "ready" status. The common point between "stopped" and "abnormally stopped" is that the first task status of the container is "stopped". The difference is that the error code of the container in "abnormally stopped" is a non-zero value, indicating that the container has an error; the error code of the container in "stopped" is 0, indicating that the container has stopped normally without errors.

[0036] The following separately explains the determination methods of these two "second task statuses related to specific business", namely, the "model training task status" and the "inference service task status": As Figure 6 shown is the flowchart for judging the model training task status.

[0037] The description of the criteria for judging the status of the model training task is as follows. Before the colon is the second task status of the model training task, and after the colon is the judgment criterion. The status mentioned after the colon refers to the first task status of the container returned by the container API; Running: The first task status of the container is "Running"; Task Succeeded: The first task status of the container is "Exited", and the error code is 0; Task Failed: The first task status of the container is "Exited", and the error code is a non-zero value.

[0038] As Figure 7 shown is the flowchart for judging the status of the inference service task.

[0039] The description of the criteria for judging the status of the inference service task is as follows. Before the colon is the second task status of the inference service task, and after the colon is the judgment criterion. The status mentioned after the colon refers to the first task status of the container returned by the container API; Loading: The first task status of the container is Running, but the health check fails; Ready: The first task status of the container is Running, and the health check succeeds; Stopped: The first task status of the container is Stopped, and the error code is 0; Abnormally Stopped: The first task status of the container is Stopped, and the error code is a non-zero value.

[0040] Regarding the implementation method of the health check, the executor attempts to request a certain HTTP service port in the container, and judges whether the health check is successful by judging the HTTP status code. For the status code of 2xx (that is, the status code >= 200 and the status code < 300), it is judged that the health check is successful, and other status codes or request failures are judged that the health check fails.

[0041] 3: The executor calls the status reporting interface of the scheduler through the load balancer, and reports the second task status set including the task status and working day logs of all containers in the worker node to the scheduler through the status reporting interface; 4: The scheduler compares and updates the task status; Specifically, the task status update process is implemented in the scheduler. The status reporting interface is provided for the executor to call in the form of an HTTP interface. The input of this interface is the task status set of a certain node reported by the executor, and the output is the set of containers to be cleaned up. The processing process is as Figure 5 shown, and the specific steps are, 1) Query the third task status set of this worker node recorded in the database; 2) Denote the second task status set reported as the "reported set", and denote the third task status set of this node recorded in the current database as the "set to be synchronized". For the containers that only appear in the "reported set", add them to the set of containers to be cleaned up; for the containers that appear in both the "reported set" and the "set to be synchronized", overwrite their status in the "set to be synchronized" with their status in the "reported set"; for the containers that only appear in the "set to be synchronized", change their status to failed; 3) Update the end time of all containers in the "set to be synchronized" that are not in the execution status; 4) Add the containers in the "set to be synchronized" whose end time is more than one week away from the currently compared time to the set of containers to be cleaned up; Specifically, the cleaning policy in 4) is to clean up the containers that have been one week since the end of their operation. Because in order to facilitate the investigation of the intermediate files generated during the operation of the containers after they have finished running, they are not automatically cleaned up on the spot and will also occupy a certain amount of disk space on the corresponding nodes. Therefore, in addition to allowing users to clean up manually, it is also necessary to design an automatic cleaning policy.

[0042] 5) Output the set of containers to be cleaned up; 5: The scheduler returns the set of containers to be cleaned up to the executor through the status reporting interface; 6: The executor cleans up the containers in the corresponding working nodes of the set of containers to be cleaned up according to the set of containers to be cleaned up.

[0043] It should be noted that the above specific implementation manners can enable those skilled in the art to understand the present invention more comprehensively, but do not limit the present invention in any way. Therefore, although this specification has described the present invention in detail with reference to the drawings and embodiments, those skilled in the art should understand that the present invention can still be modified or equivalently replaced. In short, all technical solutions and their improvements that do not depart from the spirit and scope of the present invention should be covered by the protection scope of the patent of the present invention.

Claims

1. A method for synchronizing task status in a cluster, characterized in that: S1: Query and obtain the first task status of the container: The tasks run in the form of containers on each worker node, and the executor in each worker node regularly sends a request to query the container status to the container API; the container API feeds back the first task status of all containers in the corresponding worker node to the executor; Summarize the first task status of all containers in the worker node at the current moment to obtain a first task status set; S2: Construct a second task status set: The executor traverses the first task status of each container in the first task status set obtained in S1 for the corresponding worker node; Judge whether to obtain an error code and perform a health check according to the first task status, and merge the error code, health check result and the first task status to obtain a second task status; After the traversal is completed, summarize the working logs, the first task status and the second task status of all containers in the first task status set into a second task status set; S3: Report the second task status set: The executor reports the second task status set of the corresponding worker node of the executor to the scheduler through the status reporting interface; S4: Compare and determine the containers to be cleaned: The scheduler compares the second task status set with the third task status set pre-stored in the database for the corresponding worker node of the second task status set, and obtains the set of containers to be cleaned for the worker node according to the comparison result; S5: Return the set of containers to be cleaned: The scheduler calls the status reporting interface and distributes the set of containers to be cleaned to the executor in the corresponding worker node of the set of containers to be cleaned; S6: Container cleaning: The executor cleans the containers according to the set of containers to be cleaned.

2. A method for synchronizing task status in a cluster according to claim 1, characterized in that, The method for obtaining the first task status set of the container in step S1 is: The executor in the worker node regularly sends a request to query the container status to the container API in the worker node; the container API feeds back the first task status of all containers in the worker node to the executor according to the request to query the container status; Summarize the first task status of all containers in the worker node at the current moment to obtain the first task status set of the worker node at the current moment.

3. A method for synchronizing task status in a cluster according to claim 1, characterized in that, The steps of constructing the second task status set in step S2 are: A1: Judge whether all containers in the first task status set have been traversed; if not, jump to A2; otherwise, jump to A4; A2: Judge the first task status of the currently traversed container; if the first task status of the currently traversed container is "exited", obtain the error code of the currently traversed container; merge the first task status of the currently traversed container and the error code to obtain the second task status related to the business of the currently traversed container; Collect the full amount of logs of the currently traversed container; add the full amount of logs and the second task status related to the business to the first task status set to form a second task status set; After adding, jump to A1; If the first task status is not "exited", then jump to A3; A3: Collect the working logs of the currently traversed container; determine whether there is a health check configuration for the task corresponding to the currently traversed container; if so, perform a health check; merge the health check result and the first task status of the currently traversed container to obtain the second task status related to the business of the currently traversed container. If not, set the second task status of the currently traversed container to "Running"; add the working logs and the second task status of the currently traversed container to the first task status set to form a second task status set, and jump to A1. A4: Traverse all the containers recorded in the first task status set and summarize to obtain a second task status set.

4. A method for synchronizing task status in a cluster according to claim 1 or 3, characterized in that, The second task status set includes: task information corresponding to all the containers on the working node; the task information includes: the globally unique ID, the first task status, the second task status, the working logs, the task start time, and the task end time corresponding to each container.

5. A method for synchronizing task states in a cluster according to claim 3, characterized in that, The method of the health check is as follows: B1: The executor requests the HTTP service port in the currently traversed container and judges the status code of the HTTP interface. B2: Judge whether the status code is within the set status code range; if it is within the range, judge that the health check is successful; if it is not within the range or the request fails, judge that the health check fails.

6. A method for synchronizing task states in a cluster according to claim 1, characterized in that, When the instances of the scheduler in the cluster are not unique, forward the task requests from the executor to any scheduler through the load balancer.

7. A method for synchronizing task states in a cluster according to claim 1, characterized in that, The method for reporting the second task status set in step S3 is: each executor calls the status reporting interface of the scheduler through the load balancer; report the second task status set to the scheduler through the status reporting interface.

8. A method for synchronizing task states in a cluster according to claim 1, characterized in that, The method for comparing and determining the containers to be cleaned in step S4 is as follows: S41: Query the pre-stored third task status set in the database corresponding to the working node of the second task status set and record it as the set to be synchronized. S42: Record the second task status set as the reported set; compare the set to be synchronized with the reported set to obtain a comparison result; update the task status in the set to be synchronized according to the comparison result, and record the containers that only exist in the reported set, and add the containers to the set of containers to be cleaned. S43: When the second task status in the set to be synchronized is a non-executing state, update the end time of the container corresponding to the non-executing state; add the containers in the updated set to be synchronized whose end time of the container is greater than the set cleaning period from the current comparison time to the set of containers to be cleaned.

9. A method for synchronizing task status in a cluster according to claim 8, characterized in that, The method for comparing the set to be synchronized with the reported set in S42 is: for the containers that only exist in the reported set, add them to the set of containers to be cleaned; for the containers that exist in both the reported set and the set to be synchronized, perform a status overwrite operation; the status overwrite operation is: modify the task status of the container in the set to be synchronized to the task status of the corresponding container in the reported set. For the containers that only exist in the set to be synchronized, modify the second task status of the container to failed.

10. A method for synchronizing task states in a cluster according to claim 8, characterized in that, The set to be synchronized is all task status information of the working nodes stored in the database; the task status information includes a second task status field; setting the second task status field to a status other than "running" as the "non-executing status".

11. A method for synchronizing task states in a cluster according to claim 1, characterized in that, The database is connected to each scheduler and stores the second task status of the containers corresponding to the tasks assigned by each scheduler to the working nodes.

12. A method for synchronizing task status in a cluster according to claim 1, characterized in that, The method of task cleaning is: extracting the globally unique ID corresponding to the container in the set of containers to be cleaned; finding the corresponding container in the working node according to the globally unique ID, and clearing the container.

Images