Job recovery method and computing device

Abstract

The embodiment of the application discloses a kind of job recovery method and computing device, belong to computing cluster technical field.Cluster computing system can automatically restore job, shorten job recovery time, to improve job operation efficiency.Cluster computing system includes management node and at least one computing node, and at least one job is run on the computing node of cluster computing system;The method comprises the following steps: in the case where first computing node monitors that first job stops running, first computing node sends the stop running reason of first job to management node, and first computing node is the computing node that stops running first job in cluster computing system currently;Management node determines whether first job satisfies self-recovery condition based on stop running reason, if first job satisfies self-recovery condition, first job is redistributed to the computing node in cluster computing system, and self-recovery condition is used to indicate that job can be restored and run.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computing cluster, and particularly relates to a job recovery method and a computing device. BACKGROUND

[0002] In the field of high-performance computing (HPC), many scientific problems need to be solved and simulation tasks are large in scale, and resources of multiple computing nodes need to be used to complete the tasks. In a large-scale HPC cluster, a scheduler is usually used to uniformly allocate resources and manage nodes. In order to improve the fault tolerance of the cluster computing system, the scheduler usually integrates checkpoint / restart (C / R) technology, which can restore the running state of a job that exits due to a fault, and avoid the loss caused by re-running.

[0003] At present, a user can submit a job, create a checkpoint, and recover the job by using a command line tool of the scheduler. After a management node of the cluster computing system receives a job processing request submitted by the user, and generates a resource scheduling result of the job, the management node can send the resource scheduling result to a computing node, so that the computing node runs the job in parallel according to the computing node indicated in the resource scheduling result. After the computing node runs the job until the job exits, the user can select whether to restore the running state of the job.

[0004] In the related technology, the user needs to monitor whether the running job has stopped running, and manually judge whether the running job needs to be restored when the user monitors that the running job has stopped running. In the case where the running job needs to be restored, the user needs to re-submit a request for restoring the running job, which is relatively complicated for the user to operate, and to a certain extent, affects the efficiency of restoring the running job. SUMMARY

[0005] Embodiments of the present application provide a job recovery method and a computing device. The cluster computing system can monitor abnormal exit of a job by itself, and does not need user intervention. The cluster computing system can automatically recover the job, which shortens the job recovery time, and improves the efficiency of running the job.

[0006] In a first aspect, the present application provides a job recovery method applied to a cluster computing system, the cluster computing system comprising a management node and at least one computing node, and at least one job running on the at least one computing node of the cluster computing system; the method comprising: in the case that a first computing node monitors that a first job stops running, the first computing node sending a stop running reason of the first job to the management node, the first computing node being a computing node in the cluster computing system that currently stops running the first job; the management node determining whether the first job satisfies a self-recovery condition based on the stop running reason, and if the first job satisfies the self-recovery condition, reassigning the first job to a computing node in the cluster computing system, the self-recovery condition being used to represent that the job can be recovered and run.

[0007] It can be understood that in the method, the management node in the cluster computing system can receive the stop running reason of the first job sent by the first computing node, and after comparing with the self-recovery condition, it can be determined that in the case that the stop running reason of the first job satisfies the self-recovery condition, the management node can reassign a computing node to run the first job, so as to realize self-recovery of the first job. Since the cluster computing system can monitor abnormal exit of the job by itself and does not need user intervention, the cluster computing system can automatically recover the job, shorten the job recovery time, and thus improve the job running efficiency.

[0008] In a possible implementation, the management node determines whether the first job satisfies the self-recovery condition based on the stop running reason, comprising: if the self-recovery condition is that the job stops running due to a specified reason, the management node compares whether the stop running reason of the first job is the specified reason.

[0009] It can be understood that by judging whether the stop running reason of the first job is the specified reason, it can be determined whether the first job satisfies the self-recovery condition, which facilitates judging whether the first job needs to be self-recovered, and improves the self-recovery judgment efficiency. In a possible implementation, the stop running reason is represented by an exit code, and the exit code is an identifier corresponding to the stop running reason.

[0010] It can be understood that in order to quickly determine whether the stop running reason satisfies the self-recovery condition, the management node can receive the exit code returned by the computing node, and thus improve the self-recovery judgment efficiency.

[0011] In a possible implementation, the management node determines whether the first job satisfies the self-recovery condition based on the stop running reason, comprising: if the stop running reason is represented by an exit code, the management node compares the exit code with an exit white list to determine whether the first job satisfies the self-recovery condition; the exit white list is stored in the management node and comprises exit codes of jobs that do not support recovery and continuous running.

[0012] It can be understood that, since the number of stop running reasons that do not need self-recovery is far less than the number of stop running reasons that need self-recovery, an exit code whitelist can be set, and after it is determined that the received exit code is not on the exit code whitelist, it can be quickly determined that the job needs to be self-recovered, thereby improving the efficiency of self-recovery determination.

[0013] In a possible implementation, if the first job meets the self-recovery condition, the first job is reassigned to a computing node in the cluster computing system, including: if the exit code of the first job is not in the exit whitelist, the first job is reassigned to a computing node in the cluster computing system.

[0014] It can be understood that, since the number of stop running reasons that do not need self-recovery is far less than the number of stop running reasons that need self-recovery, an exit code whitelist can be set, and after it is determined that the received exit code is not on the exit code whitelist, it can be quickly determined that the job needs to be self-recovered, the first job is reassigned to a computing node in the cluster computing system, thereby improving the efficiency of self-recovery determination.

[0015] In a possible implementation, if the first job meets the self-recovery condition, the first job is reassigned to a computing node in the cluster computing system, including: if the stop running reason of the first job meets the self-recovery condition and the number of times of resuming the first job to continue running is less than or equal to a specified number of times, the first job is reassigned to a computing node in the cluster computing system.

[0016] It can be understood that, by limiting the number of times of self-recovery of the job, the waste of computing node resources caused by frequent self-recovery of the same job due to problems of the job itself can be avoided, and manual checking of whether the job itself has a fault after the running state is changed is facilitated.

[0017] In a possible implementation, the specified number of times is a parameter stored in the management node according to the received processing job request, and the specified number of times is used to indicate the maximum number of times of supporting self-recovery and continuous running of the job, and the processing job request is used to indicate that the management node allocates a computing node for the first job.

[0018] It can be understood that, the specified number of times can be a parameter stored in the management node when the processing job request is submitted, and subsequent determination of whether the number of self-recovery times of the first job exceeds the specified number of times is facilitated, thereby avoiding the waste of computing node resources caused by frequent self-recovery of the job.

[0019] In a possible implementation, the method further includes: if the stop running reason does not meet the self-recovery condition, the management node updates the state of the first job to a completed state, and the job in the completed state does not support self-recovery and continuous running.

[0020] It can be understood that the state of the first job whose stop running reason does not meet the self-recovery condition can be updated to the completed state, and the management node can be avoided from judging again whether the job meets the self-recovery condition, thereby avoiding waste of resources of the management node to a certain extent.

[0021] In a possible implementation, the re-allocating the first job to the computing node in the cluster computing system includes: sending a message recovery request to a second computing node in the cluster computing system, so that the first job continues to run on the second computing node, the second computing node being the computing node running the first job that is rescheduled by the management node, and the message recovery request being used for recovering the running of the first job.

[0022] It can be understood that in the case where the stop running reason of the first job meets the self-recovery condition, the message recovery request can be actively sent to the second computing node, so that the first job continues to run on the second computing node, and the running of the recovery job does not need the user to access, thereby shortening the job recovery time.

[0023] In a possible implementation, the first computing node monitors the running state of the first job, and if the first computing node monitors that the first job stops running, sends a stop running reason of the first job to a management node in the cluster computing system, and in the case where the stop running reason meets the self-recovery condition, triggers the management node to re-allocate the first job to a computing node in the cluster computing system.

[0024] It can be understood that the computing node running the first job in the cluster computing system can monitor the running state of the first job, and when the first job stops running, the computing node can send a stop running reason of the first job to a management node in the cluster computing system, so that the management node can compare the stop running reason of the first job sent by the computing node with the self-recovery condition, and in the case where the self-recovery condition is met, the management node can re-allocate the computing node to run the first job, thereby realizing self-recovery of the running of the first job. Since the cluster computing system can monitor abnormal exit of the job by itself and does not need the user to intervene, the cluster computing system can automatically recover the job, thereby shortening the job recovery time and improving the job running efficiency.

[0025] In a possible implementation, if the first computing node monitors that the first job stops running, the first computing node sends a stop running reason of the first job to a management node in the cluster computing system, including: if the stop running reason is represented by an exit code, the first computing node acquires the exit code corresponding to the stop running when monitoring that the first job stops running; and the first computing node sends the exit code to the management node in the cluster computing system.

[0026] It can be understood that, in order to enable the management node to quickly determine whether the stop running reason meets the self-recovery condition, the computing node can obtain an exit code to represent the stop running reason, and send the exit code to the management node, so as to improve the efficiency of self-recovery judgment of the management node.

[0027] In a second aspect, the present application provides a job recovery device, which is used to execute any one of the job recovery methods provided in the first aspect.

[0028] In a possible implementation, the job recovery device can be divided into functional modules according to the method provided in the first aspect. For example, each functional module can be divided according to each function, or two or more functions can be inherited in one processing module. For example, the job recovery device can be divided into a receiving module, a processing module, and the like according to functions. The possible technical solutions and beneficial effects of each functional module described above can be referred to the technical solutions provided in the first aspect or the corresponding possible implementation, which will not be described here.

[0029] In a third aspect, the present application provides a job recovery device, which is used to execute any one of the job recovery methods provided in the first aspect.

[0030] In a possible implementation, the job recovery device can be divided into functional modules according to the method provided in the second aspect. For example, each functional module can be divided according to each function, or two or more functions can be inherited in one processing module. For example, the job recovery device can be divided into a monitoring module, a sending module, and the like according to functions. The possible technical solutions and beneficial effects of each functional module described above can be referred to the technical solutions provided in the first aspect or the corresponding possible implementation, which will not be described here.

[0031] In a fourth aspect, the present application provides a computing device, which includes a processor and a memory. The memory is used to store computer program instructions, which are loaded and executed by the processor to enable the computing device to implement the job recovery method described in the above aspect.

[0032] In a fifth aspect, the present application provides a computer readable storage medium, which stores at least one computer program instruction. The computer program instruction is loaded and executed by a processor to implement the job recovery method described in the above aspect.

[0033] In a sixth aspect, an embodiment of the present application provides a computer program product, which comprises computer instructions stored in a computer readable storage medium. A processor of a computing device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computing device to perform the job recovery method provided in various optional implementation manners of the first aspect.

[0034] The detailed description of the second aspect to the sixth aspect and various implementation manners thereof in the present application can refer to the detailed description in the first aspect and various implementation manners thereof; and the beneficial effects of the second aspect to the sixth aspect and various implementation manners thereof can refer to the beneficial effect analysis in the first aspect and various implementation manners thereof, which will not be described here again.

[0035] These aspects or other aspects of the present application will be more apparent in the following description. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a framework schematic diagram of a job recovery system according to an exemplary embodiment;

[0037] Figure 2 is a flow schematic diagram of a job recovery method according to an exemplary embodiment;

[0038] Figure 3 is a flow schematic diagram of a job recovery method according to an exemplary embodiment;

[0039] Figure 4 is Figure 3 a flow schematic diagram of a job self-recovery method involved in the embodiment shown;

[0040] Figure 5 is Figure 3 a framework flow schematic diagram of a job recovery method involved in the embodiment shown;

[0041] Figure 6 is a structural schematic diagram of a job recovery apparatus provided by an exemplary embodiment of the present application;

[0042] Figure 7 is a structural schematic diagram of a job recovery apparatus provided by an exemplary embodiment of the present application;

[0043] Figure 8 is a structural schematic diagram of a computing device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present application clearer, the following will further describe the embodiments of the present application in combination with the drawings.

[0045] The "multiple" mentioned in the present text refers to two or more. The "and / or" describes the association between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents that the associated objects before and after are in an "or" relationship.

[0046] Please refer to Figure 1 , Figure 1 is a framework diagram of a job recovery system to which the embodiments of the present application relate. The job recovery system is applied in a computing cluster system, and the computing cluster system includes a management node 100, a computing node 200 and a storage system 300.

[0047] The management node 100 can include a central processing unit (CPU) for running a scheduler master process, which can be used to allocate resources for running a job, that is, to allocate computing nodes for running a job. The computing node 200 can also include a CPU for running an agent process, which can be used to communicate with the scheduler master process and can receive a job message for running a job sent by the management node 100 through the scheduler master process, and can also send a stop running reason for stopping running a job to the scheduler master process of the management node 100. The storage system 300 is in communication connection with each computing node 200 and is used to store a checkpoint image file created by each computing node 200 in the process of running a job.

[0048] Among them, the job can be the total work required by the user in a submitted computing process to be done by the cluster computing system. The checkpoint image file is generated at the checkpoint time and is used to restore the running state of the job at the checkpoint time.

[0049] Optionally, the computing node 200 includes a computing master node and a computing sub-node, the scheduler master process on the management node 100 schedules the computing node resources and sends a job message to the agent process of the corresponding computing master node, the agent process of the computing master node starts a job monitoring process after receiving the job message sent by the management node 100, monitors the job state, and generates a host file (hostfile) according to the scheduling result contained in the resource job message, for example, the resource scheduling result contained in the job message can be computing node 1 and computing node 2, and the generated host file can contain computing node 1 and computing node 2, then the computing master node can pull up the local job process by using the host file through the mpirun command, and pull up the job process on the computing sub-node indicated in the scheduling result through the secure shell (SSH) protocol, and the job processes can communicate with each other through the message passing interface (MPI). Each computing node 200 can periodically create a checkpoint mirror file during the running of the job and store it in the storage system 300, when the monitoring job state determines that the job stops running, the stopping running reason of the job is obtained, and the stopping running reason is sent to the master process on the management node 100, the master process of the management node 100 can determine whether the stopping running reason meets the self-recovery condition, in the case that the stopping running reason meets the self-recovery condition, the job can be triggered to be re-allocated and restored to the running state.

[0050] The management node 100 can be a computing device, and the computing node can also be a computing device. The computing device can be a smart phone, a tablet computer, an electronic book reader, a personal portable computer, a server, etc. The storage system 300 can be a storage device, and the storage device can include a database, a memory, etc.

[0051] Optionally, the management node 100 can also include a memory, when the management node 100 receives the submitted job, the management node 100 can create an identity (ID) of the job, and store the maximum self-recovery number and the checkpoint information of the job in the memory based on the job ID, for subsequent reading when judging whether to perform job self-recovery and re-allocate and restore the job running.

[0052] Figure 2 A flowchart of a job recovery method provided by an example embodiment of the present application is shown. The job recovery method includes the following steps:

[0053] S101, the computing node monitors the running state of the first job.

[0054] In the embodiments of the present application, after receiving the job message of the first job sent by the management node, the computing node in the cluster computing system can pull up other computing nodes to run the first job in parallel according to the scheduling result included in the job message, and each computing node running the first job can monitor the running state of the first job.

[0055] The running state of the first job can include a normal running state and a stop running state.

[0056] In a possible implementation, the computing node can detect the running state of the first job running therein according to a specified period.

[0057] For example, if the computing device runs the job 1, the running state of the job 1 can be obtained every specified time interval or period.

[0058] Alternatively, if the monitoring process of the computing node monitors that the first job running on the computing node stops running, the job process of the computing node can notify the computing node that the running state of the first job becomes the stop running state.

[0059] S102, if the first computing node monitors that the first job stops running, the first computing node sends the stop running reason of the first job to the management node in the cluster computing system.

[0060] In the embodiments of the present application, if it is detected that the first job running in the computing node is in the stop running state, the reason for the first job to exit running can be obtained as the stop running reason, and the stop running reason of the first job is sent to the management node in the cluster computing system, and the first computing node is the current computing node in the cluster computing system that stops running the first job.

[0061] In a possible implementation, if the stop running reason is represented by an exit code, the exit code corresponding to the stop running is obtained in the case of monitoring that the first job stops running, and the computing node sends the exit code to the management node in the cluster computing system.

[0062] That is, the exit code corresponding to the stop running reason can be generated after the first job stops running, the computing node can obtain the exit code, and the exit code of the first job is sent to the management node.

[0063] Optionally, the stop running reason can also be represented by exit information, and the exit information can include an exit code and an exit message, wherein the exit message can be a message for describing the meaning of the exit code.

[0064] In a possible implementation, the exit code can be divided into a first exit code and a second exit code according to whether the indicated stop running reason supports job recovery running.

[0065] The first exit code indicates a stop running reason that supports resuming the job, and the second exit code indicates a stop running reason that does not support resuming the job.

[0066] For example, the monitoring process of the computing node can collect the exit information of the first job and transmit the exit information to the agent process of the computing node, and the agent process can send a job exit message containing the exit information to the master process of the management node.

[0067] S103, the management node receives the stop running reason of the first job sent by the first computing node.

[0068] In the embodiments of the present application, the management node can receive the stop running reason of the first job.

[0069] The first computing node can be a computing node that stops running the first job in the cluster computing system.

[0070] S104, the management node determines whether the first job meets a self-recovery condition based on the stop running reason, and if the stop running reason meets the self-recovery condition, the management node reassigns the first job to a computing node in the cluster computing system.

[0071] In the embodiments of the present application, the management node can determine whether the stop running reason of the first job meets the self-recovery condition according to the obtained stop running reason of the first job, and if it is determined that the stop running reason meets the self-recovery condition, the management node can reassign the first job to a computing node in the cluster computing system.

[0072] In a possible implementation, the stop running reason of the first job can be a first reason or a second reason, the first reason being used to indicate that the first job can be resumed, and the second reason being used to indicate that the first job cannot be resumed.

[0073] That is, after obtaining the stop running reason of the first job, the management node can determine whether the stop running reason of the first job is the first reason, and if the stop running reason is the first reason, it can be determined that the first job meets the self-recovery condition; if the stop running reason is the second reason, it can be determined that the first job does not meet the self-recovery condition.

[0074] In a possible implementation, the self-recovery condition can be that the stop running reason of the job is a specified reason, and the specified reason can be stored in the management node.

[0075] The stop running reason of the first job can be represented by an exit code, which is an identifier corresponding to the stop running reason. If the stop running reason is the first reason, the exit code representing the first reason can be a first exit code. If the stop running reason is the second reason, the exit code representing the second reason can be a second exit code.

[0076] In a possible implementation, if the stop running reason is represented by an exit code, the exit code is compared with an exit white list to determine whether the first job meets the self-recovery condition. The exit white list is stored in the management node and includes exit codes of jobs that do not support resuming the running of the jobs. If the exit code of the first job is not in the exit white list, the first job is reassigned to a computing node in the cluster computing system.

[0077] For example, if the exit code of the first job received by the management node is 126 or 127, since the exit code 126 and the exit code 127 are included in the exit white list, the management node determines that the first job does not meet the self-recovery condition, that is, the first job in the stop running state cannot be reassigned to resume the running by triggering the self-recovery process. The exit code 126 can indicate that the command is not executable, and the exit code 127 can indicate that the command is not found.

[0078] In another possible implementation, the first job is reassigned to a computing node in the cluster computing system in a case where the stop running reason of the first job meets the self-recovery condition and the number of times of resuming the running of the first job is less than or equal to a specified number of times.

[0079] The maximum self-recovery number of times is stored in the management node. In a case where the management node determines that the stop running reason of the first job supports resuming the running of the job, that is, the stop running reason of the first job is a specified reason that supports resuming the running of the job, the number of times of performing the self-recovery process by the first job and the maximum self-recovery number of times of the first job stored in the management node are obtained, that is, the number of times of resuming the running of the job by the management node and the maximum self-recovery number of times of the first job are obtained, and it is determined whether the number of times of resuming the running of the job is less than the maximum self-recovery number of times. In a case where the number of times of resuming the running of the first job is less than or equal to a specified number of times, the first job is reassigned to a computing node in the cluster computing system.

[0080] For example, if the self-recovery number of times of the first job does not exceed the maximum self-recovery number of times, that is, the self-recovery number of times of the job is three times at most, and more than three times can execute the first job to be updated to a failure state.

[0081] Figure 3A flowchart of a job recovery method provided by an example embodiment of the present application is shown. The job recovery method can be mainly executed by the management node interacting with the computing node, and can include the following steps:

[0082] S201, the management node receives a job processing request of a first job, and stores job information of the first job.

[0083] In the embodiment of the present application, the management node can receive a job processing request of a first job, the job processing request being used for allocating a computing node to the first job, and the job information can be an identification of the first job, a maximum self-recovery number and checkpoint information included in the job processing request.

[0084] The checkpoint information can be used for recovering the running state of the job in the case that the job stops running, and the checkpoint information can include a checkpoint period, a storage path of a checkpoint mirror file and an opening method of the checkpoint mirror file; the job processing request can be a request submitted by a user through a submission machine for allocating a computing node to the job.

[0085] In a possible implementation, the job processing request can also be used for starting a self-recovery function, and if the received job processing request is used for starting the self-recovery function, the first job corresponding to the job processing request can subsequently be self-recovered and run in the case that a self-recovery condition is met. The specified number is a parameter stored in the management node according to the received processing job request, and the maximum self-recovery number, i.e., the specified number, is used for indicating the maximum number of times that the job supports self-recovery and continues to run, and the processing job request is used for indicating that the management node allocates a computing node to the first job.

[0086] S202, the management node generates a scheduling result.

[0087] In the embodiment of the present application, the management node can schedule the computing node meeting the resource requirement as the scheduling result after obtaining the resource requirement of the first job.

[0088] That is, the management node can obtain that the preset running job needs to schedule the computing node 1 and the computing node 2 to jointly run the first job.

[0089] S203, the management node sends a job message to the computing node indicated by the scheduling result.

[0090] In the embodiment of the present application, the management node can send the job message to the computing node indicated by the scheduling result, so that the computing node indicated by the scheduling result runs the job.

[0091] The job message can include a scheduling result, the management node sends the job message to a computing master node in a computing node indicated by the scheduling result, the computing master node generates a hostfile containing the indicated computing node according to the scheduling result in the job message, and the hostfile is used to start a local job process and a job process on other computing sub-nodes by an mpirun command, so as to achieve the purpose of running the job in parallel.

[0092] In a possible implementation, at least one checkpoint mirror file of the job can be created periodically during running of the job on the computing node, and the created checkpoint mirror file is stored in the storage system.

[0093] The period of creating the checkpoint mirror file can be a checkpoint period in the job information, the checkpoint mirror file can also be stored in the storage system according to a storage path in the job information, and the creation mode of the checkpoint mirror file can also be determined according to an opening mode of the checkpoint mirror file contained in the job information.

[0094] S204, the first computing node in the cluster computing system monitors a running state of the first job.

[0095] In the embodiment of the application, the first computing node in the cluster computing system can monitor the running state of the first job according to a specified period.

[0096] In a possible implementation, the first computing node in the cluster computing system monitors the running state of the first job by a monitoring process.

[0097] The possible embodiments of monitoring the running state of the first job also include those shown in S101, which will not be repeated here.

[0098] S205, the first computing node in the cluster computing system monitors a running state of the first job.

[0099] In the embodiment of the application, the computing node can monitor the running state of the first job periodically, and can notify the first computing node of the stopping running reason of the first job after the first job stops running.

[0100] The possible embodiments of determining the stopping running reason also include those shown in S102, which will not be repeated here.

[0101] S206, the management node receives the stopping running reason of the first job sent by the first computing node.

[0102] In the embodiment of the application, the management node can receive the job exit message sent by the first computing node, and the job exit message can include the stopping running reason of the first job.

[0103] S207, if the stop running reason meets the self-recovery condition, the management node reassigns the first job to a computing node in the cluster computing system.

[0104] In the embodiments of the present application, after receiving the stop running reason, the management node can compare the stop running reason with the self-recovery condition. If the stop running reason meets the self-recovery condition, the management node can reassign the first job to a computing node in the cluster computing system.

[0105] The job self-recovery module in the management node can reassign the first job to a computing node in the cluster computing system by returning a job recovery request to the job management module.

[0106] In another possible implementation, if the stop running reason does not meet the self-recovery condition, the state of the first job is updated to a completed state, and the job in the completed state does not support self-recovery to continue running.

[0107] For example, when the job runs successfully (exit code is 0) or the job fails to run but the self-recovery function is not started, the job completion state can be directly updated.

[0108] In one possible implementation, if the stop running reason meets the self-recovery condition, the management node can determine a second computing node in the cluster computing system to reassign to run the first job, and send a message recovery request to the second computing node. The message recovery request is used to make the second computing node read the checkpoint mirror image file to recover the process of the first job.

[0109] The job recovery request can include checkpoint information, which is used to make the second computing node successfully read the checkpoint mirror image file.

[0110] S208, the computing node reads the checkpoint mirror image file of the first job to recover the job process of the first job.

[0111] In the embodiments of the present application, the re-assigned computing nodes can each read the checkpoint mirror image file of the first job to recover the running process of the first job.

[0112] The re-assigned computing node can be the same computing node as the first computing node, or can be a different computing node from the first computing node.

[0113] For example, Figure 4 is a flowchart of a job self-recovery method according to an embodiment of the present application. As shown in Figure 4As shown, the user submits a job to the scheduler master process of the management node through the submission machine (S11), the request processing module in the scheduler master process receives the job processing request, and the job management module is passed the job information included in the job processing request (S12), the job management module assigns a job ID, and records the job information (S13), determines whether the self-recovery function is enabled when the job is submitted (S14), if the self-recovery function is enabled, the job management module passes the checkpoint information to the job self-recovery module, the job self-recovery module records the checkpoint information (S19), if the self-recovery function is not enabled or the job self-recovery module completes the recording of the checkpoint information after the self-recovery function is enabled, the job management module schedules resources and sends a job message to the scheduled computing node (S15), the agent process of the computing node pulls up the job and the monitoring process (S16), the monitoring process of the computing node monitors the running of the job (S17), determines whether the monitored job exits (S18), if it is determined that the job does not exit, the monitoring of the job continues, if it is determined that the job exits, a job status message is sent to the master process of the management node (S20), the master process determines whether the job runs successfully, i.e., whether the job normally runs to completion (S21), if it is determined that the job runs successfully, the job completion status is updated, i.e., the status of the job can be updated to running successfully (S27), if it is determined that the job does not run successfully, it is determined whether the job is a job with the job self-recovery function enabled (S22), if it is determined that the job does not have the job self-recovery function enabled, the completion status of the job is updated, i.e., the status can be updated to running failure (S27), if it is determined that the job has the job self-recovery function enabled, the exit code of the abnormal exit of the job is passed to the job self-recovery module (S23), the exit code of the job is determined as a self-recovery strategy (S24), it is determined whether the job meets the self-recovery condition (S25), if not, the job completion status is updated, i.e., the status can be updated to running failure (S27), if the self-recovery condition is met, the master process can determine the re-allocated computing node by scheduling resources through the scheduling engine, send a job recovery message to the re-allocated computing node (S26), and then continue to perform S16 until the job updates the job completion status, i.e., updates to running successfully or running failure.

[0114] The scheduler master process of the management node can be divided into a request processing module, a job management module, a scheduling engine, a message transceiver module, and a job self-recovery module according to functions, and the above modules can realize the job recovery process. Figure 5 is a framework flow diagram of a job recovery method according to an embodiment of the present application. As shown in Figure 5As shown, the processor in the management node 100 can run a scheduler main process, which can include a request processing module, a job management module, a scheduling engine, a message transceiver module, and a job self-recovery module, etc.

[0115] The request processing module is configured to receive a job processing request sent by a user through a submission machine, and send job information to the job management module. The job management module is configured to manage each job and interact with the job self-recovery module, send job information to the job self-recovery module, and receive a job recovery request or a job state update request returned by the job self-recovery module. The scheduling engine is configured to interact with the job management module to implement scheduling of the computing nodes in the cluster computing system. The message transceiver module is configured to implement information transceiving with each computing node in the cluster computing system. The job self-recovery module is configured to determine whether a job performs self-recovery according to the scheme shown in the embodiments of the present application. The job self-recovery module can mainly interact with the job management module, and the specific interaction process includes: 1) The management node receives and stores job information. Specifically, after the submission of a job with the self-recovery function enabled is completed, the job management module can pass the job information to the job self-recovery module. The job information can include a job ID, a maximum number of self-recovery, and checkpoint information, which can include a checkpoint period, a mirror path, and a checkpoint method. The job self-recovery module can store the job information in a mapping manner. 2) A judgment process of whether the self-recovery of a first job in the cluster computing system is triggered. Specifically, the monitoring process of the computing node collects job exit information (including an exit code and an exit message) and passes it to the agent process. The agent process sends the job exit message to the scheduler main process through the message transceiver module. If the job has the self-recovery function enabled, the job management module passes the message to the job self-recovery module. The job self-recovery module judges according to the self-recovery strategy (whether the self-recovery condition is met) and returns a job recovery request or a job state update request. On the one hand, the job self-recovery module can be inherited from the current mainstream scheduler framework, without affecting the original job issuing and updating process. The self-recovery strategy can be flexibly configured according to needs. On the other hand, the job self-recovery module can be embedded in a scheduling framework with integrated checkpoint function, to implement job failure self-recovery, improve the HPC job disaster recovery capability, and shorten the job failure recovery time.

[0116] The above describes the scheme of the embodiments of the present application mainly from the method aspect. It can be understood that the job resuming apparatus comprises at least one of the hardware structure and the software module for implementing the respective functions in order to implement the above functions. Those skilled in the art should easily realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented in the form of hardware or the combination of hardware and computer software. Whether a certain function is implemented in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical scheme. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0117] The embodiments of the present application can divide the functional units of the job resuming apparatus according to the above method examples. For example, each functional unit can be divided according to each function, or two or more functions can be integrated in one processing unit. The integrated unit can be implemented in the form of hardware or software functional unit. It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical functional division. Actual implementation can have another division manner.

[0118] An exemplary, Figure 6 A structure diagram of the job resuming apparatus 400 provided by an exemplary embodiment of the present application is shown. The job resuming apparatus 400 is applied to a management node. The job resuming apparatus 400 comprises:

[0119] The receiving module 410 is configured to receive a stop running reason of a first job sent by a first computing node, the first computing node being a computing node in the cluster computing system that currently stops running the first job.

[0120] The processing module 420 is configured to, in a case where the stop running reason satisfies a self-recovery condition, reassign the first job to a computing node in the cluster computing system, the self-recovery condition comprising that the stop running reason of the job is a specified reason, the specified reason being a reason supporting resuming the job to continue running.

[0121] For example, in combination with Figure 2 The receiving module 410 can be configured to perform S103 as shown in Figure 2 The processing module 420 can be configured to perform S104 as shown in Figure 2

[0122] In a possible implementation, the stop running reason is represented by an exit code, the exit code being an identifier corresponding to the stop running reason. In a possible implementation, the processing module 420 is further configured to,​

[0123] In the case that the stop running reason is represented by the exit code, the exit code is compared with an exit white list, the exit white list is stored in the management node, and includes exit codes of jobs that do not support resuming job to continue running;

[0124] If the exit code is not in the exit white list, the first job is reassigned.

[0125] In a possible implementation, the processing module 420 is further configured to,

[0126] In the case that the stop running reason of the first job is the specified reason, and the number of times of resuming the first job to continue running is less than or equal to a specified number of times, the first job is reassigned to a computing node in the cluster computing system.

[0127] In a possible implementation, the specified number of times is a parameter stored in the management node according to a received processing job request, and the specified number of times is used to indicate a maximum number of times of supporting a job to continue running by self-recovery, the processing job request is used to indicate that the management node allocates a computing node for the first job.

[0128] In a possible implementation, the apparatus further includes:

[0129] An update state module configured to, in the case that the stop running reason does not satisfy the self-recovery condition, update a state of the first job to a completed state, and a job in the completed state does not support self-recovery to continue running.

[0130] In a possible implementation, the processing module 420 is further configured to,

[0131] Send a message recovery request to a second computing node to make the first job continue running on the second computing node, the second computing node is the computing node that the management node reschedules to run the first job, and the message recovery request is used to resume running the first job.

[0132] For specific description of the above optional mode, refer to the foregoing method embodiments, which will not be described here. In addition, the foregoing explanation and beneficial effect of any one of the job recovery apparatuses provided above can refer to the foregoing corresponding method embodiments, which will not be described here.

[0133] As an example, in combination with Figure 1 , the functions implemented by part or all of the receiving module 410 and the processing module 420 in the job recovery apparatus can be implemented by the management node 100 in Figure 1 .

[0134] Exemplarily, Figure 7 A structural schematic diagram of a job resuming apparatus 400 is shown. The job resuming apparatus 400 is applied to a computing node, and the job resuming apparatus 400 comprises:

[0135] A monitoring module 510 is configured to monitor a running state of a first job.

[0136] A sending module 520 is configured to, if it is monitored that the first job stops running, send a stop running reason of the first job to a management node in the cluster computing system, the stop running reason being used to determine that the management node reassigns the first job to a computing node in the cluster computing system in a case where the stop running reason meets a self-recovery condition, and the self-recovery condition comprises that a stop running reason of a job is a specified reason, and the specified reason is a reason supporting a resuming job to continue running.

[0137] In a possible implementation, the sending module 520 is configured to,

[0138] If the stop running reason is represented by an exit code, in a case where it is monitored that the first job stops running, an exit code corresponding to the stop running is acquired.

[0139] The exit code is sent to the management node in the cluster computing system.

[0140] For a specific description of the above optional manner, refer to the foregoing method embodiments, and details are not described herein. In addition, the explanation and beneficial effect of any one of the job resuming apparatuses provided above can refer to the corresponding method embodiments described above, and details are not described herein.

[0141] As an example, in combination with Figure 1 , the functions implemented by part or all of the monitoring module 510 and the sending module 520 in the job resuming apparatus can be implemented by the computing node 200 in Figure 1 .

[0142] Figure 8 A structural schematic diagram of a computing device 1100 is shown. The management node can be implemented as the computing device 1100, and the computing node can also be implemented as the computing device 1100. The computing device 1100 can be an electronic device such as a smart phone, a tablet computer, an electronic book, a portable personal computer, a server, etc. The computing device 1100 in the present application can comprise one or more of the following components: a processor 1110 and a memory 1120.

[0143] The processor 1110 can include one or more processing cores. The processor 1110 connects various parts within the terminal through various interfaces and lines, performs various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 1120, and calling data stored in the memory 1120. Alternatively, the processor 1110 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor 1110 can integrate a combination of one or several of a central processing unit (CPU), a graphics processor (graphics processing unit, GPU), and a modem. Among them, the CPU mainly processes an operating system, a user interface, and an application program, etc.; the modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 1110, but be implemented by a separate communication chip.

[0144] The memory 1120 can include a random access memory (RAM) and can also include a read-only memory (ROM). Alternatively, the memory 1120 includes a non-transitory computer-readable storage medium. The memory 1120 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1120 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above-mentioned various method embodiments, etc., and the operating system can be an Android system (including a system developed based on the Android system), an IOS system developed by Apple Inc. (including a system developed based on the IOS system) or other systems. The data storage area can also store data created by the terminal in use (such as a phone book, audio and video data, chat record data) and the like.

[0145] In addition, those skilled in the art can understand that the structure of the computing device 1100 shown in the above-described drawings does not constitute a limitation on the computing device 1100, and the computing device 1100 can include more or fewer components than those shown in the diagram, or combine certain components, or different component arrangements. For example, the terminal also includes components such as radio frequency circuitry, a camera assembly, a sensor, audio circuitry, a wireless fidelity (WiFi) assembly, a power supply, a Bluetooth assembly, and the like, which are not described here.

[0146] The embodiments of the present application further provide a computer readable storage medium, which stores at least one computer instruction. The at least one computer instruction is loaded and executed by a processor to implement the job recovery method according to any of the above embodiments. The explanation and beneficial effects of the above-mentioned related content in any of the computer readable storage media provided can refer to the corresponding embodiments described above, and will not be described here.

[0147] The embodiments of the present application further provide a chip. The chip integrates a control circuit and one or more ports for implementing the functions of the above-mentioned job recovery apparatus. Optionally, the functions supported by the chip can refer to the above, and will not be described here. Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by programs instructing related hardware. The programs can be stored in a computer readable storage medium. The above-mentioned storage medium can be a read-only memory, a random access memory, etc. The above-mentioned processing unit or processor can be a central processing unit, a general-purpose processor, an application specific integrated circuit (ASIC), a microprocessor (digital signal processor, DSP), a field programmable gate array (FPGA) or other programmable logic device, a transistor logic device, a hardware component or any combination thereof.

[0148] The embodiments of the present application further provide a computer program product containing instructions, which, when executed on a computer, cause the computer to perform any of the methods described in the embodiments of the present application. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the processes or functions according to the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or include one or more data storage devices such as servers, data centers, etc. that can be integrated with the medium. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, an SSD), etc.

[0149] It should be noted that the above devices for storing computer instructions or computer programs provided by the embodiments of the present application, such as but not limited to the above memory, computer-readable storage medium, and communication chip, etc., are non-volatile. Those skilled in the art should realize that the functions described in the embodiments of the present application can be realized by hardware, software, firmware, or any combination thereof in one or more of the above examples. When software is used, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. The computer-readable storage medium includes a computer storage medium and a communication medium, wherein the communication medium includes any medium that facilitates the transfer of a computer program from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0150] The above is only an optional embodiment of the present application and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A job recovery method, characterized in that, Applied to a cluster computing system, the cluster computing system including a management node and at least one computing node, wherein at least one job is running on the at least one computing node of the cluster computing system, the method includes: If the first computing node detects that the first job has stopped running, the first computing node sends the reason for the stop of the first job to the management node. The first computing node is the computing node in the cluster computing system that is currently stopping the first job. Based on the reason for the stop operation, the management node determines whether the first job meets the self-recovery condition. If the first job meets the self-recovery condition, the first job is reassigned to the computing node in the cluster computing system. The self-recovery condition is used to indicate that the job can resume operation. If the reason for the stop operation is represented by an exit code, the management node determines whether the first job meets the self-recovery condition based on the exit code.

2. The method according to claim 1, characterized in that, Based on the reason for the shutdown, the management node determines whether the first job meets the self-recovery conditions, including: If the self-recovery condition is that the reason for the job's stoppage is a specified reason, then the management node supports the job's resumption. The management node compares whether the reason for the first job's stoppage is the specified reason.

3. The method according to claim 1, characterized in that, The reason for the shutdown is represented by an exit code, which is an identifier corresponding to the reason for the shutdown.

4. The method according to claim 3, characterized in that, Based on the reason for the shutdown, the management node determines whether the first job meets the self-recovery conditions, including: The management node compares the exit code with the exit whitelist to determine whether the first job meets the self-recovery condition; the exit whitelist is stored in the management node and includes exit codes of jobs that do not support the continued execution of the recovery job.

5. The method according to claim 4, characterized in that, If the first job meets the self-recovery condition, the first job is reassigned to a computing node in the cluster computing system, including: If the exit code of the first job is not in the exit whitelist, the first job is reassigned to a computing node in the cluster computing system.

6. The method according to any one of claims 1 to 5, characterized in that, If the first job meets the self-recovery condition, the first job is reassigned to a computing node in the cluster computing system, including: If the reason for the first job's shutdown meets the self-recovery condition, and the number of times the first job is restored to continue running is less than or equal to the specified number, the first job is reassigned to a computing node in the cluster computing system.

7. The method according to claim 6, characterized in that, The specified number of times is a parameter stored in the management node based on the received processing job request. The specified number of times is used to indicate the maximum number of times the job can continue to run on its own. The processing job request is used to instruct the management node to allocate a computing node for the first job.

8. The method according to any one of claims 1 to 5, characterized in that, The method further includes: If the reason for the stoppage does not meet the self-recovery conditions, the management node updates the status of the first job to complete. Jobs in the completed state do not support self-recovery and continued operation.

9. The method according to any one of claims 1 to 5, characterized in that, The step of reassigning the first job to a computing node in the cluster computing system includes: A message recovery request is sent to a second computing node in the cluster computing system so that the first job can continue to run on the second computing node, which is the computing node that the management node has rescheduled to run the first job. The message recovery request is used to resume the running of the first job.

10. A computing device, characterized in that, The computing device includes a processor and a memory; the processor is coupled to the memory; the memory is used to store computer instructions, which are loaded and executed by the processor to enable the computing device to implement the job recovery method as described in any one of claims 1 to 9.

Images