Resource recovery method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a resource recovery method and device, computer equipment, a storage medium and a program product. The method comprises the steps of obtaining resource use states of a plurality of computing nodes, determining an execution progress of a target job based on the resource use states of the computing nodes, and if the target job is executed, recycling computing resources for executing the target job in the computing nodes. Wherein the plurality of computing nodes cooperatively execute jobs. By adopting the method, the computing resources for executing the graphical operation can be recycled in time, so that resource waste is avoided.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a resource recovery method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] In the computer field, a supercomputing cluster refers to a cluster composed of multiple computers that can execute graphical and non-graphical operations by integrating the computing resources of multiple computers.

[0003] Taking the execution of a graphical job as an example, in the related art, a graphical process is usually started first to display a graphical interface to continuously display the running status of the graphical job until the entire graphical job is completed.

[0004] However, in the related art, it is impossible to recycle the computing resources used to execute graphical operations in a timely manner, resulting in a problem of resource waste. Summary of the invention

[0005] Based on this, it is necessary to provide a resource recovery method, device, computer equipment, storage medium and program product to address the above technical problems, which can timely recover the computing resources used to execute graphical operations, thereby avoiding waste of resources.

[0006] In a first aspect, the present application provides a resource recovery method, the method comprising:

[0007] Obtain resource usage status of multiple computing nodes; multiple computing nodes collaborate to execute jobs;

[0008] Determine the execution progress of the target job based on the resource usage status of each computing node;

[0009] If the target job has been completed, the computing resources used to execute the target job in each computing node are recycled.

[0010] In one embodiment, obtaining resource usage status of multiple computing nodes includes:

[0011] For any computing node, obtain the initial resource running quantity, current resource running quantity and target resource running quantity in the computing node; the initial resource running quantity is the initial number of processors allocated to the computing node to execute multiple jobs; the target resource running quantity is the number of processors allocated to the computing node to execute the target job;

[0012] Determine the number of idle processors of the computing node based on the initial resource running number and the previous resource running number;

[0013] The resource usage status of the computing node is determined based on the number of idle processors and the number of target resource running.

[0014] In one embodiment, determining the resource usage state of the computing node according to the number of idle processors and the number of target resource running includes:

[0015] If the number of idle processors is greater than or equal to the number of target resource operations, the resource usage state of the computing node is determined to be abnormal; the abnormal state indicates that the computing node has finished running the target job;

[0016] If the number of idle processors is less than the number of target resource operations, it is determined that the resource usage state of the computing node is normal; the normal state indicates that the computing node is running the target job.

[0017] In one embodiment, determining the execution progress of the target job based on the resource usage status of each computing node includes:

[0018] When the resource usage status of each computing node is in an abnormal state, the number of consecutive abnormalities of each computing node is obtained; the abnormal state indicates that the computing node ends running the target job;

[0019] If the number of consecutive exceptions of each computing node is greater than or equal to the preset threshold, it is determined that the target job has been executed and completed.

[0020] In one embodiment, the method further comprises:

[0021] If the resource usage status of any computing node is in a normal state, the resource usage status of each computing node is re-acquired until the resource usage status of each computing node is in an abnormal state; the normal state indicates that the computing node is running the target job.

[0022] In one embodiment, if the target job has been executed, the computing resources for executing the target job in each computing node are recycled, including:

[0023] If the target job has been completed, a resource recycling instruction for the target job is generated; the resource recycling instruction is used to instruct to check the execution progress of the target job, and generate a job recycling instruction if the check passes;

[0024] In response to the job recycling instruction, the computing resources for executing the target job in each computing node are recycled.

[0025] In a second aspect, the present application also provides a resource recovery device, comprising:

[0026] A status acquisition module is used to obtain the resource usage status of multiple computing nodes; multiple computing nodes collaborate to execute jobs;

[0027] A progress determination module is used to determine the execution progress of the target job based on the resource usage status of each computing node;

[0028] The resource recovery module is used to recover the computing resources used to execute the target job in each computing node if the target job has been completed.

[0029] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method in any one of the embodiments of the first aspect are implemented.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any one of the embodiments of the first aspect above.

[0031] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method in any one of the embodiments of the first aspect above.

[0032] The resource recovery method, device, computer equipment, storage medium and program product obtain the resource usage status of multiple computing nodes that collaboratively execute jobs, then determine the execution progress of the target job based on the resource usage status of each computing node, and when the target job has been completed, recycle the computing resources that execute the target job in each computing node. In this method, on the basis of multiple computing nodes collaboratively executing jobs, taking into account the differences in the execution of target jobs by different computing nodes, according to the resource usage status of multiple computing nodes, the execution of the target job by each computing node is obtained, and then the overall running progress of the target job is determined to ensure the objectivity and accuracy of the running progress of the target job. Then, when it is determined that the target job has been completed, the computing resources that execute the target job in each computing node are promptly recycled to avoid resource waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0034] FIG1 is an application environment diagram of a resource recovery method in one embodiment;

[0035] FIG2 is a schematic diagram of a process flow of a resource recovery method in one embodiment;

[0036] FIG3 is a schematic diagram of a flow chart of a resource usage status determination step in one embodiment;

[0037] FIG4 is a flow chart of a resource usage status determination step in another embodiment;

[0038] FIG5 is a schematic diagram of a process for determining the execution progress of a target job in one embodiment;

[0039] FIG6 is a schematic diagram of a process flow of a target operation resource recovery step in one embodiment;

[0040] FIG7 is a schematic diagram of a process flow of a resource recovery method in another embodiment;

[0041] FIG8 is a schematic diagram of a process flow of a supercomputing cluster in one embodiment;

[0042] FIG9 is a structural block diagram of a resource recovery device in one embodiment;

[0043] FIG. 10 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] The technical background of this application is first described below.

[0046] In the computer field, when faced with computing tasks for computationally intensive jobs, an ordinary computer is usually unable to complete the computing tasks within a reasonable time, or even unable to perform the computing tasks, due to the limited computing resources of a single computer. In this context, supercomputing clusters are widely used to process computationally intensive jobs, such as simulation, modeling, and rendering, with their ability to support the integration of computing resources from multiple computers. Among them, a supercomputing cluster refers to a cluster composed of multiple computers, and each computer can be regarded as a node in the supercomputing cluster.

[0047] In the actual application scenarios of supercomputing clusters, jobs can be divided into two categories: graphical jobs and non-graphical jobs, depending on the job type. The process of supercomputing clusters executing non-graphical jobs is usually as follows: the scheduler in the supercomputing cluster allocates computing resources to the job, specifically dividing the job into multiple computing subtasks and assigning them to different computing nodes. The computing nodes then start the computing process to execute the computing tasks until the final job execution results are output, and the scheduler reclaims the computing resources related to the job.

[0048] The process of executing a graphical job in a supercomputing cluster is usually as follows: the scheduler in the supercomputing cluster allocates computing resources to the job, specifically dividing the job into multiple computing subtasks and assigning them to different computing nodes, instructing the computing nodes to start the computing process to execute the computing task; at the same time, the graphical interface of the job is started to display the graphical interface of the job, and the running status of the graphical job is continuously displayed until the entire graphical job is executed. According to the closing instruction of the graphical interface, the computing resources for executing the graphical job are recovered. In other words, before receiving the closing instruction, the scheduler will not recover the relevant computing resources, and the computing process for executing the graphical job in the computing cluster is still in the executing state, and the computing process is occupied, resulting in a waste of computing resources.

[0049] Based on this, an embodiment of the present application provides a resource recovery method, which obtains the resource usage status of multiple computing nodes that collaboratively execute a job, and uses this as a basis to determine the execution progress of the target job. If the target job has been completed, the computing resources executing the target job in each computing node are promptly recovered to avoid wasting computing resources.

[0050] The resource recovery method provided in the embodiment of the present application can be applied to the supercomputing cluster shown in Figure 1. The supercomputing cluster includes a control node 101, a computing node 102 and a display node 103. The control node 101 is responsible for managing the computing resources of the entire supercomputing cluster, the computing node 102 is responsible for running applications and executing computing tasks, and the display node 103 is responsible for graphically displaying applications.

[0051] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0052] In an exemplary embodiment, as shown in FIG2 , a resource recovery method is provided, which is described by taking the method applied to a management and control node as an example, and includes the following steps:

[0053] S201, obtaining resource usage status of multiple computing nodes; multiple computing nodes collaboratively execute a job.

[0054] The computing node refers to a physical or virtual server that is responsible for computing tasks and performing data processing and storage. The computing node in the embodiment of the present application can execute a part of the sub-jobs of a single job, and multiple computing nodes can collaboratively execute the same job.

[0055] It should be noted that there is a many-to-many mapping relationship between computing nodes and jobs, that is, one computing node can execute sub-jobs of multiple jobs, and one job can be jointly executed by multiple computing nodes. For any computing node, the resource usage status includes the initial resources allocated by the computing node for the sub-jobs of multiple jobs, the actual resources for executing sub-jobs of different jobs, etc., which can represent the progress of the computing node in executing each job on the node to which it belongs.

[0056] In terms of the method of obtaining the resource usage status of each computing node, a resource status request can be sent to each computing node in the computing cluster at a preset time interval, instructing each computing node to return the current resource usage status as a resource status response to the management and control node, or the resource usage status reported by each computing node can be received in real time.

[0057] S202: Determine the execution progress of the target job based on the resource usage status of each computing node.

[0058] In actual application scenarios, the resource usage status includes a job identifier, which is used to uniquely indicate a job, and the job identifiers of the same job on different computing nodes are consistent.

[0059] The target job refers to a complete job executed by multiple computing nodes in collaboration. The number of target jobs can be one or more. In the embodiment of the present application, the steps of determining the execution progress of each target job are performed in parallel. The following describes the process of determining the execution progress of the target job by taking the target job as one.

[0060] Based on the resource usage status of each computing node, multiple target computing nodes with the same job identifier are determined. Then, based on the resource usage status of the target computing nodes, the sub-progress of each target computing node executing the target job is determined. Then, based on the execution sub-progress of each target computing node, the execution progress of the target job is determined.

[0061] Optionally, the sub-progress of each target computing node executing the target job can be divided into completed and uncompleted, and the execution progress of the target job can be represented by the ratio of the number of completed target computing nodes to all target computing nodes.

[0062] Taking the determination of N target computing nodes corresponding to the target job as an example, if it is determined that the sub-progress of the target job executed by N target computing nodes is not completed, that is, the sub-progress of the target job executed by 0 target computing nodes is completed, then the execution progress of the target job is determined to be 0%; if it is determined that the sub-progress of the target job executed by (N / 2) target computing nodes is completed, then the execution progress of the target job is determined to be 50%; if it is determined that the sub-progress of the target job executed by N target computing nodes is completed, then the execution progress of the target job is determined to be 100%.

[0063] S203: If the target job has been completed, the computing resources used to execute the target job in each computing node are recycled.

[0064] When the target job has been completed, the calculation result of the target job can be saved, and a preset resource recycling tool can be called to automatically recycle the computing resources used to execute the target job in each computing node.

[0065] In another scenario, alarm information can also be issued through various means such as a graphic interface, sound, and buzzer to instruct users of the supercomputing cluster to intervene in a timely manner and reclaim computing resources in each computing node that execute the target job.

[0066] In an embodiment of the present application, the resource usage status of multiple computing nodes that collaboratively execute jobs is obtained, and then based on the resource usage status of each computing node, the execution progress of the target job is determined, and when the target job has been completed, the computing resources that execute the target job in each computing node are recycled. In this method, on the basis of the collaborative execution of jobs by multiple computing nodes, taking into account the differences in the execution of target jobs by different computing nodes, the situation of each computing node executing the target job is obtained according to the resource usage status of multiple computing nodes, and then the overall running progress of the target job is determined to ensure the objectivity and accuracy of the running progress of the target job. Then, when it is determined that the target job has been completed, the computing resources that execute the target job in each computing node are promptly recycled to avoid waste of resources.

[0067] As can be seen from the above embodiments, the resource usage status is the basis for determining the target job execution progress, and is crucial to the accuracy of the target job execution progress. Based on this, the following further describes the method and content of obtaining the resource usage status through an embodiment.

[0068] In an exemplary embodiment, as shown in FIG3 , obtaining resource usage status of multiple computing nodes includes the following steps:

[0069] S301, for any computing node, obtaining the initial resource running quantity, the current resource running quantity and the target resource running quantity in the computing node.

[0070] The initial resource running quantity is the initial number of processors allocated to the computing node to execute multiple jobs; the target resource running quantity is the number of processors allocated to the computing node to execute the target job.

[0071] In practical applications, a computing node includes multiple processors, which may be a central processing unit (CPU) or a graphics processing unit (GPU).

[0072] It should be noted that the initial resource running quantity and the target resource running quantity are fixed values, which are determined after the management and control node receives the jobs (including the target jobs) transmitted to the computing cluster.

[0073] Based on this, the initial resource running quantity and the target resource running quantity of each computing node can be pre-stored in the storage space, so that for any computing node, they can be read from the storage space according to the identifier of the computing node.

[0074] Unlike the fixed values ​​of the initial resource running quantity and the target resource running quantity, the current resource running quantity of the computing node is changing dynamically, and as the computing node takes longer and longer to execute computing tasks, the current resource running quantity is gradually decreasing.

[0075] The process of obtaining the current number of running resources may be to receive the usage rate of each processor reported by the computing node, then determine the processor whose usage rate is greater than a preset threshold as the target processor in use, and use the total number of target processors as the current number of running resources of the computing node.

[0076] S302: Determine the number of idle processors of the computing node based on the initial number of running resources and the previous number of running resources.

[0077] The current number of running resources is the number of processors currently used by the computing node. The current number of running resources is less than or equal to the initial number of running resources.

[0078] The difference between the initial resource running quantity and the current resource running quantity is taken as the idle processor quantity of the computing node. The idle processor quantity refers to the number of processors of the computing node that have completed the computing tasks.

[0079] S303: Determine the resource usage status of the computing node according to the number of idle processors and the number of running target resources.

[0080] It is emphasized again that the number of idle processors refers to the number of processors in the computing node that have completed computing tasks, and the number of target resources running refers to the number of processors allocated by the computing node to execute the target job. Then, by comparing whether the number of idle processors matches the number of target resources running, the execution progress of the computing node in processing the target job can be determined, and then the resource usage status of the computing node can be determined.

[0081] Optionally, if the number of idle processors matches the number of running target resources, it is determined that the processor corresponding to the idle number of processors of the computing node is the processor for the computing node to execute the target job, and it is determined that the computing node has completed the computing task of the target job; if the number of idle processors does not match the number of running target resources, it is determined that the processor corresponding to the idle number of processors of the computing node is not the processor for the computing node to execute the target job, and it is determined that the computing node has not completed the computing task of the target job.

[0082] In an embodiment of the present application, for any computing node, the number of idle processors of the computing node is determined based on the initial number of running resources and the previous number of running resources, as the number of processors that have completed computing tasks on the computing node. The idle number of processors is then compared with the target number of running resources to determine whether the processor that has completed computing tasks on the computing node is the processor that executes the target job, thereby ensuring the accuracy of the resource usage status of the computing node.

[0083] In an exemplary embodiment, as shown in FIG. 4 , the above-mentioned step of “determining the resource usage state of the computing node according to the number of idle processors and the number of target resource running” in S303 includes the following steps:

[0084] S401, if the number of idle processors is greater than or equal to the number of target resources running, determine that the resource usage state of the computing node is an abnormal state; the abnormal state indicates that the computing node ends running the target job.

[0085] In actual computing scenarios, a computing node can execute sub-jobs of a job through one or more processors. When multiple processors of a computing node execute sub-jobs, the computing resources of each processor of the computing node are equal, which means that different processors of the computing node take the same amount of time to execute sub-jobs of the same job.

[0086] If the number of idle processors is equal to the number of target resource operations, it indicates that the computing node has finished running the target job, and the resource usage state of the computing node is determined to be an abnormal state.

[0087] If the number of idle processors is greater than the number of target resource operations, it indicates that the computing node has finished running the target job and sub-jobs of other jobs, and the resource usage state of the computing node is determined to be an abnormal state.

[0088] S402: If the number of idle processors is less than the number of target running resources, determine that the resource usage state of the computing node is normal; the normal state indicates that the computing node is running the target job.

[0089] If the number of idle processors is equal to the number of target resource running, it indicates that the computing node has not finished running the target job, and the resource usage state of the computing node is determined to be a normal state.

[0090] In an embodiment of the present application, for any computing node, when the number of idle processors is greater than or equal to the number of running target resources, it is determined that the computing node has finished running the target job; when the number of idle processors is less than the number of running target resources, it is determined that the computing node is running the target job, which is equivalent to using the comparison result of the idle number of processors and the number of running target resources as the basis for whether the computing node is running the target job, thereby providing a quantitative and specific determination logic for the resource usage status of the computing node.

[0091] After determining the resource usage status of each computing node, it is necessary to integrate them to determine the overall execution progress of the target job. In an exemplary embodiment, as shown in FIG5 , based on the resource usage status of each computing node, determining the execution progress of the target job includes the following steps:

[0092] S501, when the resource usage status of each computing node is in an abnormal state, obtain the number of consecutive abnormalities of each computing node; the abnormal state indicates that the computing node ends running the target job.

[0093] In an embodiment of the present application, the resource usage status of each computing node is uniformly obtained at a preset time interval until the resource usage status of each computing node is in an abnormal state, and it is determined that the overall execution progress of the target job has been completed.

[0094] Furthermore, when it is determined for the first time that each computing node has finished running the target job, the above steps are continued to be performed, that is, the resource usage status of each computing node is uniformly obtained, and the number of consecutive exceptions of each computing node is recorded.

[0095] S502: If the number of consecutive exceptions of each computing node is greater than or equal to a preset threshold, it is determined that the target job has been completed.

[0096] If the number of consecutive exceptions of each computing node is greater than or equal to the preset threshold, it means that each computing node has completed the computing task of the target job to which it belongs, and it is determined that the target job has been executed.

[0097] In an embodiment of the present application, when the resource usage status of each computing node is in an abnormal state and the number of consecutive abnormalities of each computing node is greater than or equal to a preset threshold, it is determined that the target job has been executed. This is equivalent to continuing to obtain the number of consecutive abnormalities of each computing node after first determining that each computing node has finished running the target job, repeatedly verifying whether the overall execution progress of the target job is completed, and ensuring the accuracy of the results of the target job execution progress.

[0098] In an exemplary embodiment, the method further comprises:

[0099] If the resource usage status of any computing node is in a normal state, the resource usage status of each computing node is re-acquired until the resource usage status of each computing node is in an abnormal state; the normal state indicates that the computing node is running the target job.

[0100] At preset time intervals, the resource usage status of each computing node is uniformly obtained. If the resource usage status is in an abnormal state, it means that all computing nodes have finished executing the target job. If the resource usage status of any computing node is in a normal state, it means that there is a computing node that has not completed the target job. In this case, the resource usage status of each computing node is re-obtained until the resource usage status of each computing node is in an abnormal state.

[0101] In an embodiment of the present application, when the resource usage status of any computing node is in a normal state, the resource usage status of each computing node is reacquired until the resource usage status of each computing node is in an abnormal state, which is equivalent to regularly updating the progress of each computing node in executing the target job to improve the effectiveness of the resource usage status of each computing node, and then timely determine the time when each computing node has completed the target job, and quickly recycle the computing resources corresponding to the target job.

[0102] There are many ways to recycle resources in a computing cluster, such as powering off and restarting, executing resource recycling scripts, etc. Below, one possible way to recycle resources is described.

[0103] In an exemplary embodiment, as shown in FIG6 , if the target job has been executed, the computing resources for executing the target job in each computing node are recycled, including the following steps:

[0104] S601, if the target job has been executed, generate a resource recovery instruction for the target job; the resource recovery instruction is used to instruct to check the execution progress of the target job, and generate a job recovery instruction if the check passes.

[0105] When it is determined that the target job has been completed, a resource recovery instruction for the target job is automatically generated and displayed in the form of voice, text, etc., instructing each computing node to re-check the execution progress of the target job and generate a job recovery instruction if the check passes.

[0106] Optionally, the resource recycling instruction of the target job is displayed on the interface of the display node to instruct the user of the supercomputing cluster to manually recheck whether the execution progress of the target job is completed on each computing node, and generate a job recycling instruction if the verification is passed.

[0107] Optionally, after the resource recycling instruction of the target job, the execution progress verification script is automatically triggered to uniformly obtain the resource usage status of each computing node again, obtain whether the resource usage status of each computing node is abnormal, verify the overall execution progress of the completed target job, and generate a job recycling instruction if the verification passes.

[0108] S602 , in response to the job recovery instruction, recover computing resources for executing the target job in each computing node.

[0109] In response to the job recycling instruction, a resource recycling script of the target job is executed to recycle the computing resources associated with the target job.

[0110] In an embodiment of the present application, when it is determined that the target job has been executed, a resource recovery instruction for the target job is generated to re-check the execution progress of the target job, thereby ensuring the accuracy of the result that the overall execution progress of the target job has been completed, and further ensuring the validity of the job recovery instructions generated in the verification scenario, thereby avoiding erroneous recovery or wrong recovery of computing resources.

[0111] In an exemplary embodiment, as shown in FIG7 , the resource recovery method includes the following steps:

[0112] S701, every one minute, traverse the graphical jobs in the supercomputing cluster that are in operation.

[0113] S702, obtaining computing nodes corresponding to each graphical job.

[0114] S703: For any graphical job, analyze the resource usage of each computing node for the graphical job.

[0115] S704, determining whether there is a computing process of the graphical job on all computing nodes of the graphical job.

[0116] S705: If yes, set the number of consecutive exceptions of the graphical operation to 0.

[0117] S706: If not, record the number of consecutive exceptions of the graphics operation.

[0118] S707, determining whether the number of consecutive abnormalities exceeds a preset threshold.

[0119] S708: If yes, trigger an alarm indication message to instruct to recycle the computing resources of the graphical job.

[0120] If not, return to step S701.

[0121] In an embodiment of the present application, the resource usage status of multiple computing nodes that collaboratively execute jobs is obtained at regular intervals, and then based on the resource usage status of each computing node, the execution progress of the target job is determined, and when the target job has been completed, the computing resources that execute the target job in each computing node are recycled. This is equivalent to, on the basis of the collaborative execution of jobs by multiple computing nodes, taking into account the differences in the execution of target jobs by different computing nodes, obtaining the situation of each computing node executing the target job according to the resource usage status of multiple computing nodes, and then determining the overall running progress of the target job to ensure the objectivity and accuracy of the running progress of the target job. Then, when it is determined that the target job has been completed, the computing resources that execute the target job in each computing node are recycled in a timely manner to avoid waste of resources.

[0122] Taking the supercomputing cluster shown in FIG8 as an example, the overall implementation process of the resource recovery method is described.

[0123] In Figure 8, the supercomputer cluster includes 1 control node, 3 computing nodes, and 1 display node. The control node deploys the server service (server), and the computing node deploys the agent service (agent). Each computing node has 8 processors, and each display node has 4 processors. The processors are represented by circles (they can also be quantified computing resources that are strongly related to computing power, such as CPU, GPU, etc.).

[0124] Taking the CPU as an example, in Figure 8, there are three graphical jobs (job A, job B, and job C) running in the supercomputing cluster. Job A includes one display process running on the display node (allocated to one CPU), eight computing processes, four of which run on computing node 1 (allocated to four CPUs), and four of which run on computing node 2 (allocated to four CPUs).

[0125] Job B includes 1 display process running on the display node (allocated 1 CPU) and 2 computing processes on computing node 1 (allocated 2 CPUs).

[0126] Job C includes 1 display process running on the display node (allocated 1 CPU), 6 computing processes, 4 of which run on computing node 2 (allocated 4 CPUs), and 2 run on computing node 3 (allocated 2 CPUs).

[0127] Considering that the jobs on the supercomputing cluster are all computationally intensive and will use a lot of CPU resources, when a computing process is running on the CPU, the corresponding CPU usage rate will be relatively high. Based on this, according to the empirical value, the CPU usage threshold is set to 10%. When the usage rate of a single CPU exceeds the threshold, it is determined that a computing process is running on the CPU; when the usage rate of a single CPU is lower than the threshold, it is determined that no computing process is running on the CPU.

[0128] In the supercomputing cluster shown in FIG8 , taking the server deployed as the control node as an example, the overall implementation process of the resource recovery method includes the following steps:

[0129] (1) Regularly collect the usage of each CPU of the computing node collected by the proxy service.

[0130] (2) Traverse the running graphical jobs, namely job A, job B and job C.

[0131] (3) Obtain the computing nodes corresponding to job A, and obtain computing nodes 1 and 2.

[0132] (4) Get the number of CPUs allocated to compute node 1. Job A is allocated 4 CPUs and job B is allocated 2 CPUs. The total number of CPUs allocated to compute node 1 is 6.

[0133] (5) If (the number of CPUs allocated to the computing node – the number of CPUs on which computing processes are running) is greater than or equal to the number of CPUs allocated to job A, mark the computing node as abnormal; otherwise, mark it as normal.

[0134] (6) Get the number of CPUs allocated to computing node 2. Job A allocated 4 CPUs and job C allocated 4 CPUs. The number of CPUs allocated to computing node 2 is 8. If (the number of CPUs allocated to the computing node – the number of CPUs with running computing processes) is greater than or equal to the number of CPUs allocated to job A, mark the computing node as abnormal, otherwise mark it as normal.

[0135] (7) If the status of all computing nodes corresponding to job A is abnormal, then job A may no longer have a computing process. The number of consecutive abnormalities of job A is increased by 1. If the number of consecutive abnormalities of job A is greater than or equal to the maximum number of abnormalities (the default value is 5, and the maximum number of abnormalities can be set based on experience), an alarm for job A is triggered, and cluster users are notified to check the running status of job A. If the status of all computing nodes corresponding to job A is at least one normal, then job A still has a computing process, and the number of consecutive abnormalities of job A is set to 0.

[0136] (8) Refer to steps (3) to (7) to analyze Job B and Job C.

[0137] (9) Repeat steps (1) to (8).

[0138] Optionally, if the calculation process of job A has ended, but the calculation processes of jobs B and C are still ongoing: after the above analysis algorithm processing, jobs A and B will trigger an alarm indicating that the calculation process has ended. Cluster users will find that job A has indeed completed the calculation, and can close the graphical interface of job A at this time, and job A ends; but users find that the calculation of job B has not yet been completed, and users can reset the status of job B to running.

[0139] Optionally, if the calculation process of job B has ended, but the calculation processes of jobs A and C are still ongoing: after the above analysis algorithm is processed, job B will trigger an alarm indicating that the calculation process has ended. Cluster users will find that job B has indeed completed the calculation. At this time, they can close the graphical interface of job B and job B will end.

[0140] Optionally, if the calculation process of job C has ended, but the calculation processes of jobs A and B are still ongoing: after the above analysis algorithm is processed, job C will trigger an alarm indicating that the calculation process has ended. Cluster users will find that job C has indeed completed the calculation. At this time, they can close the graphical interface of job C and job C will end.

[0141] In the embodiment of the present application, when there are multiple jobs running simultaneously in the supercomputing cluster, the job status analysis algorithm is used to identify the jobs whose computing process may have ended, and an alarm is sent to the user, notifying the user to view the results of the job calculation and close the graphical interface in time, thus solving the problem that after the graphical job calculation is completed, the job will not end because the graphical interface is still open, and the computing resources are occupied. In addition, if after sending the alarm, the user determines that a job has been misidentified, the job can be reset to running without affecting the continued calculation of the job.

[0142] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0143] Based on the same inventive concept, the embodiment of the present application also provides a resource recovery device for implementing the resource recovery method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more resource recovery device embodiments provided below can refer to the limitations on the resource recovery method above, and will not be repeated here.

[0144] In an exemplary embodiment, as shown in FIG9 , a resource recovery device is provided, including: a state acquisition module 901, a progress determination module 902 and a resource recovery module 903, wherein:

[0145] The status acquisition module 901 is used to acquire the resource usage status of multiple computing nodes; multiple computing nodes collaborate to execute a job;

[0146] The progress determination module 902 is used to determine the execution progress of the target job based on the resource usage status of each computing node;

[0147] The resource recovery module 903 is used to recover the computing resources used to execute the target job in each computing node if the target job has been executed.

[0148] In an exemplary embodiment, the state acquisition module 901 includes: a resource information acquisition unit, an idle resource determination unit and a usage state determination unit, wherein:

[0149] A resource information acquisition unit is used to acquire, for any computing node, the initial resource running quantity, the current resource running quantity and the target resource running quantity in the computing node; the initial resource running quantity is the initial number of processors allocated to the computing node for executing multiple jobs; the target resource running quantity is the number of processors allocated to the computing node for executing the target job;

[0150] An idle resource determination unit, used to determine the idle number of processors of the computing node based on the initial resource running number and the previous resource running number;

[0151] The usage status determination unit is used to determine the resource usage status of the computing node according to the idle number of processors and the running number of target resources.

[0152] In an exemplary embodiment, a status determination unit is used to determine that the resource usage status of the computing node is an abnormal state if the number of idle processors is greater than or equal to the number of target resources running; the abnormal state indicates that the computing node has finished running the target job; if the number of idle processors is less than the number of target resources running, the resource usage status of the computing node is determined to be a normal state; the normal state indicates that the computing node is running the target job.

[0153] In an exemplary embodiment, the progress determination module 902 includes: an abnormality number acquisition unit and an abnormality number judgment unit, wherein:

[0154] The abnormality number acquisition unit is used to acquire the continuous abnormality number of each computing node when the resource usage status of each computing node is in an abnormal state; the abnormal state indicates that the computing node has finished running the target job;

[0155] The abnormality number judgment unit is used to determine that the target job has been executed and completed if the continuous abnormality number of each computing node is greater than or equal to a preset threshold.

[0156] In an exemplary embodiment, the progress determination module 902 includes: a status monitoring unit, which is used to re-acquire the resource usage status of each computing node if the resource usage status of any computing node is in a normal state until the resource usage status of each computing node is in an abnormal state; the normal state indicates that the computing node is running the target job.

[0157] In an exemplary embodiment, the resource recovery module 903 includes: a recovery instruction generation unit and a recovery instruction response unit, wherein:

[0158] A recycling instruction generating unit is used to generate a resource recycling instruction for the target job if the target job has been completed; the resource recycling instruction is used to instruct to check the execution progress of the target job and generate a job recycling instruction if the check passes;

[0159] The recovery instruction response unit is used to respond to the job recovery instruction and recover the computing resources for executing the target job in each computing node.

[0160] The resource recovery device obtains the resource usage status of multiple computing nodes that collaboratively execute the job, and then determines the execution progress of the target job based on the resource usage status of each computing node, and recycles the computing resources that execute the target job in each computing node when the target job has been completed. In this device, based on the collaborative execution of the job by multiple computing nodes, taking into account the differences in the execution of the target job by different computing nodes, the execution status of each computing node is obtained according to the resource usage status of multiple computing nodes, and then the overall running progress of the target job is determined to ensure the objectivity and accuracy of the running progress of the target job. Then, when it is determined that the target job has been completed, the computing resources that execute the target job in each computing node are promptly recovered to avoid resource waste.

[0161] Each module in the resource recovery device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0162] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG10. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store XX data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a resource recovery method is implemented.

[0163] Those skilled in the art will understand that the structure shown in FIG. 10 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0164] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the embodiments of the above-mentioned method steps when executing the computer program.

[0165] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the embodiments of the above-mentioned method steps are implemented.

[0166] In an exemplary embodiment, a computer program product is provided, including a computer program, which implements the embodiments of the above-mentioned method steps when executed by a processor.

[0167] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0168] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0169] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0170] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A resource recovery method, characterized in that: The method comprises: Acquiring resource usage status of multiple computing nodes; the multiple computing nodes collaboratively executing a job; Determining the execution progress of the target job based on the resource usage status of each of the computing nodes; If the target job has been completed, the computing resources used to execute the target job in each computing node are recycled.

2. The method according to claim 1, characterized in that The obtaining of resource usage status of multiple computing nodes includes: For any computing node, obtain the initial resource running quantity, current resource running quantity and target resource running quantity in the computing node; the initial resource running quantity is the initial number of processors allocated to the computing node for executing multiple jobs; the target resource running quantity is the number of processors allocated to the computing node for executing the target job; Determining the idle number of processors of the computing node based on the initial resource running number and the previous resource running number; The resource usage state of the computing node is determined according to the idle number of processors and the running number of the target resources.

3. The method according to claim 2, characterized in that The determining the resource usage state of the computing node according to the idle number of processors and the running number of target resources includes: If the idle number of the processors is greater than or equal to the running number of the target resources, it is determined that the resource usage state of the computing node is an abnormal state; the abnormal state indicates that the computing node ends running the target job; If the idle number of processors is less than the target running number of resources, it is determined that the resource usage state of the computing node is a normal state; the normal state indicates that the computing node is running the target job.

4. The method according to any one of claims 1 to 3, characterized in that: The determining the execution progress of the target job based on the resource usage status of each computing node includes: When the resource usage status of each computing node is in an abnormal state, obtaining the number of consecutive abnormalities of each computing node; the abnormal state indicates that the computing node ends running the target job; If the number of consecutive exceptions of each of the computing nodes is greater than or equal to a preset threshold, it is determined that the target job has been completed.

5. The method according to claim 4, characterized in that The method further comprises: If the resource usage status of any computing node is in a normal state, the resource usage status of each computing node is reacquired until the resource usage status of each computing node is in the abnormal state; the normal state indicates that the computing node is running the target job.

6. The method according to any one of claims 1 to 3, characterized in that: If the target job has been executed, the computing resources for executing the target job in each computing node are recycled, including: If the target job has been completed, a resource recovery instruction for the target job is generated; the resource recovery instruction is used to instruct to check the execution progress of the target job, and generate a job recovery instruction if the check passes; In response to the job recycling instruction, the computing resources for executing the target job in each of the computing nodes are recycled.

7. A resource recovery device, characterized in that: The device comprises: A status acquisition module, used to acquire resource usage status of multiple computing nodes; the multiple computing nodes collaboratively execute jobs; A progress determination module, used to determine the execution progress of the target job based on the resource usage status of each of the computing nodes; The resource recovery module is used to recover the computing resources used to execute the target job in each computing node if the target job has been completed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images